New AI Pentest Agent Erebus Strike: How SySS's AI-assisted Pentests Work

Modern AI models are increasingly developing capabilities that, until recently, were the exclusive domain of experienced security specialists. With the help of such models, it has also become, to some extent, more resource-efficient, cost-effective, and convenient to find exploitable vulnerabilities – for example, in exposed services – than to carry out a complex social engineering campaign. In many cases, the time required to develop specific attack tools and exploits has also been significantly reduced. What might have taken several days or weeks just a year ago can now be accomplished in a matter of minutes.

This increase in efficiency directly leads to a more severe threat landscape. However, these new capabilities can also be specifically leveraged to enhance the efficiency of penetration tests. When used correctly, artificial intelligence can, for example, help achieve broader test coverage and greater test depth in penetration tests.

Since early 2025, we at SySS have been researching how to meaningfully integrate AI into our security tests. We conducted numerous tests with different models and penetration testing agents, but without achieving sufficiently good results. At the same time, we also developed, hosted, and evaluated internal AI systems. Finally, we began developing our own pentest agent. It was particularly important to us to clearly define the framework for a penetration test project and to be able to reliably adhere to it without unnecessarily restricting the creativity and flexibility of the models used. We also wanted to demonstrate that simply using a powerful state-of-the-art model from vendors such as OpenAI or Anthropic is not sufficient to achieve robust test results. Our assumption was that effective harnessing – that is, a robust system for guiding, managing, and monitoring the AI – and a suitable agent architecture are at least as critical as the sheer performance of the underlying model. These considerations led to the development of our penetration testing agent "Erebus Strike".

Inventing Erebus Strike: How well does a pentest agent perform?

Erebus Strike first had to prove itself on a test application developed by SySS using AI-assisted vibe coding. Our developers implemented 71 vulnerabilities, spanning all OWASP Top 10 categories. In addition, there were business logic flaws – application-specific vulnerabilities in the business logic that are not covered by the OWASP Top 10. Our AI specialists rated the vulnerabilities based on exploitability and detectability on a scale of 1 to 5. A detectability score of 5 means the vulnerability is very difficult to find – in some cases, only through a source code review. An exploitability score of 5, for example, would only be possible under laboratory conditions.

After just a few development iterations, Erebus Strike achieved significantly better results in our internal test environment than the previously tested approaches. Erebus Strike found all 71 intentionally implemented vulnerabilities. Another curious finding was that the SySS agent also discovered vulnerabilities in vibe-coded code that had not been intentionally placed there.

This means that AI-assisted pentests can achieve unprecedented speeds and a level of scalability that was previously unimaginable. Erebus Strike can analyze vast amounts of source code, contexts, hypotheses, and security-relevant code paths and process them in a very short time. This includes, for example, authentication routines, input processing, and everything that enters the code from the outside – whether from users or attackers – and is subsequently processed. Erebus Strike is therefore ideally suited for all software-driven test subjects. Security analyses of web applications and web services, automated software analyses, reverse engineering, and firmware analyses (such as for embedded testing) can be performed optimally with Erebus Strike. SySS can use various models for this purpose. When we use SySS’s own infrastructure, which is hosted at our headquarters in Tübingen, all data remains within the SySS infrastructure. This is a prerequisite, particularly for security-critical clients. On the other hand, at the customer’s request, we can also use third-party providers such as Anthropic or OpenAI, or utilize the customer’s own AI environment. As a middle ground – which actually works best for most customers – we can rely on self-hosted OpenWeight models in data centers operated by EU providers and located within the EU. The customer decides which model to use.

Preparing for the Test: What is a custom harness?

Just as with traditional penetration tests, the client plays a central role, especially during test preparation. The client and the tester work together to define the custom harness. Like any other penetration test, an AI-assisted pentest is preceded by a kick-off meeting with a SySS consultant. During this meeting, relevant information and data are discussed, such as: Is there a test environment? When can testing take place? What access points are available? And so on.

The advantages and disadvantages of the various models are also discussed during the kickoff.

For AI-assisted penetration tests, it is also highly beneficial if the customer provides the source code. While human pentesters cannot review the entire source code within a limited test timeframe, Erebus Strike is capable of scanning it quickly and forming strong hypotheses. This increases the chances of finding relevant vulnerabilities. However, even in black box assessments, Erebus Strike achieves significantly better results with small models than Frontier models using simple prompts and access to the source code.

Erebus Strike is then “fed” all the information from the kick-off meeting. Individual customer requests can also be taken into account.

Moritz Abrell, the developer of Erebus Strike, explains the specific process as follows: 

"In the configuration, for example, Erebus Strike asks for the target to be tested and whether certain items are out of scope, such as specific endpoints or functions. An auxiliary prompt is very important. There, I can enter all the information the customer has mentioned. If the customer mentions during the kick-off meeting that a specific feature was implemented in the latest release – one that has never been subjected to a pentest and should be in the focus – I can enter that there. Erebus Strike takes this focus into account.”

The developer goes on to explain: “A major advantage of Large Language Models (LLMs) is that they can process large amounts of data. I can provide a wide variety of materials: source code, the results of a previous pentest, API documentation, general application documentation, the kick-off meeting minutes, the underlying threat model, as well as architecture documentation or user manuals. Erebus Strike reviews this material and incorporates it into the test. Having this information available greatly increases test efficiency.”

In the next step, the SySS tester creates accounts for various roles. This may also include “disposable” accounts used for brute-force attacks or similar tests to ensure that no other test accounts are locked out.

Afterward, the SySS consultant selects the back end – that is, the LLM – to be used for the security test. The tester can also choose different models for specific steps. If the verification step needs to be particularly robust, but the discovery step is less demanding, different models can be used for each. Using multiple models can also be beneficial for particularly complex applications.

Starting the Test: How does a pentest agent work?

To understand how a pentest agent works, we first need to clarify what an agent actually is. An agent is, first and foremost, the automation of a specific process. Automated processes in pentests have long been familiar in the form of vulnerability scans. However, while such a scan follows a relatively strict procedure, the agent-based system in an AI-assisted pentest makes specific decisions independently based on the test subject. Here, the agent acts like a human tester. The agent is capable of doing this because it is trained with knowledge of vulnerability classes, technologies, and typical IT security issues, as well as common testing methods. The agent knows everything about SQL injection, command injection, and HTML injection attacks, access control and authorization checks such as IDOR/BOLA, authentication checks, and protocol- and technology-specific tests regarding, for example, OAuth, WebSockets, gRPC, or race conditions. This is precisely where the difference to a traditional scanner becomes apparent: The agent does not strictly follow these methods but selects them based on the test subject and customizes payloads accordingly. For example, if authentication based on JSON Web Tokens (JWT) is detected, the agent automatically prioritizes the JWT test suite; if it detects a file upload, the upload specialist is activated, and so on. Last but not least, limits on the agent’s actions are defined – specifically by a human security expert. The penetration tester not only defines a system prompt but also gives the agent specific instructions on how to operate. Such an instruction might, for example, involve offloading certain test steps to subagents. Another example would be an instruction specifying the exact format in which the agent should respond.

Conducting the Pentest: What can a pentest agent accomplish on its own?

Like a human tester, the agent begins with a discovery phase: It gains an overview of the application and identifies the attack surface. It considers which attack paths and vectors might be relevant and how lucrative they are based on the threat model. When test cases arise, it attempts to exploit them. When a potential vulnerability is identified, it is verified – sometimes multiple times, depending on its severity. An orchestrator distributes small work packages to specialized subagents. Subagents perform very limited, specific tasks and report their results back to the orchestrator. The orchestrator evaluates the feedback from the subagents, processes it further, and plans the next test steps. More generally speaking: The orchestrator abstracts the test steps and determines exactly when and how they are executed – just as a human penetration tester would.

Here's an example: During the discovery phase, the orchestrator creates a subagent and prompts it with something like: “You are a pentest discovery agent and are conducting an authorized penetration test against this web interface [...].” This is accompanied by instructions on how the subagent should access the system, what it should look for, and in what format it should report its results. A task might read as follows: “Examine the application, check the sitemap, analyze JavaScript, and search for links, references, and API documentation. Document everything you find in the specified format and return it.” The orchestrator receives this information in a structured format and thus knows the existing endpoints and signals. Based on this, the orchestrator derives hypotheses and plans the next test step. For example, the orchestrator specifically tasks additional subagents with checking individual endpoints for specific classes of vulnerabilities. Each subagent reports back in a structured manner on the attack surfaces and potential vulnerabilities it has identified, whereupon the orchestrator plans the next step. If the source code is also part of the task, additional source code review specialists (subagents) are brought in to identify attack surfaces at the code level.