In November 2024, the Google Zero-Day Security Research Initiative and Google DeepMind, collectively referred to as Big Sleep, achieved a significant milestone by identifying the first publicly documented Zero-Day Memory Safety vulnerability via a Large Language Model (LLM) agent [5]. This research unveiled a null-pointer dereference within SQLite, which necessitated an in-depth investigation into the vulnerability. The Big Sleep team highlighted the substantial potential of identifying software vulnerabilities preemptively, before malicious entities can exploit them, thereby emphasizing the crucial role LLM Agents have in fostering robust software systems. Moreover, fuzzing, recognized as a cybersecurity technique involving the injection of systems with invalid, malformed, or unexpected inputs to unveil security vulnerabilities, has been validated as an effective method for mitigating system weaknesses. Accordingly, our methodology incorporates a Fuzzing Agent to address genuine zero-day attacks, as elucidated in subsequent sections.

An approach that resonated with LangChain’s multi-agent designs was proposed by Zhu etal.[2]. This paper investigates how teams of LLM agents can autonomously and collaboratively exploit zero-day vulnerabilities in web applications. It proposes a hierarchical architecture, called HPTSA (Hierarchical Planning and Task-Specific Agents), which consists of:

Hierarchical Planner—Explores the target and devises a high-level strategy.Team Manager for Agents—Invokes task agent based on planner and keeps track of partial results. Task-Specific Expert Agents—Execute the exploit as assigned by the team manager, such as attempting to exploit specific vulnerabilities. Additionally, these agents have access to special tools such as Playwright for browsing.

The precision of HPTSA demonstrates a pass@5 rate of 53% and a pass@1 rate of 33.3% when utilizing GPT-4 in the absence of a vulnerability description. The framework has established an overall success rate of 18%, which translates to a cost of $24.39 per successful exploit. Consequently, multiple iterations are generally necessary to achieve a successful result, thereby increasing the overall expenditure and effort. Nevertheless, beyond the necessity for multiple iterations, the system is encumbered by additional constraints. The employment of a circumscribed number of specialized agents, developed for known vulnerability signatures, limits the range of diagnosable vulnerabilities, thereby hindering the detection of various zero-day exploits and other unidentified vulnerabilities. Furthermore, the framework was unable to exploit two vulnerabilities listed in the CVE database owing to “undiscoverable” endpoint routes, attributed to architectural limitations.

An alternative multi-agent approach employs LLM agents to generate security vulnerabilities through penetration testing while utilizing techniques such as Retrieval Augmented Generation (RAG). Shen et al.[3] presents an LLM-based penetration testing framework that automates the processes of intelligence gathering, vulnerability analysis, and exploitation stages. This framework comprises the reconnaissance agent, the search agent, the planning agent, and the execution agent, all of which execute the three stages of penetration testing. The reconnaissance agent synthesizes environmental information regarding the target host and dispatches a summary to a specific database. The search and planning agents ascertain potential vulnerabilities and devise strategies to exploit these respective weaknesses. Ultimately, the execution agent enacts the planned exploit and transmits the execution logs to the database. Theoretically, this strategy exhibits significant effectiveness and is capable of enhancing the proficiency of penetration testing. However, practical experimentation revealed several limitations. Firstly, LLM Agents occasionally necessitated supplementary knowledge to operate certain tools such as “ysoserial”, thus requiring human intervention for successful task completion. Secondly, PENTESTAGENT was unable to recognize specific software components that do not function as standalone applications, such as plugins and underlying components, specifically Nginx, PHPMailer, or Ghostscript. Lastly, LLM hallucinations can detrimentally affect system efficiency, causing infinite loops, misfired actions, and numerous errors throughout the process.

Bianou et al.[6] introduces “PENTEST-AI”, an LLM-based multi-agent system developed to automate penetration testing by utilizing the MITRE ATT&CK framework. In a manner akin to PENTESTAGENT [3], the penetration testing process is compartmentalized into a reconnaissance agent, exploit agent, and post-exploitation agent, overseen by an overarching centralized planning agent. The LLMs provide methodical reasoning, dynamically adapt to target environments, and select suitable attack techniques based on MITRE ATT&CK tactics and techniques. In contrast to PENTESTGPT [7], which functions primarily as a solitary agent and serves as an interactive LLM-driven assistant guiding users through penetration testing steps in a linear, command-response format, PENTEST-AI proposes a collaborative, autonomous multi-agent architecture conducive to more scalable, autonomous, and intricate decision-making throughout the testing process. Nonetheless, PENTEST-AI exhibits certain limitations. The system is susceptible to LLM Hallucinations, where the LLM produces inaccurate or unsafe commands. PENTEST-AI similarly contends with nuanced real-world scenarios such as custom network configurations, proprietary software, or rapidly evolving environments where meticulous human judgment remains indispensable. Furthermore, this system may falter in coordinating between multiple LLM agents and external penetration testing tools, potentially disrupting the automation pipeline.

Moskal et al.[8] also use LLM Agents, but apply it to the concept of Network Threat Testing. They developed a Docker sandbox in order to execute and evaluate code produced by LLMs. Then, prompt engineering with the priority of directing agents and execution was discussed. A GPT-3.5-Turbo powered kill chain and threat actor SDP with prompt engineering and a controller module was created to identify zero-day network exploits. Similarly to the previous two systems, there were limitations in the capacity of the LLMs. First, the LLM’s training data plays a crucial role in determining the final form of the LLM. Most of the data used to train the GPT models was from June 2021 and, in an ever-changing cyber scope, is more than likely outdated material. Furthermore, the LLMs also had a difficult time understanding the tactical stages comprising the kill chain process. The framework does not leverage background knowledge and common sense, which are inferred qualities that a threat actor possesses.

Traditional Machine Learning and Deep Learning frameworks were also implemented to identify zero-day exploits. Touré et al.[9] utilized supervised algorithms such as Decision Trees (DT), Random Forest (RF), and K-Nearest Neighbors (KNN) to detect zero days with an accuracy of 98% - 100%. The model is very precise in pinpointing anomalies among clusters and correlation tables, since it is trained using test cases on IBM and NSL-KDD data sets. Similarly, Teymourlouei et al.[10] compare two methods of zero-day attack detection. One uses a common data reduction technique to eliminate non-threats to reveal the anomalies. The other uses deep learning through a neural network with an accuracy of more than 97% and a loss of 0.1.

Table 1visualizes features, methods, and issues of previous research compared to LLMAZD. In summary, the drawbacks of previous research resonate around 1) LLM hallucination, which poses a significant threat for cybersecurity applications by generating inaccurate or misleading outputs that can compromise system reliability; sample case studies have been demonstrated in the Appendix section A; 2) Substantial context length in cyber-related tasks, which restricts the agent’s ability to fully capture the relevant context of prior team members, indirectly increasing the average cost of the solution due to the need for multiple runs to achieve successful results; 3) Coordination between agents, where ineffective handovers in multi-agent systems lead to fragmented workflows and reduced efficiency. These gaps highlight the need for improved LLM robustness, optimized context handling, and seamless agent collaboration to enhance the efficacy of AI-driven cybersecurity frameworks.

Table 1. Literature comparative analysis based on solution features and noticeable issues across the previous frameworks.

Note: Best outcome for Features—[✓], while for Persistent Issues—[×].

1) Human-in-the-Loop: Building on the idea that LLM agents are powerful but not yet perfect, we incorporated a human feedback module to monitor the agent’s interaction with its environment. This feedback module integrates a multi-function setup providing overall control to the human; executing the framework. This approach grants the executor privileged actions such as:

Approvinga specific decision path or a tool call.Reviewinga particular output of the agent. Suggestingalternate actions that can be taken.

The introduction of such privileges facilitating human oversight has significantly mitigated the risk of escalating and compounding errors within multi-agent systems, effectively addressing a critical yet often neglected issue within the field. Our research is focused on exploring the potential for automating this feedback loop through the employment of agents.

2) Decentralized Architecture Design: Multi-agent systems derive considerable advantage from direct inter-agent collaboration [11]. To harness this capability to its fullest extent, we have developed a sophisticated conversational framework facilitating inter-agent communication. By temporarily abrogating the managerial function, the system is able to efficiently process transfer requests, thereby improving context sharing and negotiation among agents, enhancing operational efficiency, and indirectly mitigating costs associated with superfluous LLM calls.

3) Pre-Loaded Context: In lieu of employing an extensive document-based knowledge base with Retrieval-Augmented Generation (RAG), we implement a more streamlined strategy by pre-loading context, akin to Cache-Augmented Generation [12]. This methodology optimizes query execution speed and accuracy through the use of a smaller, meticulously curated collection of documents. Each agent is furnished with a customized prompt, an output schema for generation, and a selective assortment of documents that provide illustrative data on various types of attacks. Although a direct implementation utilizing a KV-cache could further augment the system, as previously demonstrated in academic research, at the time of development, both LangChain and LangGraph were devoid of supporting tools necessary for this sophisticated approach.

It is acknowledged that zero-day vulnerabilities may encompass unidentified signatures, which differ from the defined signatures that are typically recognized. This characteristic renders it nearly infeasible for a team of agents, each with specialized task-oriented roles, to effectively identify them. Consequently, the approach is bifurcated into two primary components: one sub-team is dedicated to addressing existing or known vulnerability signatures similar to a process orchestrated by Zhu et al.[2] but improved substantially using novel components, while the other sub-team is tasked with detecting unknown, potentially novel vulnerabilities and exposures. An in-depth analysis of the use cases and advantages of this strategy is presented in Section 5.4. The ensuing discussion will provide a comprehensive examination of the architecture, elucidating the types of agents, the format of prompts, and the tools employed.

This part of the framework targets known vulnerability signatures. The architecture is summarized in Figure 2.

1) Generic agents:

Exploration Agent:In the context of invasive cybersecurity methodologies, reconnaissance constitutes the foundational phase, as affirmed by prior investigations conducted by other scholars. Similarly, our approach incorporates this essential step, where a planning agent plays a pivotal role in any cybersecurity endeavor. The agent’s primary function is to systematically explore all potential pathways within a specific application and identify elements that could be vulnerable to exploitation. To achieve this, the agent deploys suitable tools, such as Playwright for web browsing, shell tools, and network mapping utilities, as required. Furthermore, we employ a structured output format to verify the traversed paths and the gathered elements. This structured method facilitates the provision of a clearly delineated plan of execution to the subsequent agent.

Figure 2. Part 1—Architecture Diagram: A: represents the exploration stage (where ExplorationAgent analyzes the exposed software endpoints), B: represents the mapping stage (using the output from ExplorationAgent to map the relevant elements), and C: represents the execution staging phase and the generation of task-specific agents.

Planner:The primary function of a manager agent is to redirect queries to designated agents as determined by the exploration agent; additionally, it supplies supplementary context when required. Instead of utilizing this approach, we evaluate an alternative component known as the planner module, which systematically correlates specific elements with predefined vulnerability signatures. As illustrated in Figure 2, it efficiently maps input fields to the corresponding agent. This approach mitigates the risk of context misinterpretation for the manager when directing queries to the relevant entity.

2) Task Specific Agents: The notion of detailed prompting has facilitated the development of role-playing LLM-based agents. Through these prompts, an agent can be aligned and adapted to assume a role necessary for the completion of a real-world task. We have developed three task-specific cybersecurity agents capable of producing outputs consistent with their attack signatures.

SQLi Agent:This agent Utilizes information from “CWE-89” [13] as its base prompt, which enables the agent to generate and iterate SQL commands that are malicious in nature. Due to the complexity of the SQL commands that need to be generated, we have included additional document references mainly containing complex SQL commands for the prompt apart from the CWE resource. XSS Agent:This agent uses concepts iterated from CWE-79 [14], enabling fluency in generating malicious JavaScript code. The agent prompt template also contains supplementary materials detailing common injection vectors, DOM manipulation techniques, and browser-specific payloads to enhance effectiveness in exploiting cross-site scripting vulnerabilities. CSRF Agent:This agent utilizes CWE-352 [15] as the primary reference, enabling the agent to effectively craft and validate malicious requests aimed at exploiting vulnerabilities related to Cross-Site Request Forgery. To accurately replicate and evaluate CSRF attacks, the agent leverages supplemental documentation outlining attack vectors, defense bypass techniques, and common security pitfalls associated with session handling and request validation.

In addition, we created a general agent to provide information from an alternative perspective whenever necessary. However, during our testing, we observed that this agent was not utilized as frequently as the others.

The second part of the framework operates on both known and effectively unknown signatures, which do not have a corresponding task-specific agent. The Exploration agent, as shown in Figure 2 and described in Part 1 of the architecture, serves as a common point of contact between the two multi-agent teams. The architecture displayed in Figure 3 consists of two additional agents, which are described in the following sub-sections.

1) Fuzzing Agent: The Fuzzing Agent is invoked to examine and diagnose a comprehensive preliminary report from the Exploration Agent containing target URLs, form fields, and parameter names. For each parameter or field type, the Fuzzing Agent generates varied payloads based on categories such as random ASCII values, known injection patterns, encoded variations, and boundary tests. Each newly generated payload is guaranteed to be unique due to human oversight; additionally, the agent monitors previous queries and verifies whether the execution against a test field was successful. After compilation, the output is sent to the Execution Agent to perform an action on each payload.

Figure 3. Part 2—architecture diagram.

2) Execution Agent: The Execution Agent performs execution based on two fundamental inputs: 1) Location of the placeholder where the input has to be filled in and 2) Payload that has to be used. The agent has access to various tools like the playwright browser, shell tool, and certain custom tools. Upon a failed payload execution or unsuccessful attempt to classify a vulnerability, the agent re-queries the Fuzzing agent (under human supervision) with sufficient context until an execution is successful.

Figure 4 illustrates a summarized workflow, outlining the process from the initial execution (Step 1) to the generation of the final vulnerability report. Notably, the framework can analyze all available endpoints when instructed by the user. The workflow continues until the user explicitly directs it to stop or manually terminates the process.

Multi-agent LLM systems offer clear and structured workflows for specialized AI agents to work together efficiently while focusing on specific tasks. Individually, each agent will operate their own tailored environment consisting of prompts, tools, and decision-making logic. By breaking down tasks into smaller, manageable components, multi-LLM architectures guarantee a more reliable and precise output.

By leveraging the key components outlined in Section 3.1, the framework enhances task delegation through responsible methods and the establishment of necessary human oversight. Agents within this workflow can communicate independently, share information across various stages of the system following verification, and autonomously determine subsequent actions. Another advantage of this architecture is improved performance, achieved through parallel computing and task delegation. Utilizing methods such as parallel tool calls supported by OpenAI’s LLMs, the framework significantly reduces context usage, as supporting agents can invoke tools simultaneously without needing to report tool outputs to the LLM one by one. This approach minimizes costs by reducing context overflow, improves system efficiency, and, most importantly, prevents the framework from being hindered by LLM hallucinations.

In this investigation, we conduct an evaluation of the proposed framework for identifying vulnerabilities by implementing an optimized workflow. The procedure commences with the manual selection of a web-based CVE entry, which comprises both a hyperlink to the affected software and a comprehensive description of the identified bug. Subsequently, we emulate the software environment on a virtual machine. Where necessary, modifications are made to the software to ensure its compatibility with our framework. For instance, the Playwright tools available in the LangChain Hub lack inherent functionality for direct file uploads from the system. To overcome this limitation, we implement an alternative mechanism by integrating a placeholder that accepts the file path, subsequently rendered through the back-end. Following these adjustments, the modified software is deployed, and the exposed link on the host system is utilized as input to the framework. This framework leverages the link to interact with the web application, facilitating vulnerability assessment.

Figure 5. Snapshot of an agent output.

In order to maintain clarity and simplicity in the proof-of-concept, the framework is tested using a web-based vulnerability (CVE-2024-24945), which illustrates the potential for assessing vulnerabilities without a task-specific agent. The Travel Journal application, developed using PHP and SQL, is susceptible to a cross-site scripting (XSS) vulnerability through the write-journal.php endpoint. The experimental setup is replicated on a virtual machine and is slightly modified by altering the file upload configuration to accept the image file path from the host machine. The endpoints of the web application are exposed to the host machine and employed as an input variable for the multi-agent workflow. The agent log for execution confirmation is shown in Figure 5.

Figure 6 showcases the final vulnerability report which can be utilized later to mitigate and fix the detected vulnerabilities.

Once the execution of the multi-agent system was complete, we manually visited the read-journal.php endpoint of the web application and cross-checked using the developer tools if the framework was successful in adding the text “vulnerability detected” (displayed in Snapshot Figure 7) using the cross-site scripting attack.

Figure 6. Snapshot of the vulnerability report.

1) Exploiting SQL Vulnerability: The framework evaluated multiple open-source softwares associated with 9 CVE-IDs listed in Table 2. It exploited SQL injection vulnerabilities primarily by targeting parameter handling weaknesses on various page sources. While parameters varied across different softwares, the most commonly exploited ones were “id”, “name”, and “subject”. The framework carefully analyzed and selected appropriate parameters to minimize execution steps. Additionally, time-based payloads were consistently used due to the ease of exploit confirmation.

2) Exploiting Cross-Site Scripting Vulnerability: We tested the framework against 8 CVE-IDs listed in Table 3 and found that the framework effectively exploited endpoint parameters by injecting malicious payloads. At times, testing required multiple iterations as identifying the correct parameter required variable form submissions.

3) Identifying File Upload(or Unrestricted Upload)Vulnerability: During routine analysis of various software, the fuzzing team flagged multiple input endpoints that accepted files as input and simultaneously hypothesized that these endpoints were vulnerable to unrestricted upload attacks. Upon testing, the executor agent discovered that it was possible to pass different file types to the endpoint, including malware script files, thereby confirming the presence of file upload vulnerabilities. The results are mentioned in Table 4.

4)Exploiting CSRF Vulnerability: The framework identified multiple CSRF vulnerabilities in different software applications (listed in Table 4) by analyzing authorization mechanisms on critical endpoints. The framework generated HTML files containing endpoint information along with a submit button. These exploits leveraged browser cookies and active sessions to bypass proper authorization controls.

5) Exploiting File Inclusion Vulnerability: The framework successfully identified local file inclusion vulnerabilities by testing various endpoint parameters for improper file handling. In all cases (listed in Table 4), the vulnerabilities were discovered through the manipulation of the “page” parameter associated with random pages, which allowed unauthorized access to server-side files. The framework demonstrated efficiency in detecting these vulnerabilities, with exploitation requiring between 5 to 10 steps. The framework systematically tested for file inclusion by attempting to retrieve sensitive files through parameter manipulation, successfully confirming the presence of these vulnerabilities when the source code of index.php was exposed.

6)Detecting Database Exposure Bugs: The fuzzing team revealed a database exposure vulnerability in Alton Systems (Table 4) during endpoint exploration. While systematically probing various endpoints, the framework discovered that full database schema information was inadvertently exposed, requiring minimal steps to uncover. This vulnerability, identified during development phase testing, demonstrated the framework’s capability to detect information disclosure issues that could pose significant risks if left unaddressed in production environments.

To evaluate generalizability of the proposed framework, we conducted comprehensive testing on more than 20 publicly available (open-source) CVE associated softwares. The results displayed in tables of Section 5 demonstrate the effectiveness of the framework to identify different classes of vulnerabilities, which include 1) SQL Injection, 2) Cross-Site Scripting (XSS), 3) Cross-Site Request Forgery (CSRF), 4) Database Exposure, 5) Unrestricted File Uploads, and 6) File Inclusion. Furthermore, as detailed in Section 3.2 and 3.3-1, while the fuzzing agent is capable of exhaustive testing, the granular implementation of the architecture allows for seamless addition of multiple agents to detect new vulnerability categories as needed, provided sufficient resources are readily available.

The methodology and novel components enable the framework to execute and follow user requests effectively, operating in two fundamental modes.

1) Full Screening: By default, the framework conducts a comprehensive search to uncover unknown vulnerabilities and bugs. Because testers (human-in-the-loop) generally cannot predict where or what an attack signature will look like (before exploration), the framework systematically explores and analyzes potential vectors to generate actionable suggestions. An example of this process is shown in Figure 10. Based on these reports, the user can review and authorize the framework to execute the recommended steps. This methodology exhaustively tests all available endpoints and attack signatures. Although full screening may increase runtime and cost, tables in Section 5 show that the additional expense is negligible, even a 100 exploration steps incur only about $0.12. This workflow provides an effective means of uncovering zero-day vulnerabilities and software exploits.

Table 3.Results for Cross-Site Scripting (XSS) Vulnerabilities.

Note: *Novel vulnerability detected using the framework (not present in the CVE database).

2) Targeted Discovery for Specific Endpoints or Signatures: The framework also supports a targeted mode for validating a specific endpoint or attack signature. While the earlier provided snapshots illustrate a zero-day scenario, a penetration tester who wishes to perform a one-day test can instruct the framework to skip the exploratory phase and immediately deploy the relevant agents against the designated endpoint accompanied by a vulnerability signature pattern. Consider an example displayed in Figure 8, the exploration agent reduces the exploration to 1 - 2 steps, and directly provides elements and the context to the next agent. This targeted approach delivers a rapid and efficient verification process.

Figure 8. Snapshot of targeted mode, where the Exploration Agent maps the input and hands off to the next agent.

The human component, though reliant on the tester’s understanding of attacks, still demonstrates exceptional execution of the framework, even when only basic logic is applied. Consider the scenario where the agent requires the user to approve its output before proceeding to the next step. As shown in Figure 9, the tester, upon verifying the output, decides to explore the first mentioned redirect link, prompting the agent to perform an additional iteration to log the information present on the requested endpoint. A similar execution, without the human feedback, would have resulted in the agent either exiting the workflow or providing vague outputs to the subsequent agent.

Once a plausible endpoint is identified, the user can request the framework to generate an action plan, which can be verified before execution. Figure 10 illustrates a sample plan being generated. A second case study demonstrates the utility of the tester providing a review (correction) of the agent’s output. Figure 11 displays a case where the generated XSS payload (1st highlight) upon execution would not have alerted the multi-agent system due to the inaccessibility of the appropriate tool in LangChain Hub to record browser pop-ups. Understanding this limitation, the human provides a correction to the payload to display sample text instead of executing an alert. Subsequently, a third case demonstrates the ability to provide suggestions to the agent. In the output displayed in Figure 12, the agent was unable to click the “Save Journal” button for an unknown reason. Consequently, it could not execute the payload. The tester, knowing that the action of the “Save Journal” button was to execute the endpoint /add-journal.php (as displayed in the agent output in Figure 9 in the second highlight box), proceeds to suggest an alternative method of utilizing the shell tool on that endpoint, as shown in Figure 12. A successful execution of the tool call is presented in Figure 5, along with the final agent output.

Table 4. Results for other vulnerabilities (CSRF, File Upload, File Inclusion, Database Exposure).

Note: *Novel vulnerability detected using the framework (not present in the CVE database). **Database exposure bug identified during development, posing a potential risk in production if unaddressed.

We evaluated a basic methodology employing a multi-agent team of three agents, without introducing any novel components. The agents’ prompts were modeled after LLMAZD, with all references to a human-in-the-loop removed; otherwise, they remained unchanged. Multiple tests failed to detect any vulnerabilities within the specified target domain. Additionally, the agents ran unsupervised, leading to increased token usage. In contrast, HPTSA [2], a fully autonomous framework, was able to exploit the tested vulnerability, but only after multiple runs with specialized prompts and carefully designed initial instructions, achieving a success rate of just 18%. Even at this rate, the system was prone to various errors. We present several execution instances to illustrate the failure points of a fully autonomous system.

1) Case1:Escalating Errors: We initialized the conversation with the host webapp’s endpoint, but the exploration agent in the multi-agent system failed to retrieve the basic schema of the web page as shown in Figure A1.

Figure A1. Case 1: The Exploration Agent fails to retrieve the page schema, leaving the Fuzzer Agent with no fields to target.

A few steps later, the executor agent successfully extracted the onclick links from the hosted software’s homepage (Figure A2), which redirected to the write-journal.php page, but the subsequent agent did not follow that link.

Figure A2. Case 1 (continued): onclick links are extracted but never followed by the next agent.

By the 10th step, the token count had already increased substantially. As a result, the exploration agent appears to have drifted into the host directory (macos.local/) instead of continuing toward the required endpoint, referring to Figure A3. After this broad navigation, we manually terminated the execution.

Figure A3. Case 1 (continued): at step 10, growing context pushes the Exploration Agent off target and into the host directory.

Figure A4. Case 2: the agents falsely claim a successful form submission that never occurred.

2) Case2:False Submission Claim and Error based Exit: After several runs, the fully autonomous methodology still failed to reach the designated endpoint to test the vulnerability. Therefore, we modified the initial message to point directly to the write-journal.php endpoint. The agents, this time, parsed the website correctly (as indicated in Figure A4) and completed one execution loop, but it produced no successful outcome.

Since the prompt only specified the write-journal.php endpoint, the agents didn’t consider visiting the read-journal.php endpoint. Manual analysis of the database confirmed there were no form submissions, proving the agents’ claim of having submitted an entry to be false. During the live run, we allowed the agents to continue without manual intervention. After a few exchanges, one agent returned multi-tool output in an incorrect format, causing an error (referring to Figure A5) and terminating the execution.

3)Case3:Unnecessary Context overflow: After explicitly specifying the write-journal.php endpoint, we ran the base implementation multiple times to test whether the multi-agent system could successfully identify the vulnerability. All attempts failed; in one run, as demonstrated in Figure A6, the agents repeatedly rediscovered the same endpoints and exploration details without making progress. The agents again falsely claimed to have submitted the form; our manual checks confirmed no new entries in the database.

Figure A5. Snapshot of an error displayed on the terminal.

Figure A6. Case 3: the agents loop over the same endpoints and again falsely claim to have submitted the form.