Author: suvroc

  • AGENTIC – AI Agents threat modeling

    AGENTIC – AI Agents threat modeling

    History

    There are already several AI threat modeling methodologies available on the market. However, they all suffer from two problems:

    • Many of them are designed exclusively for LLMs, while today the greatest security risks lie not in the language model itself, but in Agentic AI. Especially in its autonomous decision-making, access to tools, persistent memory, and self-modification capabilities.
    • The only framework that currently addresses Agentic AI comprehensively (MAESTRO) is very complex and requires significant technical expertise to apply effectively.

    This creates an unnecessary barrier. As AI agents become increasingly common, threat modeling must become more accessible so that even professionals without deep AI security expertise can independently assess the risks of agentic systems.

    Goal

    AGENTIC was designed with four primary objectives:

    • Simple by design – applicable without hours of training or extensive preparation.
    • Intuitive perspective – questions formulated in a way that can be understood without prior AI security knowledge.
    • Designed for Agentic AI – unlike frameworks built primarily for LLMs, AGENTIC addresses the unique characteristics of autonomous AI agents.
    • A complement, not a replacement – AGENTIC serves as an entry point that can naturally lead to deeper, more technical frameworks whenever additional analysis is required (see the Comparison with Other Frameworks section).

    Target Audience

    The purpose of this methodology is to simplify threat modeling for AI agents, currently the highest-risk form of AI deployment. Thanks to this simplification, AGENTIC can be used not only by AI security specialists, but also by:

    • Security professionals – enabling them to assess the security of agentic systems even without deep expertise in AI security.
    • Software developers and engineers building AI agents – who need a quick security assessment before deploying to production.
    • Security Champions within product teams – technical team members who are not dedicated security specialists but serve as the first line of security review.
    • Professionals responsible for AI agent adoption – who need a common language for discussing “what could go wrong” with technical teams.

    Elements

    ElementDescriptionQuestion
    AAutonomyThe agent’s ability to make decisions and perform actions without human involvement.How much can the agent do before anyone notices?
    GGroundingHow the agent connects its reasoning and actions to the real world.How does the agent know that what it “knows” is actually true?
    EEnvironmentThe execution environment and operational permissions available to the agent.What can the agent execute or modify?
    NNeural Memory / KnowledgeEverything the agent stores and uses as knowledge, including both short- and long-term memory.What does the agent remember?
    TToolsExternal capabilities available to the agent.What is the worst thing the agent could do by combining its available tools?
    IIterative Self-ModificationThe agent’s ability to change its own behavior over time.How can the agent change itself?
    CControlThe human ability to supervise and intervene when necessary.Can anyone stop it before it’s too late?

    A – Autonomy

    Description

    Autonomy defines the extent to which an AI agent can independently make decisions, plan actions, and execute tasks without requiring human approval.

    Detailed Description

    Autonomy can take many different forms, as shown in the table below. Moreover, different tasks performed by the same AI agent may operate under different autonomy levels.

    LevelDescription
    L0 – Human-AuthorizedAI proposes; the human performs the action.
    L1 – Human-ApprovedAI performs the action, but only after receiving human approval.
    L2 – Human-MonitoredAI acts autonomously while a human continuously monitors and can interrupt execution.
    L3 – Human-on-the-LoopAI acts autonomously with periodic or sample-based human supervision.
    L4 – Fully AutonomousAI operates entirely without real-time human oversight.

    Questions to Ask During the Analysis

    • How much can the agent do before anyone notices?
    • Which decisions can the agent make without human approval?
    • Is the autonomy level consistent across all tasks, or does it differ between, for example, read and write operations?
    • Is autonomy static, or can it change dynamically based on the system’s own risk assessment?
    • Can the agent recognize when it is exceeding its intended authority? If so, what does it do?
    • What is the worst realistic scenario in which the agent fully exploits its granted autonomy—whether due to an error or malicious intent by the person controlling it?

    Example Threats

    • The agent performs an irreversible action (sending an email, deleting a record, executing a financial transaction) without human confirmation.
    • Goal drift – the agent pursues the objective in a way that differs from the designer’s original intent (for example, optimizing a metric rather than the intended business outcome).
    • Autonomous privilege escalation in an attempt to accomplish its objective more effectively.

    Example Mitigations

    • Human-in-the-loop for high-impact actions, with approval thresholds based on business impact.
    • Scoped goals, where objectives are narrowly defined instead of being open-ended.
    • Comprehensive decision logging with full auditability and the ability to reconstruct why each decision was made.

    G – Grounding

    Description

    Grounding describes how an AI agent connects its reasoning and actions to the real world. It covers the data sources the agent relies on, retrieval mechanisms (such as RAG), and the freshness and reliability of the knowledge used to make decisions.

    Detailed Description

    Grounding defines the relationship between the agent’s internal model of the world and reality. It includes both static knowledge acquired during model training and dynamic knowledge obtained through retrieval systems, search engines, APIs, and external data sources. It also encompasses the mechanisms used to verify that knowledge.

    Grounding may exist at different maturity levels:

    • No grounding – the agent relies solely on knowledge acquired during training, with no access to current information.
    • Single-source grounding – the agent retrieves information from one source without validating it.
    • Multi-source grounding – the agent combines information from multiple sources but performs no formal trust assessment.
    • Verified grounding – the agent evaluates source credibility and can explicitly communicate uncertainty.

    Questions to Ask During the Analysis

    • How does the agent know that what it “knows” is actually true?
    • Which data sources are considered trusted, and which are considered untrusted? Does the agent distinguish between them?
    • What happens when different data sources contradict one another?
    • Does the agent express uncertainty (“I’m not sure”), or does it always present answers with equal confidence?
    • How frequently is the knowledge used by the agent refreshed?
    • Does the agent distinguish between user-provided data, system data, and content retrieved from external sources?

    Example Threats

    • A hallucinated response is treated as fact and becomes the basis for a real-world action.
    • Indirect Prompt Injection through an external document or webpage retrieved during the RAG process.
    • Data/RAG poisoning, where the agent’s knowledge source has been intentionally manipulated.
    • The agent acts on outdated information without detecting inconsistencies with the real world.

    Example Mitigations

    • Instruction/data separation to prevent retrieved content from being interpreted as instructions.
    • Trust boundaries, assigning different trust levels to different information sources.
    • Cross-checking or fact verification before performing high-impact actions.

    E – Environment

    Description

    Environment refers to the execution environment of the AI agent and the permissions it possesses within that environment, including the operating system, network, infrastructure, and the logical or physical resources accessible to the agent.

    Detailed Description

    Environment represents the infrastructure layer of the system. It includes the operating system, containers, networks, storage, databases, internal APIs, and any resources the agent can actually reach.

    Typical deployment models include:

    • Fully isolated environment – sandboxed with no network connectivity.
    • Partially isolated environment – restricted access to selected resources.
    • Shared environment – resources shared with other systems or users.

    The execution environment largely determines the risk of lateral movement and the level of monitoring required.

    Questions to Ask During the Analysis

    • What can the agent execute or modify?
    • What capabilities does the agent have within the system? Can it read files, execute system commands, or access the Internet?
    • Which specific resources (data, systems, accounts) are technically reachable from the agent’s environment?
    • Could the agent theoretically access resources belonging to another team or customer?
    • Does the environment have Internet access? If so, is it restricted to approved domains?
    • Which logs are generated during the agent’s execution, and how long are they retained?
    • Can the agent independently install software or download additional dependencies?

    Example Threats

    • The agent is granted excessive system privileges (for example, production access instead of a sandbox).
    • Lateral movement, where access to one resource enables access to additional systems.
    • Exposure of secrets or API keys due to insufficient isolation.

    Example Mitigations

    • Sandboxing or containerization of the execution environment.
    • Domain allowlisting for outbound network connections.
    • Resource limits such as rate limiting, quotas, and cost budgets.

    N – Neural Memory / Knowledge

    Description

    Neural Memory / Knowledge covers everything the agent stores and uses as knowledge during its operation. This includes both short-term memory (within a session) and long-term memory (persistent knowledge retained across sessions).

    Detailed Description

    A key distinction exists between short-term and long-term memory.

    Short-term memory is discarded once a session or task ends.

    Long-term memory persists between sessions and may include user profiles, previous interactions, learned preferences, or historical decisions.

    Possible implementations include:

    • No persistent memory – every interaction starts from scratch.
    • Read-only persistent memory – stored knowledge cannot be modified by the agent.
    • Agent-managed persistent memory – the agent writes information without human review.
    • Shared memory across multiple agents or users.

    Questions to Ask During the Analysis

    • What does the agent remember?
    • What information is stored in long-term memory, and under what conditions?
    • How long is information retained?
    • Can users easily inspect what the agent remembers about them or the organization?
    • Is there a mechanism to remove specific memory entries upon request?
    • Is one user’s memory fully isolated from another’s?
    • Is persistent memory used to personalize high-risk decisions, such as financial or medical recommendations?
    • Is stored information classified according to sensitivity, or is everything treated equally?

    Example Threats

    • Memory poisoning, where malicious information is injected into persistent memory and influences future behavior.
    • Cross-user data leakage, exposing one user’s memory to another.
    • Failure to support the right to be forgotten, leaving sensitive information permanently stored.

    Example Mitigations

    • Encryption and per-user or per-session memory isolation.
    • Validation of information before it is written to persistent memory.
    • Time-to-live (TTL) and automatic expiration of stored information.
    • Human-review capabilities for inspecting, auditing, and deleting memory entries.

    T – Tools

    Description

    Tools represent the external capabilities available to an AI agent, including APIs, functions, integrations (such as function calling or MCP servers), and other mechanisms that allow the agent to interact with the outside world.

    Detailed Description

    Tools are more than simply a list of available APIs. They also include the way these APIs are invoked—for example, whether parameters are validated by the language model itself or by an independent middleware layer.

    Another important consideration is tool composition. A single tool may appear harmless on its own, yet multiple tools combined can provide capabilities that none of them possesses individually. For example:

    • Read customer data + Send email = potential data exfiltration.

    Typical levels of tool capability include:

    • No tools – the agent only generates text.
    • Read-only tools – the agent can retrieve information but cannot modify it.
    • Read/write tools – the agent can create or modify external data.
    • High-impact action tools – the agent can perform irreversible operations.

    Questions to Ask During the Analysis

    • What is the worst thing the agent could do by combining its available tools?
    • What actions can the AI agent perform in the external world?
    • Which combinations of tools could provide capabilities beyond their intended purpose?
    • Who validates the parameters passed to each tool—the language model or an independent middleware layer?
    • Is the provider of every tool, especially external services such as MCP servers, trusted and verified?
    • Does the agent have access to tools that are never actually required in normal operation?
    • Are tools protected by rate limiting or other safeguards against abuse and execution loops?
    • Do the tools enforce authorization themselves, or is the agent responsible for deciding whether it has permission to access specific data?

    Example Threats

    • Tool chaining, where individually harmless tools become dangerous when combined.
    • Invocation of a tool with unvalidated parameters.
    • A malicious external tool (for example, a compromised MCP server or supply chain attack).

    Example Mitigations

    • Tool allowlisting with fine-grained permissions.
    • Least privilege for every tool integration.
    • Validation of inputs and outputs at every tool boundary.
    • Supply chain security practices for external integrations, including verification of MCP servers, plugins, and third-party services.

    I – Iterative Self-Modification

    Description

    Iterative Self-Modification describes an AI agent’s ability to change its own behavior over time—for example by modifying prompts, configuration, execution strategies, or even the underlying model based on feedback, learning, or self-reflection.

    Detailed Description

    Self-modification exists across a broad spectrum, ranging from simple prompt adjustments to online fine-tuning of foundation models or the creation of entirely new sub-agents.

    Typical levels include:

    • No self-modification – the agent behaves exactly as originally configured.
    • Controlled self-modification – the agent adapts only within predefined and approved mechanisms (for example, adjusting thresholds).
    • Advanced self-modification – the agent can modify components that are difficult to supervise, such as dynamically fine-tuning models.
    • Autonomous expansion – the agent can create new sub-agents or tools independently.

    Questions to Ask During the Analysis

    • How can the agent change itself?
    • Can the agent inspect its own configuration or execution logic in ways that could help it bypass imposed restrictions?
    • Can the agent modify its own configuration?
    • Which aspects of the agent’s behavior can change autonomously, and which remain immutable?
    • Can the agent create new sub-agents or autonomous processes without operator awareness?
    • Is there a distinction between changes intentionally initiated by humans and those initiated automatically by the system?
    • How quickly can behavioral changes made by one agent propagate to other agents or instances sharing the same configuration?

    Example Threats

    • The agent modifies its own system prompt to bypass restrictions.
    • Reward hacking, where the agent optimizes for the measured metric instead of the designer’s actual objective.
    • Unauthorized self-replication or autonomous creation of new sub-agents.

    Example Mitigations

    • Version control for prompts and configurations with rollback capability.
    • Immutable core instructions that cannot be modified by the agent itself.
    • Requiring human approval before new sub-agents can be created.

    C – Control

    Description

    Control represents the ability of humans to supervise an AI agent and intervene effectively whenever necessary.

    Detailed Description

    Control consists of two complementary dimensions that are often confused:

    • Observability – the ability to understand what the agent is doing.
    • Actionability – the ability to intervene and stop the agent when necessary.

    A system may provide excellent monitoring while lacking any practical way to interrupt execution. Conversely, a theoretical kill switch is of little value if operators cannot recognize when it should be used.

    Typical maturity levels include:

    • No monitoring and no shutdown mechanism.
    • Passive monitoring (logs only) with no practical way to intervene quickly.
    • Active monitoring with alerting but slow, manual intervention procedures.
    • Full real-time supervision with an independent, immediate kill switch.

    Questions to Ask During the Analysis

    • Can anyone stop it before it’s too late?
    • How quickly can a human detect undesirable behavior, and is there a practical mechanism to stop the agent immediately?
    • Has the shutdown mechanism ever been tested, or does it exist only on paper?
    • Does monitoring include the agent’s internal plans and reasoning process (where technically available), or only the final outcomes of its actions?
    • How long would it realistically take to detect abnormal behavior after it begins?

    Example Threats

    • No functioning kill switch, or one controlled by the agent itself.
    • Delayed anomaly detection caused by insufficient monitoring.
    • Alert fatigue resulting in operators ignoring warning signals.

    Example Mitigations

    • An architecturally independent kill switch.
    • Continuous monitoring and alerting for suspicious agent behavior.

    Threat Modeling Process

    1. Answer each question in sequence For each of the seven AGENTIC elements, first answer the primary diagnostic question from the main table, followed by the detailed questions for that element.
      The recommended order is: A → G → E → N → T → I → C
      Each element builds upon information gathered in the previous steps, helping reduce the risk of overlooking important dependencies.
    2. Document every answer Every response should be recorded. Together, these answers become the threat model of the entire agentic system.
    3. Identify the weakest point within each element After completing all questions for a particular element, identify the area of greatest concern, for example, a missing control, an unclear responsibility, or an overly permissive capability. These become candidates for high-risk findings.
    4. Continuously identify risks As you progress through the remaining elements, revisit previously identified weaknesses and consider what additional risks they introduce elsewhere in the architecture.

    Comparison with Other Frameworks

    AGENTIC was not created in isolation. It integrates well with other threat modeling frameworks for AI systems.

    FrameworkScope / FocusConnection with AGENTICLimitations
    PHANTOM-B (Shostack + Associates)A lightweight framework focused on LLM threats.Both methods provide a simple entry point for different types of systems. PHANTOM-B is designed for LLM-based deployments, while AGENTIC focuses on systems with real autonomy. Both can be used together depending on the type of project.Focused on LLM deployments rather than dimensions specific to full agentic autonomy.
    MAESTRO (Cloud Security Alliance)A framework focused on multi-agent system architecture, divided into 7 comprehensive layers.AGENTIC serves as a bridge toward this more comprehensive approach. MAESTRO is a good fit for AI security specialists.Requires modeling the complete layered architecture and significant technical knowledge of the solution. It is less suitable for agents created inside Agentic AI tools. High entry barrier.
    STRIDE-AI (Tsafac Nkombong & Schwarz, 2026)A methodology focused on LLM threats. It divides analysis into 5 system layers.STRIDE-AI can help focus on specific threat categories when combined with the AGENTIC methodology.The authors explicitly acknowledge that the framework is “currently optimized for LLMs” and does not yet cover full agentic autonomy or multi-step actions in the real world.
    MITRE ATLASThe most detailed and continuously updated taxonomy of specific techniques, including an increasing number of techniques dedicated to agents.MITRE ATLAS provides concrete techniques for each AGENTIC element. It can be used to deepen the analysis.It is a threat-side taxonomy, not a modeling methodology. It does not define where to start when analyzing a specific system.
    OWASP LLM Top 10A widely recognized common language with product teams, focused on LLM-related threats.When working with LLM-based systems, this classification allows more detailed analysis of LLM-specific risks. LLM Top 10 fits well as an extension of PHANTOM-B.Provides limited coverage of agent behavior in the real world (tools, persistent memory, autonomy) — it was primarily designed for LLM-based systems.
    OWASP Top 10 for Agentic ApplicationsFocused on agentic applications. A current framework with precisely defined risk categories.Complements AGENTIC. OWASP Top 10 for Agentic Applications can help focus on specific risk categories.A risk list (what can go wrong), not a questioning process — it does not guide users step-by-step through analyzing a specific architecture.
    SAILORS (Vinaya Vasudevan)Focused on security controls. Analysis is performed for each AI capability.Works well as an implementation checklist to ensure important AI agent security aspects are considered.Focused on individual AI capabilities rather than modeling the complete behavior of an autonomous agent over time. Focused on security controls rather than threat identification.

    What AGENTIC Does Not Cover

    • Threats at the model/training level — attacks against the training process, weight extraction, membership inference, adversarial examples targeting neural network architecture. These areas belong to frameworks such as MITRE ATLAS in its full scope, rather than AGENTIC.
    • Deep analysis of individual prompts/interactions — AGENTIC operates at the architecture and agent configuration level, not at the level of individual conversations. For this purpose, OWASP LLM Top 10 applied directly to the LLM layer is a better fit.
    • Regulatory compliance and governance requirements — the model was not designed as a compliance checklist for AI Act, ISO/IEC 42001, or similar regulations. Compliance requirements themselves define requirements that must be addressed by the AI agent.
    • Subjectivity of assessments — as with every qualitative model, risk matrix results depend on the knowledge and experience of the person performing the modeling. AGENTIC does not eliminate this subjectivity; it only provides structure for the analysis.

    References

    Comparative Frameworks

    Comparative Frameworks (continued)

    Threat Catalogs and Taxonomies

    Governance Standards and Frameworks

  • AGENTIC Whitepaper – practical threat modeling framework for Agentic AI systems

    AGENTIC Whitepaper – practical threat modeling framework for Agentic AI systems

    Background

    Many AI threat modeling methods already exist, but two systematic gaps keep showing up. First, most of them were designed purely for LLMs, while the greatest risk today lies in Agentic AI, not in the language model alone: autonomous action, tool access, persistent memory, and self-modification. Second, the one framework that genuinely fits Agentic AI Agents well, MAESTRO, is too complex and demands deep technical expertise to apply.

    This is an unnecessary barrier. As AI agents become mainstream, the discipline of threat modeling them needs to be simplified so that people without a specialized security background can model agent risk on their own. AGENTIC was built to close that gap.

    Purpose

    • Simple: applicable without a lengthy onboarding process.
    • Intuitive: questions phrased so they are understandable without prior AI Security knowledge.
    • Built for Agentic AI: unlike frameworks designed for LLMs, it addresses the specifics of agent autonomy and action.
    • A complement, not a replacement: AGENTIC is meant as an entry point, after which teams can move to deeper, more technical frameworks when needed (see the comparison below).

    Target audience

    AGENTIC lowers the barrier to modeling threats for Agentic AI – the highest-risk category of AI use – so the method is accessible not only to AI security specialists, but also to:

    • Security professionals who need to assess agentic solutions without deep prior expertise in AI security.
    • Developers/engineers building an agent who need a fast pre-production sanity check.
    • “Security champions” on product teams – technical people who are not security specialists but act as the first line of risk assessment.
    • Anyone responsible for deploying AI agents who needs a shared, non-technical language to discuss “what could go wrong” with a technical team.

    The seven elements

    AGENTIC structures the analysis around seven elements, each anchored by a single guiding question that keeps the discussion intuitive and non-technical.

    ElementWhat it coversGuiding question
    AAutonomyHow much the agent decides and executes without human approvalHow much can the agent do before anyone notices?
    GGroundingHow the agent ties its outputs to real-world truth and data sourcesHow does the agent know that what it “knows” is true?
    EEnvironmentThe execution environment and the scope of access/permissions it grantsWhat can the agent run or change?
    NNeural Memory / KnowledgeWhat the agent stores and reuses as knowledge, short- and long-termWhat does the agent remember between sessions?
    TToolsExternal capabilities the agent can invoke (APIs, functions, MCP)What is the worst thing the agent could do by combining its tools?
    IIterative self-modificationThe agent’s ability to change its own behavior over timeHow can the agent modify itself?
    CControlHuman ability to oversee the agent and intervene when neededCan anyone shut it down before it’s too late?

    Applying the model

    The modeling process follows four steps:

    • Work through each of the seven elements in order (A → G → E → N → T → I → C), answering first the main diagnostic question, then the deeper follow-up questions. This order limits the risk of missing dependencies, since each element partly builds on the answers from the previous ones.
    • Write every answer down; together they form a model of the entire agentic system.
    • For each element, once all questions are answered, identify the single weakest point: a missing mechanism, unclear ownership, which becomes a candidate for high risk.
    • While analyzing each subsequent element, actively question it against the risks already surfaced in earlier ones.

    What AGENTIC does not cover

    AGENTIC is deliberately scoped to agent-level architecture and behavior, so several adjacent problem areas are intentionally left to other frameworks and processes: 

    • Model- and training-level threats – attacks on the training process, weight extraction, membership inference, adversarial examples at the network-architecture level. This is the domain of frameworks such as MITRE ATLAS in its full scope, not AGENTIC.
    • Deep analysis of a single prompt or interaction – AGENTIC operates at the level of agent architecture and configuration, not individual dialogue turns. OWASP LLM Top 10, applied directly at the LLM layer, is better suited for that.
    • Regulatory compliance – AGENTIC is not designed as a compliance checklist for the EU AI Act, ISO/IEC 42001, or similar regulations; compliance requirements define their own set of demands on an AI agent.

    Elimination of subjectivity – like any qualitative model, AGENTIC’s results depend on the knowledge and experience of the person modeling. AGENTIC does not remove that subjectivity, only structures it.

    Comparison with other frameworks

    AGENTIC was not built in a vacuum. It is designed to combine well with existing frameworks rather than compete with them.

    FrameworkFocusRelationship to AGENTICLimitations
    PHANTOM-BLightweight framework focused on LLM deploymentsA peer entry point: PHANTOM-B for LLM deployments, AGENTIC for systems with real autonomyNot built for covering full agentic autonomy
    MAESTROComprehensive 7-layer architecture framework for multi-agent systemsAGENTIC acts as a bridge toward this deeper method for AI security specialistsRequires mapping the full layered architecture and strong technical background; high entry barrier
    STRIDE-AILLM-focused method analyzing 5 system layers and 6 threats categoriesCan sharpen the focus on specific threat classes once combined with AGENTICAuthors state it is currently optimized for LLMs, not full agentic autonomy or multi-step real-world action
    MITRE ATLASContinuously updated taxonomy of concrete adversarial techniquesSupplies concrete techniques for each AGENTIC element to deepen the analysisA threat dictionary, not a modeling method. It doesn’t tell you where to start
    OWASP LLM Top 10Widely known LLM threat classification, shared language with product teamsA natural extension of PHANTOM-B for LLM-specific workWeak coverage of the agent’s real-world action layer (tools, persistent memory, autonomy)
    OWASP Agentic Top 10Risk catalogue for agentic applications with precisely named categoriesComplements AGENTIC by sharpening focus on specific threat classesA list of risks, not a step-by-step process through a specific architecture
    SAILORSMitigation-focused checklist evaluated per capabilityWorks well as an implementation checklist alongside AGENTICFocused on a single AI capability, not on modeling autonomous behavior over time

    Summary

    AGENTIC gives teams a fast, language-accessible way to reason about where an AI agent’s real risk lives and a clear path to deeper frameworks once a system’s risk profile calls for it.

  • Prompt Engineering for LLMs

    Prompt Engineering for LLMs

    “Prompt Engineering” is a book that clearly communicates its target audience from the very beginning. And that’s a good thing, because this is not a book for a casual chat user who simply wants to “talk better with AI.” It is primarily aimed at developers and prompt engineers who actually build on top of models and need to understand how they work, their limitations, and best practices in order to consistently create better prompts. A regular user will quickly feel overwhelmed—and understandably so, since part of the material requires solid familiarity with LLMs, including at the API level.

    Let’s start with the less pleasant aspects: there are some language errors, occasionally distracting. Some sections feel overcomplicated and unnecessarily stretched. There are also surprising phrases, such as “LLM common sense,” which do not entirely fit—although I understand the author’s intention.

    However, once you move past these weaker moments, it becomes clear that the authors follow a thoughtful and transparent approach. They discuss various techniques, presenting both their advantages and drawbacks, which helps the reader understand exactly where specific recommendations come from. This builds real awareness and competence—not just “apply this rule,” but why you are applying it.

    A major strength lies in the visual examples—many discussed concepts are supported by graphical representations. And it works: ideas that initially sound abstract suddenly become intuitive. Additionally, there are exercises that force you to think and help structure your knowledge.

    I also appreciate the authors’ approach: before building prompts, you first need to understand how an LLM works. Why it behaves the way it does. Where incorrect answers come from. How tokenization connects to prompt precision. It may sound academic, but in practice it significantly improves later prompt work. The long introduction may feel overwhelming at first—but I know that pain, as I encounter the same reaction in my own training sessions.

    Some elements are genuinely fresh—for example, dynamic construction of system prompts. Honestly, I have not seen this framed in quite this way in other publications.

    A big plus for the research references, which allow readers to verify sources and dive deeper into specific topics.

    Unfortunately, parts of the book are already outdated. There is a strong focus on GPT-3, so some details are no longer current. You will find recommendations that no longer make sense today (e.g., regarding “echo”). On the other hand, this also highlights which concepts have proven timeless. For instance, while the description of tool calling does not mention MCP, it explains the underlying protocol mechanics quite accurately. So even though the book refers to older technologies, it does not lose its practical relevance.

    One final observation: many introduced terms initially sound unfamiliar, but this is largely due to the Polish translation. The English equivalents feel much more natural.

    Summary

    It is an uneven book—sometimes overly detailed and stretched—but at the same time full of concrete, practical, and well-explained guidance. If you build prompts professionally, it will give you solid foundations. If you use LLMs occasionally, this is not the book for you. For engineers, however—it is worth it.

  • AI Engineering. Building Applications with Foundation Models

    AI Engineering. Building Applications with Foundation Models

    “AI Engineering” is a book that combines technical precision with a practical approach to implementing artificial intelligence in organizations. The author doesn’t just describe processes—she defines them. This is not another book about AI trends, but a guide to the real mechanisms that turn a concept into a working system.

    Particular recognition is due to its comprehensive approach to AI implementation in companies. The book demonstrates how many dimensions are involved in the decision to integrate AI—from technical aspects, through organizational ones, to strategic and financial considerations. It reflects a deep understanding that AI is not just a technology, but a tool for building competitive advantage—and that every advantage comes at a cost.

    One of the book’s strengths is that it is “actionable.” It is difficult to assess whether the proposed processes are the best possible ones, but outlining them alone provides enormous value. They offer a starting point—especially for those who want to approach AI implementation in a structured way rather than navigating it blindly.

    The author does not shy away from details and complexity. She presents multiple perspectives and scenarios, at times almost too meticulously. In practice, however, this is an advantage—even if not every element will be relevant to everyone, the richness of examples and threads makes the book worth revisiting, depending on current challenges.

    For me, as someone who delivers AI training, this book was a source of inspiration and concrete examples. Many of the described cases can be immediately transferred into educational or project contexts.

    In terms of style, the book is surprisingly engaging. The chapters draw the reader in thanks to numerous references and concise summaries. The author manages to capture in just a few pages what other publications stretch across hundreds.

    There are moments when the narrative shifts from broader topics into highly technical or mathematical territory—which may be challenging for less technical readers. This is definitely not a “one-evening” read. It requires focus and reflection, but rewards the effort with substantial depth.

    Special recognition also goes to the transparency of sources—almost every claim is supported by research, links, or materials for independent verification. This is a very healthy approach, especially at a time when many publications treat AI more as a fashionable topic than as a field grounded in solid knowledge.

    In summary, “AI Engineering” is a book worth returning to. Not only for the knowledge it provides, but for the way of thinking it promotes—helping to connect strategy, technology, and practice into a coherent whole.

  • Adversarial AI Attacks, Mitigations, and Defense Strategies – book review

    Adversarial AI Attacks, Mitigations, and Defense Strategies – book review

    The book Adversarial AI Attacks is unusual.

    On the one hand, it can genuinely put you off while reading it, and yet I kept wanting to come back to it. It’s not Stockholm syndrome, but rather the real value it carries. But let’s get to the point.

    At the beginning, we are met with an introduction, and it is quite strange. On the one hand, it introduces the topic of AI, but it does so in a very condensed, slogan-like manner. It feels as if it was written exclusively for people who are already very familiar with these concepts. At times, it resembles a conversation with a friend who wants to show off how many complex terms they know.

    Another distinctive feature of the book appears here as well: the translation of technical terms into Polish. Initially, this is handled quite reasonably. Polish equivalents are provided alongside the original English terms. However, later in the book, only the Polish names are used, which makes reading more difficult, as it requires constantly recalling their English origins.

    It is clear that the book was written by a highly technical person. It is not an easy read, yet despite that, I still felt compelled to return to it.

    There are very few books on the market that focus on more sophisticated attacks on AI. Most publications stop at simpler threats, such as Prompt Injection or Unbounded Consumption. And that’s no surprise—it’s easy to compare them to classic attacks like SQL Injection or DoS. This book goes a step further and concentrates on lesser-known, more difficult attacks that often require specialized tools and/or knowledge of advanced mathematics. In this area, it offers an enormous amount of knowledge.

    The structure is fairly systematic—each attack includes a description, its variants, industry examples, and methods for independent replication. Each one is also accompanied by a reference to the original academic research.

    For this reason, I treat this book as a kind of lexicon of AI attacks, built on academic research. It is an excellent reference point—both for learning and for revisiting later when an opportunity arises to apply the described techniques (of course in testing, not in offensive use 😉). That is precisely why I see its schematic nature as an advantage.

    The same applies to the source code—on a first reading, it can be skipped, but during deeper study, it becomes very useful. The downside is that sometimes the sample code is difficult to analyze without the full version available on GitHub. Fortunately, that option exists, so the book excerpts can be treated as commentary on the repository. Unfortunately, the grayscale illustrations instead of color ones are less readable and make understanding more difficult.

    In summary: “Adversarial AI Attacks” is a highly valuable book, though written in a demanding way. It requires considerable effort and has a high entry threshold—definitely intended for readers who already have solid knowledge of AI. In return, however, it delivers an enormous amount of insight. It is hard to find another book in this field so densely packed with substantive material.

  • The Developer’s Playbook for Large Language Model Security – review

    The Developer’s Playbook for Large Language Model Security – review

    This book fits perfectly into the field of AI Security, which I work with on a daily basis. That’s why I had my eye on it for quite some time. I had heard mostly positive opinions about it. In my view, there still aren’t many titles on the market that cover this topic in a structured, example-based, and in-depth way. The subject is the book named The Developer’s Playbook for Large Language Model Security – review is just below.

    For a long time, I had been planning to order the original version. But when I noticed that a Polish edition had been released, I decided to give it a shot and see if the positive reviews held true.

    So, what is the book about? The Developer’s Playbook for Large Language Model Security is an ambitious attempt to systematize the risks, threats, and protection techniques for systems based on large language models (LLMs). The author takes on a tough challenge — describing a fast-evolving and still relatively new domain — in a methodical way, rich with examples and practical references.

    One of the book’s strongest aspects is the abundance of vivid examples that help explain attack mechanisms and possible countermeasures. The style is reminiscent of Adam Shostack’s iconic book on threat modeling — both authors dissect their topic thoroughly, illustrating each threat class with specific, concrete cases. This is definitely a major strength of the book.

    The book doesn’t try to be “cool” — but it’s solid. It reads more like a well-crafted textbook than a popular science title. However, thanks to its clear and practical examples, it doesn’t feel tedious. Reading it feels like reviewing a teammate’s notes — the kind who sketches the entire threat landscape on a whiteboard, then adds two real-world examples and a counterexample so you fully understand where something doesn’t apply.

    I rate the book very positively. The subsequent chapters turned out to be highly educational and inspiring. I found myself jotting down new techniques every few pages — ideas I could immediately apply in my daily work.

    Is this book for everyone?

    No. And that’s a good thing. It’s a book for people who know that “prompt injection” is just the beginning of the problem list, not the end. It’s for those who want to learn to think about LLM systems as real, complex applications with vulnerabilities, attacks, and deployment context.

    Would I recommend it?

    Absolutely.

    This book does an excellent job of organizing the current knowledge on AI security, particularly when it comes to integrating LLMs with broader IT systems.

    The book The Developer’s Playbook for Large Language Model Security – review is very positive. I wish we have more books like this in AI Security area.

  • AI Coding Assistants security

    AI Coding Assistants security

    I think everyone has already heard about it, and most of us have even tried it by now. What’s more, after talking to various people, many people have started to work this way simply on a daily basis, and it helps them very well.

    What is it?

    AI Coding Assistants Security. Today we will combine this with Vibe Coding and look at the security of this approach to coding.

    Writing code together with AI assistants is becoming more and more fun. From generating simple code elements based on comments, to generating entire functionalities. We are slowly starting to move from thinking about writing code to thinking more holistically about creating functions and systems.

    This is a very positive change. We now need to focus more on high-level and architectural thinking, rather than tediously writing line after line of classes and properties.

    Today, with the current tools, we really have tremendous possibilities. Both the ability to choose from many different tools such as GitHub Copilot, Cursor, Windsurf. Each of them also has different capabilities to support the creation of systems such as Background Agents in Cursor.

    But most of all, AI Assistants give us the ability to quickly modify code, as well as quickly create entire applications. It’s not always accurate, but it’s great for prototyping different kinds of applications, which you can then further develop yourself or with the help of AI Assistants.

    This is how the basic architecture model of using AI Assistants looks like.

    We have:

    • codebase – the code of our application
    • assistant rules – a file of common rules used during each generation
    • prompt – queries to the model
    • MCP servers and external resources – which can process data or perform actions outside the AI models (see more)

    Does this mean that from now on, anyone can create an application?

    Yes

    But can anyone create any application?

    Not really anymore.

    And are such apps ready to be used by users?

    Definitely not.

    That’s when the AI coding assistants security comes into the stage.

    Threats of AI Assistants

    Generating erroneous code

    A major drawback of AI Assistants is their literalism. When we don’t specify in the instructions exactly what is to be written, the model can come up with something of its own. Something that won’t quite work for us. For example, it will make a CSS style error that will cause the page to look strange. This doesn’t sound particularly dangerous. Sooner or later we will detect it.

    However, it may turn out that the AI Assistant will also generate an error in the application logic. This too can be detected during testing.

    And what if it generates an error in the logic for logging into the system, or uses dangerous algorithms in the code. After all, he was learning on code from GitHub, not always of the best quality and latest.


    In the following study (dated February 2025) of various models, you can see what percentage of the code generated by a given model was safe, and what percentage contained unsafe insertions.

    https://baxbench.com/ (February 2025)

    Important annotation

    The models are only prompted to complete the coding task. The prompt contains no security-specific instructions, reflecting a realistic interaction with a developer that does not make explicit security considerations.

    The AI model itself can also generate very problematic code completely by accident. A good example is LLMs generating non-existent libraries. This is known as slopsquatting

    When AI generates a reference to a library that doesn’t exist, an attacker can create one, duplicating the original one, and so take over access to the application code, executing whatever he wants in it.

    Generating malicious code

    The headline sounds similar, but there is a rather significant difference. In the previous case, it was the AI model that made the error, in the second it is someone who forces it. Still, both types of errors can be equally serious, but in this case it is a deliberate attack on the system.

    How can this take place?

    For example, using malicious instructions sewn into MDC (common configuration for AI assistants) files.

    A more sophisticated way is to inject Prompt Injection into the model via MCP servers. Through this, in a rather unnoticed way, someone can influence the way code is generated.

    Data leakage

    Another serious risk is that by working with code and sending it to the AI model, there is the possibility of leaking our source code. This alone may be problematic for us, but it can become much more dangerous if our code contains sensitive data or secrets used in the system. Then the problem gets much more serious. Such leaks can happen directly through the AI model or through various MCP servers.

    How to deal with it?

    Secure code in the repository

    The basis of secure code generation is the proper formulation of prompts. However, let’s be honest. Developers do not always keep this in mind, and it is difficult to add such annotations to every even minor prompt. That’s why MDC’s shared prompt configuration files were created. They allow us to create a prompt element that is used in every query. We can put their instructions for ensuring the appropriate level of AI coding assistants security. Because as we said – AI is literal – if we don’t tell it about something, it won’t do it. As inspiration, I recommend examples of MDC files in various projects.

    Another method to ensure that the generated code is secure is a tight Pull Request policy. We need to make sure that the code cannot be merged into the branch from which the application is built for production without proper human verification.

    In general, code generated by the AI Assistant should meet at least as stringent security rules as human-generated code. Both sources can make mistakes.

    Code security

    When it comes to ensuring the security of our code as vital data, it’s worth starting with the basics. Let’s use only trusted AI Assistants, through trusted plugins, to minimize the chances of leakage.

    Some cloud model providers also allow us to declare a request not to use our data to train models. It’s worth keeping this in mind, as there have already been more than once situations where a generic model could get at data from a training collection (see the Samsung case).

    I hope that this brief summary of threats and ways to protect against them will help you create secure code in the most effective way.

    Because it’s a shame to waste time writing the same thing a hundred times, you can do it faster, but not at the cost of security.

    Sources:

    https://threats.backslash.security

    https://cloudsecurityalliance.org/blog/2025/04/09/secure-vibe-coding-guide#

  • MCP Security (Model Context Protocol) – short summary

    MCP Security (Model Context Protocol) – short summary

    Introduction

    MCP (Model Context Protocol) is the recently announced standardized way for AI models to communicate with the outside world, announced by Claude. Specifically, in the context of accessing data from the outside world.

    Source: https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6de5b04a-5e6c-47cf-a81e-e332dd3570df_2906x990.png

    The above architecture shows that it’s an interface that is able to connect various kinds of APIs using a language that is familiar to AI models. It seems to be just a minor standardization, but it has triggered a creation of multiple new servers for more or less popular services. It’s easy enough that none of the big players want to be left behind, and everyone wants to provide an interface for contact between AI and their system.

    You can see how popular it is when you look at the number of new implementations of this protocol.

    https://github.com/punkpeye/awesome-mcp-servers

    https://mcpservers.org

    https://mcp.ing/explore

    https://mcpverse.dev

    Let’s start by taking a closer look at what MCP (Model Context Protocol) is.

    We have to start with the official MCP (Model Context Protocol) specification

    https://modelcontextprotocol.io/specification

    A short and simple explanation of what MCP is:

    https://read.highgrowthengineer.com/p/mcps-simply-explained

    Also, an interesting and more in-depth description of the protocol itself (MCP), as well as the possibilities for its use and potential future expansion:

    https://a16z.com/a-deep-dive-into-mcp-and-the-future-of-ai-tooling

    The big advantage of this new protocol, as well as its rapid adoption, is that it allows AI to use specialized tools to increase its efficiency. A good example of this is reverse engineering. So far, LLMs have had to rely on their own abilities to do this and analyze code purely at the language layer. With interfaces such as GhidraMCP (https://github.com/LaurieWired/GhidraMCP), it can take advantage of mature solutions and process already preprocessed data.

    MCP and security

    The topic of MCP security can be summarized up in one sentence:

    The article sums it up nicely:

    https://elenacross7.medium.com/️-the-s-in-mcp-stands-for-security-91407b33ed6b

    Tool Poisoning Attack threat description

    https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks

    You can also see the vulnerabilities that can exist in the MCP code by seeing the examples.

    Intentionally Vulnerable MCP Server (Built to test SQL Injection (SQLi) and Remote Code Execution (RCE) vulnerabilities via FastAPI, JSON-RPC, and LLM-based decision logic.)

    https://github.com/evrenyal/mcpsecurity

    https://github.com/harishsg993010/damn-vulnerable-MCP-server

    A very structured overview of MCP server threats with examples of problematic calls.

    Source: https://evren.ninja/mcp-security.html

    Another list of threats related to MCP servers:

    1. Command Injection Vulnerabilities – running malicious code passed in parameters.
    2. Tool Poisoning Attacks – injecting malicious instructions into the server’s action description.
    3. The Rug Pull: Silent Redefinition – the ability to substitute server behavior.
    4. Cross-Server Tool Shadowing – replacing the results of one MCP by another.
    5. Context Leakage Risks – a long-held session can leak sensitive information.
    6. Prompt Injection – overwriting the default behavior of a prompt by a response from an MCP.
    7. Memory Poisoning and Context Corruption – corruption of session context through a malicious response from MCP.

    https://elenacross7.medium.com/️-the-s-in-mcp-stands-for-security-91407b33ed6b

    https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks

    https://medium.com/@sebuzdugan/understanding-the-security-implications-of-using-mcp-9bd3323ad42d

    Here instead we have a very insightful analysis of MCP (Model Context Protocol) threats

    1. Name Collision – impersonation of trusted servers.
    2. Installer Spoofing – injection of an infected server.
    3. Code Injection/Backdoor – malicious code in a server implementation.
    4. Sandbox Escape – exiting a command outside a trusted environment and gaining unauthorized access to the operating system.
    5. Slash Command Overlap – conflict in the names of methods available to AI.

    https://arxiv.org/pdf/2503.23278

    So, how to defend against these threats?

    There are many methods, but they can be summed up to a simple Zero Trust approach and taking care of classic security principles:

    • session management,
    • authentication and authorization,
    • encryption on transit,
    • good monitoring.

    https://medium.com/@sebuzdugan/understanding-the-security-implications-of-using-mcp-9bd3323ad42d

    In addition, we have several security verification tools available.

    Concept of using AI to evaluate MCP servers

    https://github.com/JeredBlu/custom-instructions/blob/main/mcpevaluatorv3.md

    An interesting way to verify publicly available MCP servers from different vendors. It is interesting that we can also use this method to verify the security of other libraries or tools

    https://www.youtube.com/watch?v=LYUDUOevtqk

    MCP server security verification tool

    https://github.com/invariantlabs-ai/mcp-scan

    On the other hand, MCPs can also help us in daily application security.

    Source: https://github.com/cyproxio/mcp-for-security

    Final thoughts

    The big problem with MCP servers is the implementation itself. Especially when it is so easy to create such a server. Even more so in the vibe coding model.

    The protocol itself does not define any security, which can provoke a poor implementation. However, with the right Zero Trust approach from both sides. As a Client, we don’t know what MCP will return and whether it’s secure, and as an MCP Server, we also have to be distrustful of the input.