AI Security

• How the attack surface changes

  Understand how the attack surface changes when models can process untrusted content, access private data, call tools, and take actions.

• Trust boundaries

  Identify trust boundaries across prompts, retrieved data, tools, memory, model outputs, and external systems.

• Model output is untrusted

  Treat model-generated outputs and actions as untrusted until validated.

Outcome · Threat-model an LLM application before adding defenses.

• Direct vs indirect injection

  Distinguish direct prompt injection from indirect injection hidden inside documents, web pages, emails, retrieved chunks, or tool outputs.

• What injection attacks attempt

  Understand attacks that attempt to override instructions, manipulate tools, expose private context, or redirect an agent's objective.

• Untrusted input + tools + an exit path

  Understand the dangerous combination of untrusted input + powerful tools/private data + an exfiltration path.

• Layered defences

  Defend using instruction/data separation, least privilege, authorization checks, restricted tools, confirmation gates, and output validation.

• Why "ignore malicious instructions" fails

  Understand why prompting the model to “ignore malicious instructions” is not a sufficient security boundary.

Outcome · Build applications that remain safe even when malicious instructions reach the model.

• Jailbreaks vs injection

  Distinguish jailbreaks from prompt injection and recognize common jailbreak patterns.

• Input & output guardrails

  Add input and output guardrails, moderation, validation, and policy checks.

• Why one layer is not enough

  Understand the limitations of model-based guardrails and why multiple layers of defense are required.

• Testing against obfuscated input

  Test guardrails against adversarial and obfuscated inputs instead of only normal requests.

Outcome · Reduce unsafe model behavior without relying on a single defensive prompt.

• Least privilege for agents

  Apply least privilege so agents only receive the tools and permissions required for the task.

• Read-only vs consequential actions

  Separate read-only actions from consequential actions such as sending messages, modifying records, deleting data, purchasing, or executing code.

• Validating tool use outside the LLM

  Validate tool selection, arguments, permissions, and outputs independently of the LLM.

• Human approval for irreversible actions

  Add human approval for sensitive or irreversible operations.

• Step, runtime & cost limits

  Limit agent steps, runtime, cost, and tool usage to prevent runaway execution.

• Sandboxing generated code

  Sandbox model-generated code and treat generated SQL, commands, URLs, or API calls as untrusted.

Outcome · Give agents useful autonomy without giving them unrestricted control over real systems.

• Retrieved documents are data, not instructions

  Treat retrieved documents as untrusted data, not trusted instructions.

• Retrieval poisoning

  Understand retrieval poisoning and malicious instructions hidden inside indexed documents.

• Document-level permissions

  Enforce document-level permissions so users retrieve only information they are authorized to access.

• Preventing cross-tenant leakage

  Prevent cross-user or cross-tenant data leakage from vector stores and memory systems.

• Preserving provenance

  Preserve source/provenance information for retrieved content.

Outcome · Secure the knowledge layer underneath a RAG application.

• Identifying sensitive data

  Identify sensitive data including PII, credentials, financial information, private conversations, and confidential business data.

• Redaction & minimization

  Detect, redact, mask, or minimize sensitive information before sending it to models when appropriate.

• Keeping PII out of logs & traces

  Avoid unnecessarily storing sensitive data in prompts, logs, traces, memories, vector stores, or evaluation datasets.

• Retention, deletion & residency

  Understand data retention, deletion, residency, access control, and third-party model-provider considerations.

• Private connectivity & zero retention

  Use the controls that make regulated deployments possible: private connectivity to the model endpoint, no public egress, and contractual zero-retention or no-training terms.

• Protecting secrets from the model

  Protect secrets so they are never unnecessarily exposed to the model.

Outcome · Minimize the amount of sensitive information exposed throughout an AI system.

• Building adversarial test suites

  Build adversarial tests for prompt injection, jailbreaks, malicious documents, tool misuse, data extraction, and unsafe agent actions.

• Attacking RAG through the corpus

  Test RAG systems by injecting malicious content into retrievable documents.

• Attacking agents through their tools

  Test agents by attempting to make them exceed permissions, misuse tools, reveal secrets, or enter loops.

• Vulnerabilities become regression tests

  Turn discovered vulnerabilities into permanent security regression tests.

• Attack-success & leakage rates

  Track measures such as attack-success rate, sensitive-data leakage, and unsafe-action rate where useful.

Outcome · Test defenses adversarially instead of assuming they work.

• Bias, transparency & human oversight

  Understand core concerns around hallucination, bias/fairness, transparency, accountability, and human oversight.

• Documenting use, limits & risks

  Document intended use, known limitations, important risks, evaluations, and human-control mechanisms.

• The EU AI Act & India's DPDP

  Understand the practical implications of relevant regulations such as the EU AI Act and India's DPDP framework.

• Auditability

  Maintain auditability through model/prompt versions, logs, approvals, and evaluation records.

Outcome · Recognize when an AI application creates governance or compliance obligations.