Security & Privacy for AI

AI systems introduce entirely new classes of security vulnerabilities that traditional application security doesn't address. Prompt injection, data poisoning, model manipulation, and PII leakage are risks unique to AI — and they're being actively exploited. This module covers the attack surface of AI systems, practical defenses, privacy compliance, red teaming methodologies, and a security best practices checklist for production AI applications.

Prompt Injection Attacks

Prompt injection is the most prevalent vulnerability in LLM-powered applications. It occurs when an attacker crafts input that causes the model to ignore its instructions and follow the attacker's instructions instead. It's conceptually similar to SQL injection — untrusted input being mixed with trusted instructions — but harder to defend against because there's no strict grammar separating instructions from data.

Direct Prompt Injection

The attacker communicates directly with the model and attempts to override its system prompt or manipulate its behavior.

Indirect Prompt Injection

The more dangerous variant. The attacker embeds malicious instructions in content that the AI system will process — web pages, documents, emails, or database records. The user never sees the attack; it's triggered when the AI retrieves or processes the poisoned content.

Prompt Injection Defenses

• Input filtering: Scan user inputs for known injection patterns, suspicious instructions, and role-play attempts. Use both pattern matching and a classifier model trained to detect injection attempts.
• Output filtering: Validate model outputs before returning them to the user. Check for leaked system prompts, PII, harmful content, and off-topic responses.
• Privilege separation: Don't give the model access to capabilities it doesn't need. If the chatbot doesn't need to send emails, don't give it an email tool. Apply the principle of least privilege.
• Sandboxed tool execution: When AI agents execute tools, run them in sandboxed environments with limited permissions. Require human confirmation for high-impact actions (deleting data, sending communications, financial transactions).
• Instruction hierarchy: Modern model APIs support explicit separation between system prompts and user messages. Anthropic and OpenAI both emphasize prioritizing system-level instructions over user-level instructions when conflicts arise.
• Delimiter-based separation: Use clear delimiters to mark the boundaries between instructions and untrusted data within prompts, such as XML tags or unique boundary tokens.

Data Privacy in AI Pipelines

AI systems process vast amounts of data — training data, user inputs, retrieved documents, and generated outputs. Each stage introduces privacy risks that must be managed.

PII Detection and Handling

Personally Identifiable Information (PII) — names, emails, phone numbers, addresses, Social Security numbers, financial data — flows through AI systems in multiple places. You need strategies for each.

For PII detection, use dedicated tools like Microsoft Presidio (open-source), Google Cloud DLP, or AWS Comprehend. These use a combination of regex patterns, NLP models, and contextual analysis to identify PII across 40+ entity types. LLMs themselves can also serve as PII detectors, though purpose-built tools are faster and more reliable for this specific task.

Regulatory Compliance for AI

GDPR (EU General Data Protection Regulation)

GDPR has significant implications for AI systems processing data of EU residents:

• Transparency for automated decisions: Under Articles 13–15 and 22, users have the right to "meaningful information about the logic involved" in automated decision-making. For AI systems making consequential decisions (credit scoring, hiring), you must be able to provide an explanation of how the decision was reached.
• Right to erasure: Users can request deletion of their data. This means you need the ability to identify and delete all data associated with a user — including RAG documents, conversation logs, embeddings, and any fine-tuning data derived from their interactions.
• Data minimization: Collect and process only the data necessary for the specific purpose. Don't send entire user profiles to the model if only the user's name is needed.
• Data processing agreements: When using third-party model providers, establish DPAs that specify how data is handled, stored, and retained. Verify that providers don't train on your data without consent.

EU AI Act

The EU AI Act, which began its phased implementation in 2025, establishes risk-based regulations for AI systems:

• Unacceptable risk: Banned — social scoring, real-time biometric surveillance (with narrow exceptions), manipulative AI.
• High risk: Heavily regulated — AI in hiring, credit scoring, education, law enforcement. Requires conformity assessments, risk management, documentation, and human oversight.
• Limited risk: Transparency obligations — chatbots must disclose they are AI, deepfakes must be labeled, AI-generated content must be identifiable.
• Minimal risk: No specific requirements — most general-purpose AI applications fall here.

Red Teaming AI Systems

Red teaming is the practice of systematically testing your AI system by trying to make it fail, produce harmful outputs, or behave in unintended ways. It's the AI equivalent of penetration testing and should be a regular part of your development and release process.

Red Team Testing Categories

Running Effective Red Team Exercises

• Assemble diverse testers: Include security engineers, domain experts, and creative thinkers. Different backgrounds find different vulnerabilities. External red teams bring fresh perspectives.
• Automate where possible: Use tools like Garak (open-source LLM vulnerability scanner) or Microsoft PyRIT to automatically probe for known vulnerability categories. Manual testing then focuses on novel attack vectors.
• Document and track findings: Treat vulnerabilities like security bugs — document reproduction steps, severity, and impact. Track remediation progress.
• Test regularly: Red team before every major release, after prompt changes, and after model updates. What was safe with one model version may not be safe with the next.

Content Filtering and Safety Layers

Safety layers are the guardrails that prevent your AI system from producing harmful, inappropriate, or off-brand outputs. They operate at both the input and output stages.

Input Safety

• Content classification: Run user inputs through a content classifier that detects harmful requests, prompt injection attempts, and off-topic queries. Route flagged inputs to specialized handling (rejection, human review, or sanitized processing).
• Rate limiting: Limit the number of requests per user per time window to prevent abuse and automated attacks.
• Input sanitization: Strip or escape special characters, encoding tricks (base64, ROT13), and formatting that could be used to bypass content filters.

Output Safety

• Content moderation: Run model outputs through a moderation classifier before returning them to the user. OpenAI's Moderation API and Anthropic's built-in safety features provide baseline content filtering.
• PII scanning: Scan outputs for PII that the model may have included from retrieved context or training data. Redact before displaying.
• Factuality checks: For applications where accuracy matters (medical, legal, financial), compare generated claims against source documents and flag unsupported statements.
• Brand and tone guards: Ensure outputs match your brand voice and stay within topic boundaries. A customer support bot should not generate political opinions, even if asked.

Supply Chain Risks

AI systems depend on a supply chain of models, datasets, and libraries that introduce risks beyond traditional software dependencies.

Model Poisoning

If you download open-source models from the internet, you're trusting that the model weights haven't been tampered with. A poisoned model can behave normally on most inputs but produce specific harmful outputs when triggered by a secret pattern (a "backdoor").

• Download models only from trusted sources (official Hugging Face repos with verified organizations)
• Verify model checksums against published hashes
• Run evaluation suites on downloaded models before deployment
• Monitor for unexpected behavior patterns in production

Data Poisoning

Training data or fine-tuning data can be poisoned to introduce biases or backdoors. Even RAG knowledge bases are vulnerable — an attacker who can insert documents into your knowledge base can influence model outputs.

• Validate and review data sources before ingestion
• Implement access controls on knowledge base modifications
• Audit knowledge base changes with version control
• Run data quality checks and anomaly detection on new data

Security Best Practices Checklist

Resources