AI risk: threat modeling
This course applies threat modeling to AI systems, where text reaching a model is both data and instructions and behavior cannot be enumerated. It covers direct and indirect prompt injection, the attack surface of tool-calling agents, trust between chained agents, and risks in training data, model files and dependencies. It also separates practical model-facing attacks from mostly academic ones for a typical Rails product. The last lesson gathers defenses built around a privilege boundary: least-privilege tools, human approval for irreversible actions, sandboxing, spend limits and reconstructable logs.
What you'll learn
- ✓Why model input is both data and instructions, and what that means for threat modeling
- ✓Direct and indirect prompt injection, and the limits of prompt-level defenses
- ✓Agent risks: tool abuse, confused deputy, excessive agency, unbounded loops and memory poisoning
- ✓A deny-by-default tool authorization policy based on granted scope and reversibility
- ✓Supply chain risks in training data, model hubs, model file formats and AI dependencies
- ✓Defenses such as sandboxing, spend limits, output validation, provenance and incident logging
Harm by accident, harm by attacker
The senior course "AI ethics: fairness, transparency, accountability" was about harm a system causes while working as designed. A model trained on biased history reproduces that bias at scale. A decision can't be explained to the person it affected. Nobody attacked that system. It did what it was…
Read the full lesson free →Curriculum
7 lessonsHarm by accident, harm by attacker
Why AI systems need threat modeling: their input mixes data with instructions, and their behavior cannot be enumerated.
Prompt injection: direct and indirect
Direct versus indirect prompt injection, why prompt-level defenses stay probabilistic, and why the privilege boundary is the control that holds.
The agent-specific attack surface
Tool abuse, the confused deputy problem, excessive agency, unbounded loops and memory poisoning in tool-calling agents.
Multi-agent trust and the privilege boundary
How chaining agents multiplies risk, and a Rails policy object that authorizes each tool call by scope and reversibility.
Data and supply chain risk
Training-data poisoning, backdoored and typosquatted models, unsafe pickle-based formats and dependency hygiene in the AI toolchain.
Model-facing attacks: extraction and disclosure
Model extraction, membership inference and output disclosure, sorted into practical and mostly academic risks for hosted-API Rails apps.
Defenses that hold
Least privilege, human approval, output validation, sandboxing, spend limits, provenance and logging that makes incidents reconstructable.