What is AI Security? Protecting AI Systems from Attacks

A financial services firm deployed a document-processing AI to extract data from invoices. Within two weeks, a supplier had discovered that embedding hidden text in invoice images could cause the system to extract incorrect amounts. Nobody wrote a SQL injection payload. Nobody exploited a memory buffer. The attack worked by exploiting how the model itself processed image data.

That's the core challenge of AI security. The attack surface isn't just the surrounding infrastructure, it's the model, the training data, the prompts that control behavior, and the outputs that feed downstream systems. Traditional cybersecurity disciplines protect software. AI security protects the AI itself.

Why AI Security Is Distinct from Traditional Cybersecurity

Traditional software security protects against attackers exploiting code: buffer overflows, injection attacks, authentication bypasses. These attacks work because the code behaves deterministically, and an attacker finding an input that triggers unexpected code paths can cause predictable damage.

AI systems introduce a different kind of vulnerability. Models are trained on data, and that training process can be manipulated. Models respond to inputs in ways that can be influenced by carefully crafted adversarial examples that look normal to humans. Models that process text can be given instructions embedded in their inputs that override their intended behavior. And models themselves, which represent significant commercial value, can be stolen or replicated.

None of these attacks require finding a bug in the code. They exploit properties of how machine learning works.

This doesn't mean traditional security is irrelevant for AI systems. Infrastructure security, access controls, and secure software development all apply. But they're not sufficient. AI security adds a layer of concerns specific to model behavior, training integrity, and inference-time manipulation.

The Main AI Security Threat Categories

Adversarial attacks manipulate inputs to cause models to produce incorrect outputs. In computer vision, this means adding carefully computed pixel-level noise to an image that looks normal to a human but causes a classification model to return a completely wrong label, with high confidence. In text systems, adversarial attacks craft inputs that exploit specific weaknesses in how models represent language. These attacks matter for any AI system making consequential decisions based on its inputs, from fraud detection to content moderation to medical imaging.

Data poisoning corrupts the training process. If an attacker can influence what data a model is trained on, they can create a "backdoor": a specific pattern that causes the model to behave incorrectly whenever it appears in production, while performing normally otherwise. A model trained on web-scraped data is exposed to any content that can be placed on the web. Supply chain attacks on training datasets are a growing concern, particularly for organizations that use publicly available data or third-party data providers.

Prompt injection targets large language models and other systems that follow natural language instructions. An attacker embeds instructions in content that the AI will process, and those embedded instructions override the system's intended behavior. A customer service bot told to "summarize this document" can be sent a document containing hidden instructions telling it to instead reveal its system prompt, ignore its content filters, or exfiltrate information. As AI systems take on more agentic workflows with access to tools and databases, prompt injection becomes a serious security threat: a successfully injected instruction can cause the agent to take actions its operators never intended.

Model theft and extraction target the model itself as a valuable asset. Through repeated queries, an attacker can reconstruct an approximation of a proprietary model's behavior, effectively stealing the intellectual property embedded in the model without ever accessing the model's weights. Organizations that have invested millions in training or fine-tuning models face genuine IP theft risk from well-resourced adversaries.

Model inversion extracts information about training data. In some cases, attackers can query a model in ways that reveal details about what it was trained on, including potentially sensitive data from individuals whose information was in the training set. This creates a privacy risk distinct from data breaches: the sensitive information isn't stolen from a database, it's extracted from a model.

How AI Security Differs from AI Safety

The terms are often confused, but they address different threats.

AI safety is concerned with AI systems behaving in unintended ways due to misalignment, edge cases, or capability failures. Safety asks: what happens when the AI does something wrong through no adversarial intent? Examples include a recommendation system that optimizes for engagement at the expense of user wellbeing, a robustness failure when a model encounters out-of-distribution inputs, or an agentic workflow that achieves its objective in a way its designers didn't anticipate.

AI security is concerned with deliberate attack. Security asks: what can an adversary do to make the AI behave in ways that benefit the attacker? The same underlying technical concepts, like adversarial inputs, sometimes appear in both fields. But safety research focuses on unintentional failures, while security research focuses on intentional exploitation.

Both matter. A production AI system needs safety engineering to handle unexpected inputs gracefully and security engineering to handle deliberate attacks.

AI Security in Enterprise Practice

For organizations deploying AI, security considerations translate into concrete practices.

Threat modeling before deployment. Before a model goes into production, work through the specific attack surfaces it exposes. Who has the ability to send it inputs? What actions can it take? What would a motivated attacker gain from manipulating it? This analysis shapes which security controls are worth investing in.

Input validation and sanitization. For systems that process user-provided content, implement filters on inputs before they reach the model. For LLM-based systems, this means screening for prompt injection patterns, though no filter is complete against a determined attacker. For document processing systems, treat every document as potentially adversarial.

Prompt injection defenses for agentic systems. AI agents with tool access need special attention. Architectural controls, such as separating the instruction space from the content space, limiting what tools an agent can access, and requiring human confirmation for sensitive actions, reduce the blast radius of a successful injection. Defense-in-depth is the right mental model: no single control is sufficient.

Output monitoring and anomaly detection. AI observability tools that track what models produce in production can catch anomalous behavior that might indicate an ongoing attack. Unusual output patterns, unexpected tool calls in agentic systems, or statistical drift in outputs are all signals worth monitoring.

Access controls on model APIs. Model endpoints should be treated as sensitive assets. Rate limiting reduces the feasibility of extraction attacks. Authentication ensures only authorized clients can query the model. Logging creates an audit trail for forensic analysis.

Supply chain security for training data. Organizations training on external data should apply the same scrutiny to training data provenance that they apply to software dependencies. Curated, verified datasets are more secure than large undifferentiated web scrapes. When third-party data is unavoidable, periodic red-teaming for backdoor behavior is worth the investment.

The Regulatory Dimension

AI security is becoming a compliance concern, not just a technical one. The EU AI Act requires that high-risk AI systems implement appropriate security measures, including protection against adversarial attacks. The NIST AI Risk Management Framework includes security as a core component of responsible AI governance. Organizations in regulated industries, financial services, healthcare, critical infrastructure, are increasingly expected to demonstrate that their AI systems are secure, not just functional.

This regulatory pressure is raising the bar for AI security documentation. AI model cards and AI audit trails increasingly need to address how models have been security-tested, what known vulnerabilities exist, and what mitigations are in place.

Building AI Security Capability

For most organizations, AI security capability builds on existing security foundations. Security teams already understand threat modeling, secure architecture, and incident response. What they need in addition is knowledge of the AI-specific threat categories and the techniques used to test for them.

AI red-teaming is the most direct way to develop both knowledge and defenses. Red team exercises against production AI systems reveal actual vulnerabilities in actual deployment contexts, rather than abstract threat scenarios. Organizations that run regular AI red teaming develop both the defenses and the organizational muscle to maintain them.

The alternative, learning about AI security weaknesses after a production incident, is considerably more expensive.

Related AI Concepts

• AI Safety - Protecting AI from unintentional failure modes
• AI Red Teaming - Adversarial testing to find security vulnerabilities
• AI Guardrails - Technical controls on model outputs
• AI Observability - Monitoring model behavior in production
• AI Governance - Organizational accountability for AI security
• Agentic Workflows - High-risk AI systems for security consideration
• AI Audit Trail - Logging for forensic and compliance purposes

External Resources

• NIST AI Risk Management Framework - US government framework covering AI security requirements
• OWASP Top 10 for LLM Applications - The security community's canonical list of LLM vulnerabilities including prompt injection
• MITRE ATLAS - Adversarial threat landscape for AI systems, with real-world attack case studies

FAQ