Describe the difference between an evasion attack and a poisoning attack.

An evasion attack occurs at test time (inference), where an attacker modifies an input to fool a pre-trained model. For example, adding imperceptible noise to an image to cause misclassification. A poisoning attack occurs at training time. The attacker injects malicious data into the model's training set to compromise the learning process itself. This can create a 'backdoor' or degrade the model's overall performance on legitimate inputs later on.

Adversarial machine learning

Advanced

Adversarial machine learning is a research field and technique that involves creating malicious inputs, known as adversarial examples, to fool or mislead machine learning models. The primary goal is to cause a model to make an incorrect prediction. This field explores both the attacks on machine learning systems and the corresponding defenses to make them more secure and robust against such manipulations.

First used·2013-2014

Definitions·1

Synonyms·3

Category·Artificial Intelligence

Also known as

Adversarial AIAdversarial Attacks on MLML Security

Definitions

What it means.

01
Adversarial Machine Learning in Cybersecurity
Adversarial Machine Learning is a field at the intersection of machine learning and cybersecurity that focuses on the security of ML models in the presence of an intelligent adversary. It is broadly divided into attacks and defenses.

Attacks

Attacks are methods used to compromise a model's integrity, availability, or confidentiality. They are often categorized by the attacker's goal:

Evasion Attacks: The most common type, where an attacker crafts malicious inputs (adversarial examples) to fool a trained model during inference. For example, a spam email is slightly modified with invisible characters to bypass a spam filter.

Poisoning Attacks: The attacker injects malicious data into the training set to corrupt the learned model. This can degrade overall performance or create a 'backdoor' that the attacker can later exploit with a specific trigger.

Model Stealing (or Extraction): The attacker queries a model (often via an API) to reconstruct the model itself or extract sensitive information from its training data.

Defenses

Defenses are techniques designed to make models more resilient to such attacks. The goal is to improve model robustness.

Adversarial Training: This is a proactive defense where adversarial examples are generated and included in the model's training data. This process teaches the model to correctly classify these malicious inputs, making it more robust.

Input Preprocessing: This involves sanitizing or transforming inputs before they are fed to the model to remove potential adversarial perturbations. Examples include image compression or feature squeezing.

Gradient Masking/Obfuscation: These techniques aim to hide the model's gradient information, which attackers often use to craft effective adversarial examples. However, this is sometimes seen as providing a false sense of security, as attacks can be adapted to bypass it.

The field of Adversarial AI is often described as a 'cat-and-mouse' game, where the development of new attacks drives the creation of more sophisticated defenses, and vice versa.

Origin

Where it comes from.

Etymology

The term combines 'Adversarial,' from the Latin 'adversarius' meaning 'opponent' or 'rival,' with 'Machine Learning.' It aptly describes a scenario where an adversary, or opponent, actively works to subvert the functioning of a machine learning system.

Historical context

The concept of adversarial manipulation has roots in early cybersecurity fields like spam filtering, where spammers constantly evolved their techniques to bypass filters. However, the term Adversarial Machine Learning gained significant prominence in the context of modern deep learning.

A pivotal moment occurred between 2013 and 2014 with the publication of papers like 'Intriguing properties of neural networks' by Szegedy et al. and 'Explaining and Harnessing Adversarial Examples' by Goodfellow et al. These studies revealed a shocking vulnerability: high-performing, state-of-the-art neural networks could be tricked into making confident, incorrect predictions by adding tiny, often human-imperceptible perturbations to their inputs.

This discovery launched a wave of research. Initially, attacks were 'white-box,' meaning the attacker had full knowledge of the model's architecture and parameters. Research quickly expanded to more realistic 'black-box' scenarios where the attacker has limited or no knowledge of the model. The scope of Adversarial AI also grew beyond image recognition to affect natural language processing (NLP), audio systems, and reinforcement learning, demonstrating it was a fundamental weakness, not just a quirk of image classifiers.

Usage

In context.

The cybersecurity team is studying Adversarial Machine Learning to build more resilient fraud detection systems that can't be easily bypassed.
In a famous example of an Adversarial AI attack, researchers placed small stickers on a stop sign, causing an autonomous vehicle's classifier to misinterpret it as a speed limit sign.
To improve our model's defenses, we are implementing adversarial training, a key technique to protect against Adversarial Attacks on ML.
The field of ML Security is growing rapidly as more critical systems begin to rely on artificial intelligence.

FAQ

Common questions.

The primary goal is to cause a machine learning model to make an incorrect prediction or classification. This is achieved by feeding it a specially crafted, malicious input called an 'adversarial example.' This input is often designed to be nearly indistinguishable from a legitimate one to a human observer, highlighting the model's vulnerabilities.

Taxonomy

What it means.