AI Security: Adversarial Attack on Image Classifier
A company uses an image classification model to detect defects in manufactured parts on an assembly line. The model achieves 98% accuracy on test data. However, an attacker with access to the assembly line's camera feed can subtly alter images (e.g., by adding small stickers) to cause the model to misclassify defective parts as non-defective. This could allow defective products to pass quality control. The company is concerned about both security and safety. They have implemented network security but not considered adversarial attacks. The model is retrained quarterly. The attacker could be a malicious insider or an external actor who gains access to the camera system.
1. Explain the nature of the adversarial attack and why the model is vulnerable despite high test accuracy.
Read the worked answer
The attack is an adversarial example attack, specifically an evasion attack at inference time. The attacker adds small perturbations (stickers) to the input image that are imperceptible to humans but cause the model to misclassify. High test accuracy does not imply robustness to adversarial examples because the test data is drawn from the same distribution as training data, while adversarial examples are deliberately crafted to exploit model vulnerabilities. Deep neural networks are vulnerable due to their linear nature and the high-dimensional input space; small changes in pixel values can accumulate to change the model's output. The model's decision boundary may be close to legitimate samples, so small perturbations cross the boundary. The attacker's access to the camera feed allows real-time manipulation. The quarterly retraining does not address this because the vulnerability is inherent to the model architecture and training.
2. Recommend defenses to mitigate adversarial attacks and improve the system's security posture.
Read the worked answer
Defenses: 1. Adversarial training: Augment training data with adversarial examples to improve robustness. This can be done during quarterly retraining. 2. Input preprocessing: Apply transformations (e.g., JPEG compression, spatial smoothing) that may remove adversarial perturbations. 3. Anomaly detection: Monitor for unusual patterns in images or camera feed integrity. 4. Ensemble models: Use multiple models and majority vote to increase robustness. 5. Physical security: Secure cameras and network to prevent unauthorized access; use tamper-proof cameras. 6. Human oversight: Implement random manual inspections. 7. Model diversity: Use different architectures. 8. Certified defenses: For critical applications, use provable defenses (e.g., randomized smoothing). 9. Regular red teaming: Hire ethical hackers to test the system. 10. Incident response plan: Have a process to detect and respond to attacks. Note that no defense is perfect; combine multiple layers. Also, consider the safety impact: if defective parts pass, there could be liability. The company should prioritize defense-in-depth.