Black-Box Neural Network Hardening Through Surrogate Model Defense Integration

This repository provides the implementation of a comprehensive framework for enhancing the adversarial robustness of black-box neural networks. It is designed for scenarios where the defender has query-only API access to the model and lacks access to its internal weights, architecture, or original training data.

Framework Overview

Adversarial robust neural network deployment is a critical challenge in applied machine learning security. This project solves the problem of defending proprietary black-box APIs by creating a high-fidelity surrogate model via conditional data synthesis and knowledge distillation. The surrogate serves as a transparent proxy on which advanced defense mechanisms are integrated, and those defensive properties are subsequently transferred back to protect the original black-box.

Seven-Stage Pipeline Architecture

The methodology is organized into a formal seven-stage pipeline:

1. Model Wrapping: The target model is wrapped in a standardized query API supporting ONNX and PyTorch formats. This enforces the black-box constraint by restricting access to only predict() and predict_proba() outputs.
2. Conditional Data Synthesis: A conditional WGAN-GP generator ($G: \mathcal{Z} \times \mathcal{Y} \rightarrow \mathcal{X}$) synthesizes labeled training data without access to the original dataset.
3. Surrogate Model Creation: The surrogate model learns the decision boundaries of the black-box through knowledge distillation (minimizing Kullback-Leibler divergence with temperature scaling).
4. Adversarial Attack Testing: The baseline vulnerability of the surrogate is evaluated against standard and adaptive attacks.
5. Defense Integration: Robustness is injected into the model through adversarial training and non-differentiable preprocessing.
6. Model Export & Watermarking: The hardened surrogate is exported with embedded metadata watermarks.
7. Robustness Validation: The system evaluates defense transferability and calculates statistical confidence intervals.

Implemented Defenses

Because gradients cannot be calculated directly through a black-box API, the framework utilizes the surrogate model to apply a layered defense stack:

• Adversarial Training: Implemented in the adversarial_training function, the model is trained on a 50/50 mixture of clean data and adversarial examples generated via PGD. The $\epsilon$ perturbation bounds are dynamically scaled depending on the dataset (e.g., $\epsilon=0.3$ for MNIST, $\epsilon=0.03$ for CIFAR-10).
• JPEG Compression: The jpeg_compression_defense function acts as a preprocessing step with a quality factor of 75 to disrupt high-frequency adversarial noise.
• Bit-Depth Reduction: The bit_depth_reduction function truncates pixel values to 4 bits, acting as a non-differentiable filter to eliminate imperceptible adversarial perturbations.
• Total Variation (TV) Denoising: The tv_denoising function smooths the image by minimizing total variation, stripping localized adversarial artifacts.

Evaluated Attack Vectors

To rigorously validate the defense transfer, the framework evaluates the model against three categories of adversarial threats:

1. Discrete Attacks

Standard $L_{\infty}$ bounded iterative and optimization-based attacks are evaluated:

• FGSM (Fast Gradient Sign Method)
• PGD (Projected Gradient Descent)
• BIM (Basic Iterative Method)
• MIM (Momentum Iterative Method)
• C&W (Carlini & Wagner L2)
• EAD (Elastic-net Attacks)

2. Compound Attacks

Compound attacks combine multiple adversarial strategies sequentially to bypass single-mode defenses. The framework utilizes a compound_attack pipeline testing the following chains:

• FGSM + PGD
• PGD + C&W
• BIM + MIM
• AutoAttack: A benchmark-grade, parameter-free ensemble attack (incorporating APGD-CE) designed to identify the absolute lower bound of true robustness.

3. Adaptive Attacks

To ensure defenses are not merely relying on gradient obfuscation (a false sense of security), adaptive attacks are executed against the full defense stack:

• BPDA (Backward Pass Differentiable Approximation): Approximates gradients through the non-differentiable JPEG and Bit-Depth reduction layers using the identity function in the backward pass.
• EOT (Expectation Over Transformation): Averages gradients over multiple stochastic passes to bypass randomized defenses.

Experimental Results

The methodology was validated across MNIST, CIFAR-10, and TinyImageNet datasets. Statistical validation confirms significance across all datasets with $p < 0.001$ and Cohen’s $d > 1.0$.

Surrogate Fidelity

Knowledge distillation utilizing the WGAN-GP synthesized data resulted in highly accurate surrogates:

• MNIST: 94.7% Agreement
• CIFAR-10: 85.4% Agreement
• TinyImageNet: 78.0% Agreement

Robustness Improvements (CIFAR-10 Example)

The full defense stack successfully thwarted standard and compound attacks, matching approximately 80% of the white-box adversarial training ceiling without requiring model access.

Attack Type	Baseline (Undefended)	Hardened Surrogate	Improvement
PGD	0.20%	54.60%	+54.40%
C&W	2.00%	54.60%	+52.60%
FGSM+PGD (Compound)	0.60%	54.60%	+54.00%
AutoAttack (Compound)	1.00%	54.60%	+53.60%
BPDA (Adaptive)	0.20%	20.00%	+19.80%

(Data derived from comprehensive evaluation reports)

Defense Transfer Validation

Defensive properties successfully transferred to the original target black-box:

• MNIST: +64.0% transfer improvement
• CIFAR-10: +76.5% transfer improvement
• TinyImageNet: +49.5% transfer improvement

Setup and Requirements

The codebase is optimized for high-performance computing utilizing NVIDIA A100 GPUs.

Prerequisites

• Python 3.8+
• torch, torchvision (PyTorch framework)
• onnx, onnxruntime (For ONNX black-box interaction)
• numpy, scipy, matplotlib

Execution

1. Install requirements using pip install torch torchvision onnx onnxruntime scipy numpy matplotlib.
2. Execute the blackbox_neural_network_hardening_poc.ipynb notebook.
3. The script handles downloading required pre-trained weights, creating the BlackBox API object, launching WGAN-GP data synthesis, running distillation, executing adversarial attacks, and producing the CSV/JSON result tables automatically.