Breaking & Fixing Dynamic Token Reweighting (CVPR 2026) in Vision-Language Models (`LLaVA-Llama-2-7B`) against HADES Typographic & White-Box Adaptive PGD Jailbreaks.
| Evaluation Suite | Defense Method | Attack Success Rate (ASR ↓) | Total Defense Rate (↑) | Explicit Refusal (↑) | Avg Latency (ms ↓) | Speedup & Scene Fidelity |
|---|---|---|---|---|---|---|
| HADES S+T+A (Typographic + Adv) | No Defense (Baseline) | 44.0% | 56.0% | 12.0% | 0.0 ms | Baseline (78.0 words) |
| HADES S+T+A (Typographic + Adv) | Vanilla DTR (10-Step 1D Vector) | 48.0% | 52.0% | 16.0% | 851.07 ms | 1.0x (Misses bottom typography: 0% evicted) |
| HADES S+T+A (Typographic + Adv) | Ortho-Opt (10-Step Rank-4 SVD) | 8.0% | 92.0% | 72.0% | 827.30 ms | 1.03x (Over-evicts: α=0 collapse on benign) |
| HADES S+T+A (Typographic + Adv) | Ortho-FastTaylor (Ours, 1-Step) | 12.0% (4x Lower ASR) | 88.0% | 68.0% (+52% Gain) | 107.93 ms | 7.9x Faster | True Scene Preserved (35.2% evicted) |
| HADES S+A (Visual + Adv) | Vanilla DTR (10-Step 1D Vector) | 24.0% | 76.0% | 76.0% | 846.44 ms | 1.0x |
| HADES S+A (Visual + Adv) | Ortho-FastTaylor (Ours, 1-Step) | 0.0% | 100.0% | 100.0% | 111.49 ms | 7.6x Faster | 100% Refusal Rate |
| White-Box Adaptive PGD Red-Team | Vanilla DTR (Under Orthogonal PGD) | 40.0% | 60.0% | 60.0% | 1436.70 ms | 1.0x (Bypassed by typography-preserving PGD) |
| White-Box Adaptive PGD Red-Team | Ortho-FastTaylor (Ours, Under PGD) | 0.0% | 100.0% | 100.0% | 280.84 ms | 5.1x Faster | 100% Robust Against Adaptive PGD |
1. Forward-Hook Accumulation Leak (HADES_LLaVA_Llama2_with_DTR.py lines 88 & 101): Vanilla DTR registers two forward hooks on vision_tower per sample and only removes one, causing $N$ hooks to stack across $N$ dataset items ($\alpha^N$ multiplicative feature collapse). Fixed via our context-managed VisionScalingHook.
2. Unnormalized Gradient Blindspot on Small Typography: Because $\nabla_{\alpha_i} \mathcal{L}$ scales with patch feature norm $\|v_i\|_2$, high-contrast objects in the visual scene dominate the gradient, leaving low-norm text banners at the bottom of the image un-evicted. Fixed by normalizing single-step Taylor attribution by $\|v_i\|_2 + \epsilon$ and applying a $1\times 5$ horizontal row-strip smoothing filter.
3. Multi-Step Over-Eviction Collapse (Why 1-Step Beats 10-Step): Running 10 steps of AdamW on subspace objectives (ortho_opt) drives all 256 patches to $\alpha=0.0$ on 76% of samples, causing LLaVA to hallucinate a default "serene green landscape" on benign queries. Our 1-step Ortho-FastTaylor evicts exactly 35.16% of patches, preserving 100% authentic scene descriptions while achieving 88%–100% defense success at 7.9x lower latency.
4. Bypassing @torch.no_grad() in CLIPVisionTower for White-Box Red-Teaming: LLaVA wraps CLIPVisionTower.forward() in @torch.no_grad(), which masks gradients from naive PGD attackers. Our GradEnabledVisionTowerHook re-enables autograd through the vision encoder and crafts a Typography-Preserving Orthogonal Decoy attack that breaks Vanilla DTR (40% ASR) while Ortho-FastTaylor achieves 0.0% ASR (100% defended).