Model evasion and adversarial inputs¶

Evasion represents an attack performed at inference time against a trained and deployed machine learning model. The attacker leaves model weights and training datasets untouched, focusing instead on crafting inputs designed to trigger misclassifications by exploiting how the model evaluates data.

What makes evasion unique to learned models is the operational gap between human assessment and machine classification. While written rules fail in predictable ways that analysts can read directly, a machine learning classifier fails along a complex decision boundary that human operators cannot fully visualise. An attacker who places an input just beyond that boundary produces a payload that remains malicious in execution while scoring as benign, allowing both properties to co-exist.

Deployed classification targets¶

Common classification targets vulnerable to evasion include:

• Malware and file classifiers scoring uploads before sandbox execution.

• Moderation systems filtering user-generated content.

• Fraud scoring engines evaluating financial transactions.

• Risk engines assessing login behaviour and device telemetry.

• Spam and phishing filters scanning message text and header metadata.

In each scenario, the model outputs a classification label alongside a confidence score, which downstream systems process without re-evaluating the underlying data. Evasive inputs target that automated handoff.

Boundary probing and evasion techniques¶

Attackers manipulate decision boundaries through several distinct methods:

Perturbing malicious files allows executables to bypass feature-based classifiers. Probing a classifier feature by feature enables attackers to append benign sections, pad files with clean strings, or reorder content. These modifications alter the classifier’s assessment without changing how the file executes, shifting the sample’s score across the decision boundary.

Black-box probing enables boundary mapping without access to model weights. By submitting multiple input variations and tracking which pass or fail, attackers map local decision boundaries to phrase malicious content within permitted parameters. This technique requires query access rather than internal gradient visibility.

Feature-space evasion against fraud models targets specific behavioural metrics, such as transaction velocity, transfer amounts, device age, or time of day. Shaping transactions to remain below individual feature thresholds prevents the overall transaction from triggering fraud alerts.

Transferable adversarial examples leverage surrogate models. Attackers unable to query target models heavily train stand-in models on similar datasets. Adversarial inputs crafted against stand-in models frequently transfer to target models, allowing attackers to approximate private boundaries without direct access.

Semantic evasion circumvents standard input validation. Format checks verify that inputs are well-formed and free from traditional injection attacks. Evasive inputs pass these checks because they represent valid data structures, such as real images or valid financial transactions, placing the misclassification within statistical evaluations rather than format checks.

Confidence shaping avoids human review queues. Security architectures often route only low-confidence classifications to human analysts. Nudging an evasive input toward high confidence on a benign label bypasses manual review entirely.

Military and geopolitical applications¶

Military and intelligence operations leverage evasion techniques where differences between human perception and automated classification create tactical advantages.

Adversarial patterns on reconnaissance imagery¶

Research into texture- and shape-based attacks on overhead imagery shows that adversarial textures applied to a vehicle defeat the detectors that scan aerial and satellite pictures for it. The vehicle is not relabelled as something harmless; it stops being detected at all. The study also prices the trade-off that decides whether any of this is wearable in the field: unconstrained textures work best, and the constrained ones that can actually be painted or printed, at low resolution, over a limited area, in a restricted palette, give up performance for practicality. Shape changes alone do less, though shape and texture together approach the unconstrained result.

Decoy systems and adversarial patches¶

Research into AI-based military target detection shows that lightweight decoys fitted with heat sources can use surface patches to mimic high-value missile systems. Opposing classifiers prioritise targeting the decoy while genuine assets lacking the patch register as low-priority targets, manipulating automated targeting logic without deceiving human observers.

Evading detection across visible and thermal sensing¶

A survey of physical adversarial attacks on surveillance systems outlines physical perturbations designed to defeat object tracking, including patterned clothing and dual-modal attacks operating across visible and thermal spectrums. Operating directly in front of the sensor eliminates network footprint and leaves no digital intrusion traces.

Surface modification and treaty verification¶

Modifying surface textures on missile silos alters features extracted by automated image classifiers, causing systems to classify launch sites as featureless terrain. Automated treaty-monitoring platforms then undercount active silos, while surface modifications remain plausible as routine maintenance.

Adversarial camouflage in the Russia-Ukraine conflict¶

Russian armoured vehicles in the Russia-Ukraine conflict have featured prominent geometric patterns in orange and black with limited visual camouflage value. Analysts suggest these designs align with adversarial patch concepts aimed at defeating automated target classifiers rather than human eyes, though specific targeting mechanisms remain debated. Regardless of intent, the conflict demonstrates evasion as a live operational factor.

Retraining classifiers to recognise new patterns prompts attackers to deploy updated variations, creating a continuous cycle with no stable end state.

AI-generated phishing and malware evasion¶

Evasion extends beyond physical image domains. The GREYVIBE group, linked to Russian state interests, uses commercial language models like ChatGPT and Google Gemini to generate phishing lures and malware variants. Outsourcing payload generation to AI produces novel phrasing and code structures that target classifiers have not encountered during training, enabling cheap variant generation that exceeds classifier coverage.

Autonomous malware navigating device UI¶

PromptSpy, an Android backdoor, uses the Gemini API to navigate victim device interfaces in real time. The malware generates touch inputs and navigation commands to achieve objectives and avoid uninstallation by manoeuvring around removal options. Resembling human user interaction allows the malware to bypass heuristic process monitoring.

Generation artefacts in machine-written exploits¶

Google Threat Intelligence Group reported a case of threat actors using a language model to produce a functional zero-day exploit. The script carried a hallucinated CVSS score and textbook docstrings, which read as indicators of machine generation. Those elements were not operational failures but routine artefacts of the generation process, and the same artefacts gave the exploit’s synthetic origin away.

Hardening systems against boundary probing¶

Defending classification pipelines requires specific operational controls:

Evaluating models against deliberate adversarial pressure rather than relying solely on clean validation datasets. High accuracy on held-out test data does not guarantee resilience against targeted boundary manipulation.

Treating confidence scores as potentially manipulated metrics rather than absolute safety measures. Input scores near or far from decision thresholds require scrutiny when originating from untrusted sources.

Retaining human analysts within critical decision workflows, positioning model outputs as supporting inputs rather than final verdicts. Removing automated single-step processing eliminates the leverage evasive inputs exploit.

Monitoring for clustered, near-boundary inputs originating from single sources to identify boundary probing before attackers deploy finalised evasive payloads.