Can the best defense be a good offense? Evolving (mimicry) attacks for detector vulnerability testing under a ‘black-box’ assumption

This thesis proposes a 'black-box' approach for automating attack generation by way of Evolutionary Computation. The proposed 'black-box' approach employs just the anomaly rate or detection feedback from the detector. Assuming a 'black-box' access in vulnerability testing presents a scenario different from a 'white-box' access assumption, since the attacker does not possess sufficient knowledge to constrain the scope of the attack. As such, this thesis contributes by providing a 'black-box' vulnerability testing tool for identifying detector weaknesses and aiding detector research in designing detectors which are robust against evasion attacks.

The proposed approach focuses on stack buffer overflow attacks on a 32-bit Intel architecture and aims to optimize the various characteristics of the attack. Three components exist in a common stack buffer overflow attack: the shellcode, NoOP and return address components. Therefore, automation of attack generation is realized in three stages: (1) identifying the suitable NoOP and return address components, (2) designing the shellcode at the assembly level, and (3) designing the shellcode at the system call level. The first and second stage address the evasion of misuse detectors by employing obfuscation, whereas the third stage addresses the evasion of anomaly detectors by employing mimicry attacks.

In short, the proposed approach takes the form of a 'black-box' search process where the attacks are rewarded according to two main criteria: (a) their ability to carry out the malicious intent, while (b) minimizing or eliminating the detectable attack characteristics. Furthermore, it is demonstrated that there are two parts to buffer overflow attacks: (i) the preamble and (ii) the exploit. Therefore, the anomaly rate of the whole attack is calculated on both parts. Additionally, the proposed approach supports multi-objective optimization, where multiple characteristics of attacks can be improved. The proposed approach is evaluated against six detectors and four vulnerable applications. The results show that attacks which the proposed approach generates under a 'black-box' assumption are as effective as the attacks generated under a 'white-box' assumption adopted by previous work.