Can the best defense be a good offense? Evolving (mimicry) attacks for detector vulnerability testing under a ‘black-box’ assumption

Kayacik, Hilmi Gunes.   Dalhousie University (Canada) ProQuest Dissertations Publishing,  2009. NR50074.

Abstract (summary)

This thesis proposes a 'black-box' approach for automating attack generation by way of Evolutionary Computation. The proposed 'black-box' approach employs just the anomaly rate or detection feedback from the detector. Assuming a 'black-box' access in vulnerability testing presents a scenario different from a 'white-box' access assumption, since the attacker does not possess sufficient knowledge to constrain the scope of the attack. As such, this thesis contributes by providing a 'black-box' vulnerability testing tool for identifying detector weaknesses and aiding detector research in designing detectors which are robust against evasion attacks.

The proposed approach focuses on stack buffer overflow attacks on a 32-bit Intel architecture and aims to optimize the various characteristics of the attack. Three components exist in a common stack buffer overflow attack: the shellcode, NoOP and return address components. Therefore, automation of attack generation is realized in three stages: (1) identifying the suitable NoOP and return address components, (2) designing the shellcode at the assembly level, and (3) designing the shellcode at the system call level. The first and second stage address the evasion of misuse detectors by employing obfuscation, whereas the third stage addresses the evasion of anomaly detectors by employing mimicry attacks.

In short, the proposed approach takes the form of a 'black-box' search process where the attacks are rewarded according to two main criteria: (a) their ability to carry out the malicious intent, while (b) minimizing or eliminating the detectable attack characteristics. Furthermore, it is demonstrated that there are two parts to buffer overflow attacks: (i) the preamble and (ii) the exploit. Therefore, the anomaly rate of the whole attack is calculated on both parts. Additionally, the proposed approach supports multi-objective optimization, where multiple characteristics of attacks can be improved. The proposed approach is evaluated against six detectors and four vulnerable applications. The results show that attacks which the proposed approach generates under a 'black-box' assumption are as effective as the attacks generated under a 'white-box' assumption adopted by previous work.

Indexing (details)

Computer science
0984: Computer science
Identifier / keyword
Applied sciences; Attack generation; Stack buffer overflow attacks; Vulnerability testing
Can the best defense be a good offense? Evolving (mimicry) attacks for detector vulnerability testing under a ‘black-box’ assumption
Kayacik, Hilmi Gunes
Number of pages
Degree date
School code
DAI-B 70/08, Dissertation Abstracts International
Place of publication
Ann Arbor
Country of publication
United States
Dalhousie University (Canada)
University location
Canada -- Nova Scotia, CA
Source type
Dissertation or Thesis
Document type
Dissertation/thesis number
ProQuest document ID
Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works.
Document URL