Content area
System software serves as one of the most fundamental cornerstones of today’s cyberspace. They operate the Internet-of-Things (IoT) devices (e.g., the firmware), factory machines (e.g., programmable logic controllers), and data centers (e.g., the Linux kernel). Their essential role of managing computing hardware and providing computing services makes them ubiquitous and highly important. However, due to their closeness to hardware and demand for efficiency, they are usually implemented in relatively low-level languages (e.g., C/C++ or assembly), which do not necessarily have the security features that are commonly available in high-level languages (e.g., Python or Java). Moreover, because of their important role, they often run in high-privilege mode. These characteristics make them highly lucrative for cyberattacks and exploitation. Thus, identifying the security risks before they are exploited by the attacker is crucial for the safety of today’s cyberspace.
In the meantime, symbolic execution and fuzz testing have been arguably the most prominent and sophisticated automated testing techniques over the past decades. Together, they have seen great success in discovering numerous software defects and vulnerabilities. However, applying them to the system software has specific challenges due to the aforementioned characteristics. Specifically, we focus on the following three challenges in this thesis.
• Some system software (e.g., the Linux Kernel) commonly suffers from undefined behaviors. At the same time, it is well-known for being complex and large in size. This challenges the scalability of the current undefined behavior detection technique due to the issue of path explosion.
• Some system software (e.g., firmware) interacts with diverse peripherals, yet they run in a highly resource-constrained environment (e.g., the MCU). This makes applying the sophisticated testing techniques to them extremely difficult or inefficient.
• Besides the second challenge above, there are often cases where we only have access to the binary code. For example, MCU vendors often provide binary-only libraries or drivers for the user to interact with their hardware (e.g., proprietary device drivers). The lack of semantics of the source code poses difficulties to the testing techniques.
I tackled these three challenges with three novel and efficient solutions. First, to address the scalability problem, I present KUBO, a static undefined behavior detector for OS kernels. To mitigate the scalability issue of the path explosion, we first studied fifty-five undefined behavior bugs in the Linux kernel. Then, we proactively prune the paths as we decide the triggerability of the identified bugs through linking them to user inputs. As a result, we identified 24 new undefined behaviors bugs in the Linux kernel. Second, to address the inefficiency issue of testing firmware, I present CO3, a novel concolic execution framework. CO3 tackles the inefficiency of the existing works that primarily use debugging interfaces to communicate program states between the MCU and the workstation. In replacing the debugging interface with CO3’s proposed protocol, we improved the speed of the concolic executing the firmware by three orders of magnitude. Third, I present DRIFT, which focuses on testing the binary-only firmware. Through utilizing hardware breakpoints and interrupts, along with other commonly available hardware features, we are able to fuzz test firmware in the native hardware environment, outperforming the state-of-the-art with no emulation or instrumentation.