Content area
Full Text
Abstract: As software systems have become essential to critical infrastructure and private information security, software weakness detection becomes a necessity to reduce the risk of exploitation by attackers. By experimenting on carefully-designed test suites, we explored the effectiveness of several static analysis tools that detect some of the most prevalent software weaknesses. We downloaded the Juliet Test Suite for Java from Software Assurance Reference Dataset (SARD) to test the selected static analysis tools. This test suite was created based on the Common Weakness Enumeration (CWE), a community-driven database of common software weaknesses. In an attempt to cover a large portion of developers, we chose tools that were open source, and written for Java. Specifically, we tested CodePro Analytix, JLint, FindBugs, and VisualCodeGrepper. We found a continuum of success among the tools, both in accuracy and in coverage of planted weakness detection. This result emphasizes the importance of the CWE in evaluating the performance of static analysis tools, and shows that there is room for improvement in the effectiveness of these tools. Although it is possible to eliminate weaknesses in code and make programs more efficient, static analysis does not guarantee the absence of runtime errors. Additionally, static analysis tool performance should be tested on specific weaknesses in both human-generated code and computer-generated code, to determine whether accuracy and coverage are comparable. The Juliet Test Suite that we used is comprised of computer-generated code, which may not have the same characteristics as human-generated code. Thus, further research is necessary to equally examine the weaknesses in human-generated code. We tested a handful of common weaknesses, but in order to show the true coverage of a static analysis tool, representative code for more CWEs should be tested.
Keywords: static analysis, Juliet test suite, CWE, software vulnerability
(ProQuest: ... denotes formulae omitted.)
1. Introduction
Attackers have been exploiting the many of the same weaknesses in software for decades. The exploits change to suit the platform or specific implementation of a vulnerability, but the underlying weakness is the same, and the work to exploit a vulnerability is largely the same. Buffer overflows, SQL injection, etc. are the low hanging fruit for would-be attackers. Even though the software development process has evolved a lot, the vulnerabilities mapping back to...