Content area
The improvement and adoption of security-focused static analysis tools have significantly improved the detection of vulnerabilities, such as crypto-API misuse and data leaks. We are continuously becoming more dependent on these security analysis techniques because of their convenience automation, continuous integration and development support, and statically finding vulnerabilities efficiently.
However, there is a critical gap in these tools' practical and effective application. Other than static benchmarks, we have yet to devise a mechanism to identify previously unknown flaws in these tools. Furthermore, how industry professionals perceive these tools and their limitations is unknown. As a result, the current progress towards designing and developing effective, practical static analysis-based security tools is hindered.
To address these gaps, we (1) contextualize mutation testing techniques by proposing and implementing a framework called μSE. μSE systematically evaluates static analysis-based data-leak detectors, identifying previously unknown soundness issues/flaws and exploring the propagation of 25 found flaws that may propagate or even resurface, across the lifecycle of three data leak detectors, due to implicit dependencies, assumptions, or similar design principles. Next, (2) we evaluate cryptographic API misuse detectors (crypto-detectors). To do this, we create a taxonomy of crypto-API misuse based on the existing state-of-the-art literature and documentation from industry sources spanning over the past 20 years. By analyzing the patterns of underlying crypto-APIs, we develop mutation operators and mutation scopes for creating mutations of crypto-API misuse. An implementation of this approach, namely MASC, is used to systematically evaluate 14 prominent crypto-detectors from industry and academia, finding 25 previously unknown flaws affecting these crypto-detectors. Based on our discussion with the developers of the crypto-detectors about the nature of the found flaws, we identify and highlight the need to shift from a technique-centric to a security-centric approach to address evolving software security challenges. Afterward, (3) we study the gap that exists in the design and adoption of static analysis-based security tools. Through interviews with 20 real-world practitioners, we analyze their perceptions, expectations, and challenges with SAST tools. By applying thematic analysis, we identify critical insights into developer needs and discuss areas for improvement in SAST design and development.
Finally, we qualitatively analyze a statistically significant sample of existing bug reports of open-source static analysis based security testing tools to identify the internal, implicit factors that influence the acknowledging, addressing, and prioritization of the reported issues as bugs and/or feature requests, and identify the conflicting perspectives of designers and developers stemming from the duality of the vulnerability detectors; as security-assurance tools and developer-friendly tools.
