Content area
Software is increasingly expected to operate across widely varying environments. If not handled appropriately, any differences between these environments could cause even well- tested applications to fail upon deployment. Anticipating and being prepared to handle all the possible combinations of software and hardware that can affect an application’s ability to operate is a major challenge. To make matters worse, recovering from post-deployment failures is expensive, harmful to user experience, and damaging to developer reputations.
What is needed is a way to detect situations where an application may fail before it is deployed so that its deficiencies can be corrected without the negative consequences of a crash. We observed that the causes of these environment-related failures often can be seen in the communications between an application and its environment. These communications can include the system calls the application makes or the messages it sends across a network. By taking advantage of this concept, we were able to develop two techniques that allow developers to catch environmental bugs before an application is deployed.
The first technique, known as Simulating Environmental Anomalies (SEA), allows developers to record features from one environment in which an application has failed and use them to determine if other applications are likely to fail in a similar manner. Using this technique enabled us to find many high impact bugs involving mishandling of unusual files, moving files across disks, and inappropriate network timeouts in popular, battle tested applications.
The second technique is PORT, a new domain specific programming language that uses event processing techniques to simplify procedures for recording and simulating an anomaly. PORT allows developers to quickly write expressive programs that can analyze a stream of communications between an application and its environment. PORT programs can both detect opportunities to simulate an anomaly, and modify the stream so that anomaly is present.
We tested how well PORT achieved its intended purpose by using it to recreate and replace the anomalies used in our original SEA work. These new PORT programs were shorter, easier to read, and simpler to maintain than the lengthy and complex Python scripts they replaced. We also took advantage of PORT’s extensibility to apply SEA to USB traffic. As SEA enabled us to write programs that could identify patterns in, and modify recordings of, USB communications, we were able to simulate BADUSB style attacks by modifying innocent streams so that they contained the harmful features of such an attack. We were also able to simulate device identifier conflicts – a situation that has long plagued operating system developers – by using PORT to modify a recording so these conflicts would be present in device registration frames. In both cases, these recordings could then be used to test how a system might respond should they occur in the real world.
Our hope in developing these tools and techniques was to see them widely used in improving application reliability and security. But, to do so, these tools needed to be accurate, efficient, and easy to use. To test the latter quality, we conducted a user study to see how well developers could integrate SEA into their workflows. At the conclusion of the study, we found that participants were able to find new bugs in widely deployed software, build corrective patches, and submit them to the appropriate upstream maintainers.
Unfortunately, in spite of some success finding and fixing bugs, our developers met resistance in getting their fixes incorporated. This resistance sprang from two sources: improper patch submission on the part of our participants, and a reluctance on the part of maintainers to view certain misbehaviors as bugs. The former may be dealt with by improving how we teach novices to interact with maintainers. The latter will require an effort to educate project maintainers on the value of identifying and fixing environmental bugs. Having reliable tools and techniques that can simplify the implementation of these processes should make these educational efforts easier to promote.