Comprehensive study of SIR: Leading SUT repository for software testing

Abstract

Software testing is performed by executing test cases against the system under test (SUT). For researchers, it is slightly challenging to find real case SUT. As an alternative, researchers utilize the existing SUT from repositories as their research objects. Software-artifact Infrastructure Repository (SIR) provides some reasonably comprehensive SUTs that had been used by over 50% of software testing literature. Nevertheless, it is still rare among researchers to conduct profound studies of SIR’s SUTs. This paper aims to fill the gap by explaining SIR’s SUTs specification and how it’s organized. Each of the C-language SUT was exposed and recorded its features comprising the size of SUT, lines of code (LOC), number of test cases, as well as the number and types of faults. SIR’s SUTs divided into two parts namely “older” and “newer” objects which both provided with various versions of the test program as well as predefined test sets along with its specification. These results indicate that SIR’s SUTs is carefully prepared to conduct software testing research.