Reliability model of open source software considering fault introduction with generalized inflection S-shaped distribution

Abstract

Recently, the open source software (OSS) reliability has become one of hot issues. Owing to the uncertainty and complexity of OSS development, testing and debugging environments, OSS are completed dynamically. When detected faults are removed for OSS, they are likely to introduce new faults. Moreover, under the different OSS debugging environments, fault introduction will show different changes. For example, the fault introduction rate shows a decrease change, or increasing first and then decreasing change over time. Considering the complex and dynamic changes in fault introduction, an OSS reliability model that fault introduction obeys a generalized inflection S-shaped distribution is proposed in this paper. Experimental results indicate that the fitting and predictive performance of the proposed model is good. The established model in this paper can adapt the dynamical and complicated changes of fault introduction during OSS debugging. Moreover, the established model can accurately forecast the number of remaining faults in OSS, and assist developers to evaluate the actual OSS reliability.

Article highlights

An OSS reliability model considering that fault introduction obeys GISS distribution is proposed.

The developed model can be used to assess the reliability of OSS under complicated environments.

The developed model can get with the complex testing and development environment of OSS.

Introduction

During the last several decades, OSS has been gradually accepted and used by people. The development mode of OSS is quite different from that of closed source software (CSS). CSS has hierarchical management in the development and testing processes, and is completed according to the pre-arranged plan and target. Therefore, developers, testers and debuggers are relatively stable during development and testing. However, OSS is developed, tested and debugged dynamically by developers, users and volunteers in the network and open environment. In order to improve the OSS reliability, the industry generally uses the method of frequent release to improve it. But there are two problems. One is that if the OSS is released too early, there will be many faults in the software. It will seriously affect the use of OSS. Second, if OSS is released too late, users and volunteers will be impatient to wait and turn to other OSS to replace it. Therefore, the reliability of OSS will be widely questioned.

In order to solve the problem of OSS reliability evaluation, some researchers have developed a few reliability models of OSS. For instance, Tamura and Yamada [1] established a reliability model of OSS using stochastic differential equation. Li et al. [2] observed that the error detection rate of OSS increased first and then decreased, and established the corresponding OSS reliability model. Yang et al. [3] built delayed OSS reliability models considering the relation between fault introduction and fault detection. Aiming at the debugging activities of OSS, Lin and Li [4] established an OSS reliability model with a rate-based queue theory. Huang et al. [5] proposed a reliability model of OSS considering fault detection with a bounded generalized Pareto distribution. Singh et al. [6] built a multi-version reliability model for OSS based on entropy and an optimal release strategy considering user and volunteer satisfaction level. Wang and Mi [7] established a reliability model for OSS considering that the fault detection rate has a decreasing change trend.

Although the reliability model for OSS can effectively assess the OSS reliability under certain open source conditions, owing to the variability and complexity of OSS development, testing and debugging environments, the established reliability model for OSS cannot fully satisfy the actual OSS reliability assessment.

In the development and testing processes of CSS, there are static testing methods, such as peer review, walk through and inspection, to ensure the quality improvement of CSS. When faults are detected, developers (or debuggers) and testers can communicate face to face and discuss the causes of faults and how to remove them. Thus, in the CSS testing environment, faults can be well described and completely removed. However, faults detected during the OSS test are passed to the developers by users and volunteers through the network, and then the developers organize personnel to remove the faults. In the open environment, users or volunteers cannot clearly describe fault information to developers, which will cause developers to remove the fault incompletely and new faults are introduced. Moreover, in the fault tracking system of OSS, the fault state will be modified from closed to reopened. This indicates that the removed faults have not been completely eliminated, or new faults may have been introduced. Therefore, in the process of reliability modeling of OSS, it certainly needs the reasonable and effective study of the phenomenon of fault introduction in the OSS debugging process.

Considering the complexity and non-linear changes of fault introduction during OSS debugging, the fault introduction rate will show the change of decreasing, or increasing first and then decreasing over time, etc. Therefore, assuming that fault introduction obeys a single law of change, it does not accord with the actual situation of fault introduction in the process of OSS debugging. The reliability model of OSS established by this method cannot satisfy the actual reliability assessment of OSS. At least, the adaptability of the reliability model of OSS established by this method is very poor. It cannot adapt to the complex reliability assessment of OSS.

Additionally, because OSS is very different from CSS in the testing and development processes, the software is evolved through the active participation of volunteers and users located at different geographical locations. Moreover, each release of OSS will add some features, functions or components compared with the previous version. Those changes in OSS lead to the complexity of fault introduction. One is the introduction of new faults for fault removal in the current version of software, and the other is the newly generated faults caused by newly added features, functions and components. When the newly generated fault is removed, a new fault will also be introduced. These two kinds of introduced faults make the debugging of OSS complex. Considering those characteristics, we can use the GISS distribution to simulate the complex changes of introduced faults for OSS. Since the GISS distribution can well simulate the complex changes of introduced faults for OSS, the proposed model in this paper is different from the CSS reliability model which fault detection obeys the GISS distribution in the existing literature [8]. Thus, the established model can fully get with the OSS testing and development environment, and effectively carry out residual fault prediction and software reliability evaluation in OSS.

In this paper, we develop an OSS reliability model considering that fault introduction obeys a generalized inflection S-shaped (GISS) distribution. Assuming that the fault introduction obeys the GISS distribution, the fault introduction rate will show a variety of complex nonlinear changes, for example, the fault introduction rate increases first, then decreases over time; the fault introduction rate decreases or increase over time, and linear changes, for instance, the fault introduction rate is a constant. The OSS reliability model established by this method will have strong adaptability and robustness, and it can adapt to the fault introduction changes during development, testing and debugging of various OSS. Therefore, the established model can be used to assess the OSS reliability.

The contributions of this paper are as follows,

1. To the best of our knowledge, it is the first to propose that fault introduction obeys GISS distribution.

2. The proposed model can adapt to the complicated changes of fault introduction in the process of actual OSS development, testing and debugging.

3. The proposed model can be used to assess the reliability of actual OSS projects.

4. We solve the problem of establishing software reliability model with various fault introduction changes under complex open source environments.

The structure of this paper is organized as follows,

Related work is introduced in Sect. 2. Section 3 presents the fault introduction rate with the GISS distribution, and introduces the proposed model. Section 4 introduces OSS fault data sets, model comparison criteria and parameter estimation method of the proposed model. Experiments of model performance comparison are done in Sect. 4. Sensitivity analysis of the proposed model parameters is done in Sect. 5. The implication of the study is discussed in Sect. 6. Section 7 is threats to validity of the developed model. Last section is conclusions on this paper.

Related work

In the past of decades, the reliability of OSS has become one of the hot issues in the industry. “Release early, release often.” [9] has become a way for developers to improve the OSS reliability. But it faces two problems: (1) if the software is released too early, there will be too many faults in the software, and users will lose interest in using the software. As a result, the software cannot be widely tested and used. (2) If the software is released too late, users will use other open source software to replace the software. After the software was released, it was not used, tested and abandoned.

By studying the change rule of fault data of OSS, researchers propose that the reliability model of CSS can also be applied to evaluate the OSS reliability. For example, Zou and Davis [10] proposed that the reliability models of CSS can evaluate the OSS reliability, and the reliability model of CSS based on the Weibull distribution has the better fitting and predictive performance than other CSS reliability models. Anbalagan and Vouk [11] proposed that traditional reliability models can evaluate OSS reliability by studying problem reports. Similarly, Rossi et al. [12] study bug repositories and think that the reliability of OSS can be evaluated by traditional CSS reliability models. Chiu et al. [13] proposed a software reliability model in consideration of learning effects of fault detection. Okamura et al. [14] studied the distribution of failure time and established a software reliability model based on the normal distribution. Additionally, considering the weighted value changes of support vector regression (SVR), Utkin and Coolen [15] proposed a corresponding software reliability model. Wang and Zhang [16] used a deep learning method with RNN encoder-decoder to predict software reliability. Ke and Huang [17] developed software reliability models based on changing points of testing efforts.

Because the development and testing environment of OSS is completely different from that of CSS, the method of evaluating the reliability of OSS by using CSS reliability model is widely questioned. Therefore, researchers have proposed some reliability models of OSS. For instance, Tamura and Yamada [18] established an OSS reliability model of the logarithmic Poisson execution time and used the Analytic Hierarchy process to estimate the model parameters. Gyimothy et al. [19] used various object-oriented metrics to study fault prediction for OSS. Tamura and Yamada [20] used neural networks to build OSS reliability model and gave an optimal software release method. Syed-Mohamad and Mcbride [21] indicate that the testing and development process of each OSS is different, and the reliability model can be established according to the specific situation. Tamura and Yamada [22] developed an OSS reliability model using the deterministic chaos theory.

Singh et al. [23] built a reliability model of OSS considering different kinds of faults in the fault data sets. Singh et al. [24] proposed an OSS reliability model based on change-point. Considering the dynamic changes of OSS development and testing, some researchers use a random differential equation to build the corresponding OSS reliability model [1, 25–27]. Considering the frequent release of OSS, some researchers proposed a few reliability models of multi-version or multi-release OSS [6, 28–30]. Tamura et al. [31] proposed a reliability model of open source cloud computing based on jump diffusion. Tamura [32] proposed a reliability model of open source cloud computing considering the irregular fluctuation of the fault detection rate. Tandon et al. [33] developed multi-release OSS reliability models based on entropy. Through studying changing-point changes, Kapur et al. [34] developed two-dimensional OSS reliability models. Ivanov et al. [35] used the goal question metric (GQM) method to evaluate and forecast software reliability under open source situation considering mobile operating environments. Additionally, Barack and Huang [36] used a few software reliability models to evaluate and forecast the reliability of OSS. Wang [37] proposed an OSS reliability model taking account into fault introduction based on the Pareto distribution.

Yang et al. [38] built a reliability framework based on a change point for OSS using masked data, and used expectation maximization algorithm to solve the likelihood function as estimating model parameter values. Considering imperfect debugging and changing-point, Saraf et al. [39] developed a multi-release framework on fault correction and fault detection for OSS. Sun and Li [40] proposed a few software imperfect debugging models taking into account fault introduction and fault levels.

Owing to the complexity and dynamic change of the OSS test and development, software reliability model cannot be used in all OSS reliability assessment. Thus, Ullah and Morisio [41] used the optimal selecting model method to determine which software reliability model is used to estimate the current OSS reliability. Considering the complexity of fault detection during software development and tests, Raghuvanshi et al. [42] established a time-variant software reliability model. In addition, considering the dynamic changes of OSS testing and development processes, Saraf et al. [43]. developed multi-release reliability models with imperfect debugging and change-point for OSS. To evaluate the OSS reliability in various complex situations, we develop an OSS model which can adapt to various open source environments and considers fault introduction.

Building OSS reliability model

Fault introduction with a generalized inflection S-shaped distribution

Suppose that the fault introduction follows the GISS distribution, then the fault introduction rate can be expressed as follows,

1

$$\omega (t)=\frac{f(t)}{1-F(t)}$$

2

$$F(t)=\frac{1-exp\left(-\alpha t^d\right)}{1+\beta exp\left(-\alpha t^d\right)},\alpha >0,d>0,\beta \ge 0$$

3

$$f(t)=\frac{\text{d}F\left(t\right)}{\text{d}t}=\frac{\left(1+\beta \right)\alpha \text{d}{t}^{d-1}exp\left(-\alpha t^d\right)}{{\left(1+\beta exp\left(-\alpha t^d\right)\right)}^2}$$where $\omega (t)$ presents the fault introduction rate function. F(t) denotes the GISS distribution. d represents a shape parameter. $\alpha$ and $\beta$ are scale parameters.

In general, two main fault types for open source software are stochastically dependent and mutually independent faults. When independent faults are removed, the possibility of introducing new faults is relatively small. When interdependent faults are removed, it is more likely to introduce new faults, or they cannot be completely removed. In other words, in the debugging process of open source software, the possibility of introducing new faults when simple faults are removed is less than that when complex faults are removed. Thus, we can establish the following differential equation,

4

$$\left\{\begin{array}{c}\omega (t)=\alpha \left[\gamma +(1-\gamma )F(t)\right]\\ \omega (t)=\frac{f(t)}{1-F(t)}=\frac{\frac{\text{d}F(t)}{\text{d}t}}{1-F(t)}\\ \end{array}\right)$$where $\gamma$ and $(1-\gamma )$ denote the fractions of introducing faults in terms of the first and second types, respectively.

Solving the Eq. (4), we can derive the following distribution of fault introduction.

5

$$F(t)=\frac{1-exp(-\alpha t)}{1+\frac{1-\gamma }{\gamma }exp(-\alpha t)}=\frac{1-exp(-\alpha t)}{1+\beta exp(-\alpha t)},\alpha >0,0<\gamma \le 1,\beta =\frac{1-\gamma }{\gamma }\ge 0$$where $\beta$ is called as an inflection factor.

In the debugging process of open source software, software complexity is also an important factor affecting fault introduction. Herein, software complexity refers to the complexity of software itself and the complexity of software debugging environment. The former includes the complexity of software scale, algorithm complexity, logic complexity, architecture complexity, parallel storage and parallel computing, etc. The latter refers to the complexity of the debugger's personnel, the complexity of the debugging resources owned by the debugger, and the complexity of the debugger's debugging environment, etc. Because open source software is mainly completed by developers, users and volunteers, there is no fixed debugging personnel. The detected faults are randomly assigned to debuggers by developers. The skills, resources and experience of debuggers will affect the introduction of faults when detected faults in open source software are removed. Therefore, we can use a function to express the influence of software complexity on fault introduction in the debugging process of open source software, such as $\phi (t)$, that is assumed to be integrable in (0,t) and nonnegative, denotes the effect on $\omega (t)$ for software complexity.

In consideration of minimizing the influence of too many parameters for $\phi (t)$, we use two parameters for software complexity function $\phi (t)$. Because the more parameters a function has, the more complex and difficult it is to estimate parameter values. We use the simple power-law function $\phi (t)=d{t}^{d-1}$, which a good tradeoff is provides between flexibility and simplicity. Thus, we can extend Eq. (4) as follows,

6

$$\left\{\begin{array}{c}\omega (t)=\alpha \left[\gamma +\left(1-\gamma \right)F(t)\right]\phi (t)\\ \omega (t)=\frac{f(t)}{1-F(t)}=\frac{\frac{\text{d}F(t)}{\text{d}t}}{1-F(t)}\\ \phi (t)=\text{d}{t}^{d-1}\\ \end{array}\right)$$Solving the Eq. (6), we can derive the following distribution of fault introduction considering software complexity.

7

$$F(t)=\frac{1-exp\left(-\alpha t^d\right)}{1+\frac{1-\gamma }{\gamma }exp\left(-\alpha t^d\right)}=\frac{1-exp\left(-\alpha t^d\right)}{1+\beta exp\left(-\alpha t^d\right)},\alpha ,d>0,0<\gamma \le 1,\beta =\frac{1-\gamma }{\gamma }\ge 0$$where d is a shape parameter. Despite Eq. (7) is a simple structure, it is very flexible considering that fault introduction obeys the GISS distribution.

Substituting (2) and (3) into (1), then

8

$$\omega (t)=\frac{\alpha d{t}^{d-1}}{1+\beta exp(-\alpha t^d)}$$where in Eq. (8), $\alpha$ represents the scale parameter. d is the shape parameter. $\beta$ denotes an inflection factor.

In Eq. (8), when t goes to infinity, $\omega (t)=\alpha d{t}^{d-1}$. Thus, when $t\to \infty$ and d < 1, $\omega (t)=0.$ When $t\to \infty$ and d = 1, $\omega (t)=\alpha d.$ When $t\to \infty$ and d > 1, $\omega (t)$ tends to infinity.

In Fig. 1a–c), the fault introduction rate $\omega (t)$ shows the complex changes when parameters d and $\beta$ change over time, respectively. From Fig. 1a, we can see that $\omega (t)$ tends to zero when d < 1 and $t\to \infty$. In Fig. 1a, when $\beta =100$, the fault introduction rate function first increases and then decreases over time. In Fig. 1b, when d = 1, $\omega (t)$ tends to the constant. In Fig. 1c, when d > 1, $\omega (t)$ tends to infinity. Moreover, when $\beta =100$, the fault introduction rate function shows an S-shaped change over time in Fig. 1c.

Fig. 1 [Images not available. See PDF.]

The changes of the fault introduction rate $\omega (t)$ over time

Fault introduction which obeys the GISS distribution can represent a variety of complex changes, and can get with the complicated changes of actual fault introduction in the OSS debugging process. Thus, fault introduction with the GISS distribution is consistent with the situation of fault introduction in the actual OSS debugging process. The reliability model of OSS which fault introduction obeys the GISS distribution can fully satisfy the actual OSS reliability evaluation.

Proposed model

The assumptions of the proposed model are as follows,

1. The fault detection process obeys the non-homogeneous Poisson process (NHPP).

2. The number of fault detected in $(t,t+\mathrm{\Delta }t)$ is proportional to the number of remaining faults in the software.

3. In the process of open source software debugging, new faults can be introduced when detected errors are eliminated.

4. Fault introduction obeys the GISS distribution.

5. The number of introduced faults is relevant to that of detected faults.

From Assumptions 1 and 2, the differential equations can be derived as follows,

9

$$\frac{\text{d}\mu (t)}{\text{d}t}=\theta \left(\psi \left(t\right)-\mu \left(t\right)\right)$$where $\mu (t)$, $\theta$ and $\psi (t)$ denote the mean function and the content function, respectively. $\theta$ denotes the fault detection rate.

From Assumptions 3, 4 and 5, the following equation can be derived,

10

$$\psi (t)=\omega (t)\mu (t)+\eta$$where $\omega (t)$ denotes the fault introduction rate function, and $\eta$ presents the expected number of initially detected faults.

Substituting (4) and (6) into (5), we can derive the following equation,

11

$$\mu \left(t\right)=\frac{\eta exp\left(-\theta t+\theta \alpha t^d\right){\left(1+\beta exp\left(-\alpha t^d\right)\right)}^{\theta }\left(exp\left(\theta t\right)-1\right)}{{\left(1+\beta \right)}^{\theta }}$$Formula (11) is the developed model expression. Please see the Appendix for detailed derivation process.

Numerical examples

Illustrations of fault data sets, comparison models and model comparison criteria

In this paper, we collected three fault data sets from three projects of Apache products of OSS. Its website is https://issues.apache.org. Each OSS fault data set includes three successive fault data subsets. Please see Table 1 for details of the fault data sets. Note that detected faults of OSS are stored in the bug tracking system. Resolution of faults in the bug tracking system includes FIXED, INVALID, WONTFIX and DUPLICATE, etc. We remove faults whose resolutions are INVALID, WONTFIX and DUPLICATE, and the remaining faults are collected in our OSS fault data sets.

Table 1. OSS fault data sets

	Fault datasets	Cumulative number of detected faults	Total test time	The time period of fault data sets collection
DS1	AIRFOLW 1.10.1 (DS1-1)	124	133 weeks	From August 2016 to February 2019
	AIRFOLW 1.10.2 (DS1-2)	224	155 weeks	From June 2016 to May 2019
	AIRFOLW 1.10.3 (DS1-3)	238	172 weeks	From June 2016 to June 2019
DS2	Jena 3.6.0 (DS2-1)	31	81 days	From May 2013 to December 2017
	Jena 3.7.0 (DS2-2)	53	32 weeks	From September 2017 to April 2018
	Jena 3.8.0 (DS2-3)	49	44 months	From October 2012 to June 2018
DS3	CLI 1.1 (DS3-1)	32	58 months	From November 2002 to June 2007
	CLI 1.2 (DS3-2)	18	32 months	From October 2005 to March 2009
	CLI 1.3 (DS3-3)	36	77 months	From July 2008 to November 2014

To fully verify the performance of the developed model, we compared the proposed model with other models using six model comparison standards [44] and five software reliability models. Six model comparison criteria include the mean square error (MSE), R², RMSE, TS, Bias and AIC. Five software reliability models include the G–O model, Weibull distribution model, generalized inflection S-shaped (GISS) model, Wang model and Li model. The G–O model, Weibull distribution model and generalized inflection S-shaped (GISS) model are CSS reliability models. The Li model and Wang model are reliability models for OSS. Tables 2 and 3 list the detailed information on model comparison criteria and comparison models used in this paper, respectively. Note that the G–O model is used as a comparison model. We mainly consider that it is a classical CSS reliability model with a concave curve.

Table 2. Illustrations of model comparison criteria

Number	Model metrics	Description or summary
1	$\text{MSE}=\frac{{\sum }_{i=1}^{\kappa }{\left(\mu \left(t_i\right)-{\mu }_i\right)}^2}{\kappa }$	where $\mu (t_i)$ represents the cumulative number of detected faults. ${\mu }_i$ denotes the number of actually observed faults. $\kappa$ is the sample size. The smaller $\mathit{MSE}$ value is, the better performance the model has
2	$R^2=1-\frac{{\sum }_{i=1}^{\kappa }{\left({\mu }_i-\mu \left(t_i\right)\right)}^2}{{\sum }_{i=1}^{\kappa }{\left({\mu }_i-\sum _{i=1}^{\kappa }{\mu }_i/\kappa \right)}^2}$	where $\mu (t_i)$ represents the cumulative number of detected faults. ${\mu }_i$ denotes the number of actually observed faults. $\kappa$ is the sample size. The bigger $R^2$ value is, the better fitting power the model has
3	$\text{RMSE}=\sqrt{\frac{{\sum }_{i=1}^{\kappa }{\left(\mu \left(t_i\right)-{\mu }_i\right)}^2}{\kappa }}$	where $\mu (t_i)$ represents the cumulative number of detected faults. ${\mu }_i$ denotes the number of actually observed faults. $\kappa$ is the sample size. The smaller $\mathit{RMSE}$ value is, the better performance the model has
4	$TS=\sqrt{\frac{\sum _{i=1}^{\kappa }{(\mu (t_i)-{\mu }_i)}^2}{\sum _{i=1}^{\kappa }{\mu }_i^2}}\times 100\%$	where $\mu (t_i)$ represents the cumulative number of detected faults. ${\mu }_i$ denotes the number of actually observed faults. $\kappa$ is the sample size. The smaller $\mathit{TS}$ value is, the better performance the model has
5	$\text{Bias}=\frac{{\sum }_{i=1}^{\kappa }\left(\mu \left(t_i\right)-{\mu }_i\right)}{\kappa }$	where $\mu (t_i)$ represents the cumulative number of detected faults. ${\mu }_i$ denotes the number of actually observed faults. $\kappa$ is the sample size. The smaller $\mathit{Bias}$ value is, the better performance the model has
6	AIC = − 2log(χ) + 2y	where $\chi$ and $\text{y}$ denote the maximum likelihood function value and the number of model parameters, respectively

Table 3. Illustrations of software reliability models

Model name	Mean value function (MVF)	Description of model parameters
G–O model [45]	m(t) = a(1 − exp(− bt))	Where a and b represent the expected number of initially detected faults and the fault detection rate, respectively
Weibull distribution model [46]	m(t) = a(1 − exp(− btc))	Where a, b and c represent the expected number of initially detected faults, the fault detection rate and the shape parameter, respectively
Generalized Inflection S-shaped (GISS) model [8]	$m(t)=\frac{a\left(1-exp\left(-b{t}^c\right)\right)}{1+\phi exp\left(-b{t}^c\right)}$	Where a, b and c represent the expected number of initially detected faults, the fault detection rate and the shape parameter, respectively. $\phi$ denotes the inflection factor
Li model [2]	$m(t)=a(1-exp(-N(\frac{1}{1+\varphi exp(-\upsilon t)}-\frac{1}{1+\varphi })))$	Where a denotes the expected number of initially detected faults. $\upsilon$ denotes the shape parameter. N and $\varphi$ represent the scale parameters, respectively
Wang model [7]	$m\left(t\right)=a^d\left(1-exp\left(-bt\right)\right)/\left(1+\beta exp\left(1-exp\left(-bt\right)\right)\right)$	Where a and b represent the expected number of initially detected faults and the fault detection rate, respectively. d and $\beta$ denote the scale parameters and the inflection factor, respectively
Proposed model	$\mu (t)=\frac{\eta exp(-\theta t+\theta \alpha t^d){(1+\beta exp(-\alpha t^d))}^{\theta }(exp(\theta t)-1)}{{(1+\beta )}^{\theta }}$	Where $\eta$, $\theta$, $\alpha$,$\beta$ and d represent the expected number of initially detected faults, the fault detection rate, the fault introduction rate, the inflection factor and the shaped parameter, respectively

A parameter estimation method of the proposed model

In general, there are two common estimation methods of model parameters. Such as the maximum likelihood estimation (MLE) and least square estimation (LSE) methods. There are two reasons why we consider using LSE to estimate the parameters of the model. (1) Although in the case of large samples, the MLE is superior to LSE, in the terms of small samples, LSE is not inferior to MLE. Generally speaking, due to the limitation of test time, the collected fault data sets are all small samples. (2) It is possible that there is no maximum likelihood function value to estimate model parameters [8]. In order to be able to fully estimate the parameters of the model for model comparison, we use LSE and MlE to estimate the parameters of the model in this paper.

The least square estimation (LSE) method can be written as follows,

12

$$L=\sum _{i=1}^{\kappa }{\left(\mu \left(t_i\right)-{\mu }_i\right)}^2$$where $\mu (t_i)$, ${\mu }_i$ and $\kappa$ represent the cumulative number of detected faults, the actual observed number of faults and the sample size, respectively.

Equation (12) calculates the partial differential.

13

$$\frac{\partial \left(L\right)}{\partial \alpha }=\frac{\partial \left(L\right)}{\partial d}=\frac{\partial \left(L\right)}{\partial \theta }=\frac{\partial \left(L\right)}{\partial \eta }=\frac{\partial \left(L\right)}{\partial \beta }=0$$

The maximum likelihood estimation (MLE) method can be denoted as follows,

14

$$LL=Pr\left\{N\left(t_1\right)=n_1,N\left(t_2\right)=n_2,\dots ,N\left(t_k\right)=n_k\right\}=\prod _{i=1}^k\frac{{\left[\mu \left(t_i\right)-\mu \left(t_{i-1}\right)\right]}^{(n_i-n_{i-1})}exp\left[\mu \left(t_{i-1}\right)-\mu \left(t_i\right)\right]}{\left(n_i-n_{i-1}\right)!}$$where $N(t_i)$ and $n_i$ denote a count process and the number of actual detected faults, respectively.

The model Parameters’ values can be estimated by partial differential on both sides of Eq. (14),

15

$$\frac{\partial (LL)}{\partial \alpha }=\frac{\partial (LL)}{\partial d}=\frac{\partial (LL)}{\partial \theta }=\frac{\partial (LL)}{\partial \eta }=\frac{\partial (LL)}{\partial \beta }=0$$By solving the differential Eqs. (9,11), the estimated parameter value ($\alpha \ast ,\beta \ast ,\eta \ast ,\theta \text{*,d*}$) of the developed model can be obtained using LSE and MLE, respectively.

Model performance comparison for model parameters estimation using LSE

In terms of goodness-of-fit, from Tables 4, 5, 6, 7, 8 and 9, as can be seen that, in terms of the fitting performance, the developed model is the best among models. In Table 4, we can see that MSE of the developed model is nearly 2.3 times as small as that of the G–O model using 100% of data for DS1-1. Table 5 shows that RMSE of the Weibull distribution model is nearly 1.74 times as larger as that of the developed model using 100% of data for DS1-2. Table 6 indicates that MSE of the developed model is about 1.44 times as small as that of the Li model using 100% of data for DS1-3. In Table 7, although R² of the Weibull distribution model is approximately equal to that of the proposed, other metrics (i.e. MSE, RMSE, TS and Bias) are larger than those of the developed model using 100% of data for DS2-1. Similarly, from Table 8, R² of the developed model is approximately the same as that in the Weibull distribution model, but MSE, RMSE, TS and Bias of the developed model are lower than those of the Weibull distribution model using 100% of data for DS2-2. From Table 9, we can see that MSE of the developed model is about 1.82 times as little as that of the GISS model using 100% of data for DS2-3.

Table 4. Model comparison results using AIRFOLW 1.10.1 (DS1-1)

Model	100% of data (DS1-1)						90% of data (DS1-1)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 14,430,$b=3.7\times {10}^{-5}$	772	0.5209	27.78	56.22	23.29	a = 13,603,$b=2.7\times {10}^{-5}$	5796.3	76.13	62.05	7.44
Weibull distribution model	a = 1001.9, b = 0.000359, c = 1.0923	735.44	0.5436	27.12	54.87	22.5	a = 594.69,$b=7.9\times {10}^{-5}$, c = 1.4703	4521.9	67.24	54.81	6.57
Generalized Inflection S-shaped (GISS) model	a = 727.6, b = 0.13822, $\phi =99.07,$c = 0.6123	598.54	0.6286	24.47	49.5	21.25	a = 413.89, b = 0.084372, $\phi =76\text{.633},$c = 0.7184	3327.7	57.69	47.02	5.63
Wang model	a = 150, b = 0.010465, $\beta =0\text{.19178},$d = 0.886	1094	0.3211	33.08	66.93	24.96	a = 150, b = 0.009771, $\beta =0\text{.50221},$d = 0.8749	8936	94.53	77.05	9.24
Li model	a = 150, $\nu =0.025225,$$\phi =1.1999,$N = 0.5789	1177.8	0.2691	34.32	69.44	23.65	a = 150, $\nu =0\text{.022927},$$\phi =1\text{.2402},$N = 0.4472	8679.6	93.16	75.93	9.11
Proposed model	$\eta =299.99,$$\theta =0.000626,$$\alpha =0.23647,$ d = 1.9159,$\beta =0.0198,$	177.94	0.8896	13.34	26.99	9.96	$\eta =109\text{.96},$$\theta =0\text{.00183},$$\alpha =0\text{.001154},$ d = 2.8067,$\beta =0\text{.5795}$	432.29	220.79	16.95	1.73

Table 5. Model comparison results using AIRFOLW 1.10.2 (DS1-2)

Model	100% of data (DS1-2)						90% of data (DS1-2)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 29,063,$b=3\times {10}^{-5}$	2579	0.5549	50.78	53.97	42.87	a = 26,299,$b=2.5\times {10}^{-5}$	15,547	124.69	56.22	12.07
Weibull distribution model	a = 2612.8, b = 0.000273, c = 1.0495	2509.5	0.5669	50.09	53.24	42.34	a = 1503.2,$b=8.87\times {10}^{-4}$, c = 0.84421	17,927	133.89	60.37	12.96
Generalized Inflection S-shaped (GISS) model	a = 491.09, b = 0.4568, $\phi =30\text{.485},$c = 0.3242	3537.9	0.3894	59.48	63.21	52.24	a = 599.65, b = 0.19286, $\phi =60\text{.695},$c = 0.5308	11,692	108.13	48.75	10.46
Wang model	a = 150, b = 0.008549, $\beta =0\text{.39523},$d = 1.0462	4075.2	0.2966	63.84	67.84	47.77	a = 150, b = 0.00792, $\beta =0\text{.94742},$d = 1.0769	28,606	169.13	76.25	16.37
Li model	a = 150, $\nu =0\text{.029219},$$\phi =9\text{.9361},$N = 1.0208	3450.2	0.4045	58.74	62.43	40.78	a = 741.41, $\nu =0\text{.023194},$$\phi =379\text{.72},$N = 3.9662	1468	38.31	17.27	3.44
Proposed model	$\eta =138\text{.83},$$\theta =0\text{.00357},$$\alpha =0\text{.017106},$ d = 1.9891,$\beta =17\text{.595}$	822.37	0.8581	28.68	30.48	23.72	$\eta =309\text{.98},$$\theta =0\text{.001244},$$\alpha =0\text{.008746},$ d = 2.3478,$\beta =0\text{.3834}$	871.35	29.52	13.31	2.4

Table 6. Model comparison results using AIRFOLW 1.10.3 (DS1-3)

Model	100% of data (DS1-3)						90% of data (DS1-3)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 15,830,$b=4\times {10}^{-5}$	2883.6	0.4369	53.7	64.72	42.4	a = 12,917,$b=2.7\times {10}^{-5}$	30,451	174.5	75.36	17.21
Weibull distribution model	a = 2154.8, b = 0.000441, c = 0.90251	3084.3	0.3977	55.54	66.93	42.47	a = 831.61,$b=1.6\times {10}^{-5}$, c = 1.6963	25,225	158.82	68.59	15.67
Generalized Inflection S-shaped (GISS) model	a = 582.32, b = 0.59353, $\phi =56\text{.617},$c = 0.287	3716.7	0.2742	60.96	73.47	52.68	a = 795.3, b = 0.16997, $\phi =148\text{.6},$c = 0.5482	25,055	158.29	68.36	15.61
Wang model	a = 250, b = 0.00707, $\beta =0\text{.56151},$d = 0.9299	4048	0.2095	63.62	76.68	41.57	a = 250, b = 0.008441, $\beta =0\text{.86442},$d = 0.8525	40,907	202.25	87.35	19.96
Li model	a = 996.98, $\nu =0\text{.023481},$$\phi =569\text{.25},$N = 2.6964	845.14	0.835	29.07	35.04	23.17	a = 250, $\nu =0\text{.016917},$$\phi =2\text{.4277},$N = 0.3056	36,560	191.21	82.58	18.86
Proposed model	$\eta =270\text{.7},$$\theta =0\text{.000979},$$\alpha =0\text{.002933},$ d = 2.595,$\beta =0\text{.4686}$	585.05	0.8858	24.19	29.15	20.38	$\eta =300,$$\theta =0\text{.000563},$$\alpha =0\text{.000543},$ d = 3.0772,$\beta =0\text{.0177}$	2245.4	47.39	20.47	4.14

Table 7. Model comparison results using Jena 3.6.0 (DS2-1)

Model	100% of data (DS2-1)						90% of data (DS2-1)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 5314.1,$b=5.7\times {10}^{-5}$	24.06	0.7828	4.91	32.71	4.4	a = 4460.8,$b=6.2\times {10}^{-5}$	56.62	7.52	26.04	0.74
Weibull distribution model	a = 122.61,b = 0.000107, c = 1.7834	6.64	0.9401	2.58	17.19	2.17	a = 324.72,$b=1\times {10}^{-6}$, c = 2.7561	388.07	19.7	68.17	1.93
Generalized Inflection S-shaped (GISS) model	a = 134.66, b = 0.17413, $\phi =103\text{.53},$c = 0.6819	6.75	0.9391	2.6	17.32	2.15	a = 222.99, b = 0.002365, $\phi =31\text{.106},$c = 1.5473	63.12	7.94	27.49	0.75
Wang model	a = 588, b = 0.009703, $\beta =1\text{.1973},$d = 0.7163	47.41	0.5722	6.89	45.92	5.85	a = 150, b = 0.000002, $\beta =1\text{.3652},$d = 2.5111	96.28	9.81	33.95	0.97
Li model	a = 332.58, $\nu =0\text{.002928},$$\phi =0\text{.032452},$N = 10.869	27.98	0.7475	5.29	35.27	4.7	a = 149.93, $\nu =0\text{.009334},$$\phi =0\text{.041759},$N = 6.0677	134.42	11.59	40.12	1.14
Proposed model	$\eta =50\text{.001},$$\theta =0\text{.002961},$$\alpha =0\text{.8375},$d = 1.3963,$\beta =0\text{.5336}$	6.09	0.945	2.47	16.46	2.04	$\eta =1321\text{.1},$$\theta =0\text{.000208},$$\alpha =0\text{.074914},$d = 1.9638,$\beta =9\text{.6471}$	36.35	6.03	20.86	0.59

Table 8. Model comparison results using Jena 3.7.0 (DS2-2)

Model	100% of data (DS2-2)						90% of data (DS2-2)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 9290, b = 0.000133	82.03	0.7466	9.06	36.06	7.84	a = 8640.7, $b=1.26\times {10}^{-4}$	335.45	18.32	35.22	1.72
Weibull distribution model	a = 195.67, b = 0.000246, c = 2.0541	15.2	0.953	3.9	15.52	3.13	a = 133.81, $b=2.77\times {10}^{-4}$, c = 2.1278	46.78	6.84	13.15	0.63
Generalized Inflection S-shaped (GISS) model	a = 240.47, b = 0.27144, $\phi =65\text{.97},$c = 0.6845	36.2	0.8881	6.02	23.95	4.99	a = 242.39, b = 0.16942, $\phi =55\text{.302},$c = 0.7944	71.1	8.43	16.21	0.78
Wang model	a = 50.011, b = 0.001225, $\beta =5\text{.3553},$d = 2.2441	86.41	0.7331	9.3	37	8.02	a = 250, b = 0.000008, $\beta =2\text{.5611},$d = 2.3705	336.51	18.34	35.27	1.72
Li model	a = 250, $\nu =0\text{.008559},$$\phi =0\text{.1958},$N = 4.5736	96.85	0.7008	9.84	39.18	8.45	a = 250.02, $\nu =0\text{.005344},$$\phi =2\text{.2817},$N = 3.8694	373.79	19.33	37.18	1.81
Proposed model	$\eta =49\text{.997},$$\theta =0\text{.009219},$$\alpha =0\text{.62758},$ d = 1.6164,$\beta =0\text{.5491}$	13.17	0.9593	3.63	14.45	2.96	$\eta =50\text{.149},$$\theta =0\text{.011516},$$\alpha =0\text{.79449},$ d = 1.4404,$\beta =8\text{.196}$	10.34	3.22	6.18	0.26

Table 9. Model comparison results using Jena 3.8.0 (DS2-3)

Model	100% of data (DS2-3)						95% of data (DS2-3)
	Estimated parameters	MSE	R2	RMSE	TS	Bias	Estimated parameters	MSEpredict	RMSE	TS	Bias
G–O model	a = 5046.5, b = 0.000059	55.36	0.3157	7.44	69.62	4.28	a = 2691.5, $b=7.3\times {10}^{-5}$	1194.7	34.56	80.39	1.54
Weibull distribution model	a = 84.604, b = 0.000018, c = 2.5153	41.09	0.4921	6.41	59.97	3.44	a = 300.13, $b=1.8\times {10}^{-5}$, c = 2.0394	991.79	31.49	73.25	1.4
Generalized Inflection S-shaped (GISS) model	a = 233.84, b = 0.000034, $\phi =72\text{.348},$c = 2.9695	19.82	0.755	4.45	41.65	2.89	a = 323.11, b = 0.00827, $\phi =89\text{.164},$c = 1.384	937.72	30.62	71.22	1.36
Wang model	a = 250, b = 0.000049, $\beta =4\text{.9413},$d = 1.898	55.39	0.3153	7.44	69.64	4.2	a = 333, b = 0.000009, $\beta =0\text{.6726},$d = 1.8173	1164.6	34.13	79.37	1.52
Li model	a = 250.03, $\nu =0\text{.017522},$$\phi =7\text{.8893},$N = 0.5639	52.51	0.351	7.25	67.8	4.12	a = 333, $\nu =0\text{.002342},$$\phi =0\text{.46907},$N = 1.1806	1201.7	34.67	80.63	1.55
Proposed model	$\eta =50,$$\theta =0\text{.002187},$$\alpha =0\text{.000001},$d = 5.4636,$\beta =0\text{.523}$	10.88	0.8655	3.3	30.87	1.86	$\eta =50,$$\theta =0\text{.000318},$$\alpha =0\text{.000005},$d = 5.7483,$\beta =1.11$	58.49	7.65	17.79	0.3

On the whole, the MSE, RMSE, TS and Bias values of the developed model are lower than those of other models using 100% of DS1 and DS2, respectively. Moreover, the R² values of the developed model are larger than those of other models using 100% of DS1 and DS2, respectively. Thus, the established model has the better fitting power than other models. From Fig. 2a–f, we can see clearly that the fitting power of the developed model is better than other models.

Fig. 2 [Images not available. See PDF.]

Plots of the cumulative number of detected faults over time. a, b and c Cumulative number of detected faults using 100% of DS1-1, DS1-2 and DS1-3, respectively. d, e and f Cumulative number of detected faults using 100% of DS2-1, DS2-2 and DS2-3, respectively

In terms of prediction, from Tables 4, 5, 6, 7, 8 and 9, the predictive power of the established model is the best among all models. Table 4 shows that MSE_predict of the developed model is nearly 6.69 times less than that of the GISS model using 90% of data for DS1-1. From Table 5, we can see that RMSE of the proposed model is approximately 1.29 times less than that of the Li model using 90% of data for DS1-2. Table 6 shows that TS of the GISS model is about 2.3 times larger than that of the developed model using 90% of data for DS1-3. Table 7 shows that RMSE of the G–O model is about 1.24 times as big as that of the developed model using 90% of data for DS2-1. Table 8 shows that MSE_predict of the Weibull distribution model is approximately 3.5 times larger than that of the developed model using 90% of data for DS2-2. From Table 9, we can see that TS of the developed model is about three times less than that of the GISS model using 95% of data for DS2-3.

In Tables 4, 5, 6, 7, 8 and 9, the MSE_predict, RMSE, TS and Bias values of the developed model are less than those of other models using 90% of DS1, DS2-1 and DS2-2, respectively. Furthermore, the MSE_predict, RMSE, TS and Bias values of the developed model are less than those of other models using 95% of DS2-3, respectively. Therefore, the predictive power of the developed model is better predictive power than that of other models. From Fig. 3a–f, we can see clearly that the predictive power of the developed model is the best among models used in this paper. Note that 95% of the fault data sets (DS2-3) are used, mainly because the random selection of fault data sets can more fairly compare the power of the model than the subjective selection of fault data sets.

Fig. 3 [Images not available. See PDF.]

Plots of the cumulative number of detected faults over time. a, b and c Cumulative number of detected faults using 90% of DS1-1, DS1-2 and DS1-3, respectively. d, e and f Cumulative number of detected faults using 90% of DS2-1, 90% of DS2-2 and 95% of DS2-3, respectively

Overall, the fitting and predictive performance of the Weibull distribution model and the GISS model is better than other models except for the developed model. It also verifies that the CSS reliability model based on the Weibull distribution can be used to assess the reliability of OSS [10]. However, the reliability models of OSS, such as Wang model and Li model does not perform well. It is more advantageous to explain the complexity of OSS, especially the different forms of fault introduction in different open source development environments. Because fault introduction of the developed model has many forms, the proposed model which well meets the changes of OSS fault introduction has better adaptive ability than other models used in this paper. Therefore, the developed model has good adaptability and robustness, which can assist developers to assess the actual OSS reliability in the development and testing processes.

Model performance comparison for model parameters estimation using MLE

In order to compare AIC values, we use the third fault data set to compare the model performance, and use MLE method to estimate the model parameter values. From Table 10, MSE and AIC values of the developed model are less than those of other models and R² values of the developed model are larger than those of other models. R² values of the developed model are larger than 0.9. These show which the fitting power of the developed model is better than those of other models. Except for DS3-2, MSE, R² and AIC values of other models are very close. In DS3-2, the AIC value of the G–O model is closed to that of the proposed model, and the R² value of the Weibull distribution model is close to that of the developed model.

Table 10. Model performance comparisons using MLE for 100% of data (DS3)

Model	DS3-1				DS3-2				DS3-3
	Estimated parameters	MSE	R2	AIC	Estimated parameters	MSE	R2	AIC	Estimated parameters	MSE	R2	AIC
G–O model	a = 241.21, b = 0.002454	4.52	0.9368	124.71	a = 4302.9, b = 0.000131	8.07	0.7893	74.25	a = 43.616, b = 0.022665	3.78	0.9688	150.89
Weibull distribution model	a = 1250.6, b = 0.00062, c = 0.91939	4.2	0.9413	126.53	a = 15.203, b = 0.000184, c = 2.7391	4.97	0.8703	77.9	a = 43.352, b = 0.0223, c = 1.0075	3.71	0.9693	152.89
Generalized Inflection S-shaped (GISS) model	a = 1266.6, b = 0.000613, $\phi =0\text{.010058},$c = 0.9212	4.18	0.9416	128.53	a = 44.154, b = 0.005802, $\phi =0.000013,$c = 1.2992	5.8	0.8486	77.54	a = 43.285, b = 0.022632, $\phi =0\text{.010023},$c = 1.0062	3.71	0.9693	154.88
Li model	a = 3080.6, $\nu =0\text{.002311},$$\phi =0\text{.000377},$N = 220.95	4.51	0.9369	128.71	a = 2425.8, $\nu =0\text{.000249},$$\phi =0\text{.02052},$N = 47.594	8.14	0.7875	78.3	a = 3200.8, $\nu =0\text{.023101},$$\phi =0\text{.021847},$N = 0.6388	3.76	0.9689	154.87
Proposed model	$\eta =58\text{.138},$$\theta =0\text{.01276},$$\alpha =0\text{.0000001},$d = 5.0232,$\beta =46\text{.981}$	2.81	0.9608	123.82	$\eta =16\text{.006},$$\theta =0\text{.027431},$$\alpha =0\text{.025753},$d = 1.9827,$\beta =1\text{.5693}$	3.65	0.9049	73.99	$\eta =43\text{.614},$$\theta =0\text{.022666},$$\alpha =0\text{.00149},$d = 0.0001,$\beta =0\text{.0046}$	3.66	0.9781	150.68

From Table 11, we can see that MSE and AIC values of the proposed model are less than those of other models. In DS3-1, the AIC value of the G–O model is closed to that of the developed model. In DS3-2, the MSE value of the Li model is close to that of the developed model and the AIC value of the Weibull distribution model is close to that of the developed model. In DS3-3, MSE values of other models are very close and larger than those of the developed model.

Table 11. Model performance comparisons using MLE for 90% of data (DS3)

Model	DS3-1			DS3-2			DS3-3
	Estimated Parameters	MSE	AIC	Estimated parameters	MSE	AIC	Estimated parameters	MSE	AIC
G–O model	a = 33.45, b = 0.02646	5.91	94.6	a = 2990.3, b = 0.000185	0.15	68.58	a = 45.726, b = 0.021014	0.85	144.33
Weibull distribution model	a = 26.849, b = 0.012069, c = 1.367	7.14	95.49	a = 448.8, b = 0.000219, c = 1.5166	0.39	68.4	a = 48.078, b = 0.022324, c = 0.96028	0.96	146.31
Generalized Inflection S-shaped (GISS) model	a = 26.847, b = 0.012082, $\phi =0\text{.000993},$c = 1.3669	7.14	99.48	a = 605.27, b = 0.000166, $\phi =0\text{.00787},$c = 1.5113	0.41	72.39	a = 48.077, b = 0.022323, $\phi =0\text{.0000001,}$c = 0.9603	0.96	150.31
Li model	a = 5932.7, $\nu =0\text{.027021},$$\phi =0\text{.01281},$N = 0.443	6.01	100.58	a = 2768.6, $\nu =0\text{.00008}$, $\phi =0\text{.077708},$N = 37.358	0.14	74.58	a = 2619.7, $\nu =0\text{.020792},$$\phi =0\text{.00021},$N = 84.095	0.85	150.34
Proposed model	$\eta =33\text{.45},$$\theta =0\text{.02646},$$\alpha =0\text{.000008},$d = 0.2928,$\beta =1\text{.6007}$	4.9	93.6	$\eta =3462,$$\theta =0\text{.000159},$$\alpha =0\text{.016339},$d = 1.7597,$\beta =1981\text{.2}$	0.12	67.99	$\eta =45\text{.727},$$\theta =0\text{.021014},$$\alpha =0\text{.000077},$d = 0.0055,$\beta =0\text{.0582}$	0.81	141.68

In summary, through the above comparison, it can be concluded that the fitting and prediction performance of the developed model is better than other models. And the fitting and prediction power of other models is not stable. This shows which the developed model has good adaptability and flexibility in the complex environment of OSS testing and development, while other models cannot work well in the complex environment. Finally, it should be noted that when using MLE to estimate the model parameters for the Wang model, there is no existing maximum likelihood function value. So we didn't compare the model in Tables 10, 11.

Sensitivity analysis

Sensitivity analysis is carried out by changing a parameter and fixing other parameters of the proposed model. From Fig. 4, we can see that parameter changes of the developed model. The expected number of initially detected faults $\left(\eta \right)$, the fault detection rate $\left(\theta \right)$, the fault introduction rate $\left(\alpha \right)$ and the shape parameter (d) for the proposed model have a major impact.

Fig. 4 [Images not available. See PDF.]

Plots of sensitivity analysis of the developed model using 100% of DS1-2. a, b and c Parameters $\left(\eta ,\theta ,\alpha ,d,\beta \right)$ of the developed model change using 100% of DS1-2, respectively

There are mainly the following reasons,

1. The number of faults in OSS has an important influence on OSS Therefore, estimating the total number of faults in OSS is the basis of establishing the OSS reliability model.

2. The change of fault detection rate has an important influence on the estimation of the number of detected faults in the OSS.

3. Fault introduction is a problem that must be considered in reliability modeling of OSS. The change of the fault introduction rate directly affects the power of the OSS reliability model.

4. Owing to the dynamics, complexity and diversity of the OSS development, and to ensure the robustness and stability of the developed model, the shape parameters of the proposed model must be considered.

In addition, parameter $\beta$ is not an important parameter. Owing to the diversity and complexity of OSS testing and development environments, the cumulative number curve of introduced faults can show a variety of complex changes, not necessarily S-shaped curve. Parameter $\beta$ is also called an inflection factor. Therefore, it can be seen from this point that the fault introduction changes of OSS and CSS are different.

Implication of the study

The implication of this study has the following points,

1. Because OSS is mainly completed by users and volunteers in the open source community. The dynamic changes of users and volunteers cause the complexity and uncertainty of OSS testing and development. In view of the complexity of OSS testing and development environment, we conduct corresponding research and propose an effective OSS reliability model. This study lays a foundation for OSS reliability modeling under complex environments.

2. Owing to the uncertainty and complexity of OSS testing and development, the introduction of faults in the OSS debugging process shows a variety of changes. This study studies many changes of fault introduction in the OSS testing and development process. Meanwhile, fault introduction obeying the GISS distribution is proposed, which lays a foundation for various change modeling of fault introduction in the future.

3. The existing OSS reliability models rely too much on specific OSS development and testing environment, so their applicability and adaptability are very poor. The model proposed in this study fully considers the complicated changes of OSS testing and development environment, and an OSS reliability model with the GISS distribution is established. Because fault introduction obeys the GISS distribution, it can well simulate various changes of actual fault introduction. Therefore, the developed model can be effectively used to the reliability evaluation of actual OSS testing and development. In a sense, the modeling process of the developed model also provides methods and guidance for the further research on the reliability of OSS in the future.

Threats to validity

The threats to validity of the developed model have two factors. One is external factors, the other is internal factors.

External factors: To evaluate the performance of OSS reliability model, more kinds of fault data sets should be selected. However, we have selected three OSS fault data sets to fully evaluate the power of software reliability models. Three OSS fault data sets can meet the basic experimental requirements.

Internal factors: we used Taylor formula to simplify the equation and gave an approximate solution. In order to simplify the calculation, we assume that the failure detection rate is constant. Owing to the complexity of OSS reliability modeling, it is very beneficial to simplify the calculation properly. Moreover, OSS reliability modeling is also a trade-off between complex modeling and practical simplified use.

Conclusions

In this paper, we develop the OSS reliability model considering that fault introduction obeys the GISS distribution. Because fault introduction obeys the GISS distribution, the fault introduction rate can be represented by complex nonlinear changes, such as decreasing, increasing first and then decreasing over time. Therefore, the developed model can get with a variety of different reliability evaluation of OSS. To verify the adaptive and robust performance of the developed model, we use three OSS fault data sets, six model comparison criteria and five comparison models for sufficient experiments. The experimental results show that the developed model has best fitting and prediction performance among all models. We also carry out the relevant model parameter sensitivity analysis, and the results show that the parameters of the developed model, such as the expected number of initially detected faults $\left(\eta \right)$, the fault detection rate $\left(\theta \right)$, the fault introduction rate $\left(\alpha \right)$ and the shape parameter (d), have an important impact. The developed model can be used to assess the reliability for OSS, and it can also assist developers to evaluate OSS reliability.

Due to the complex and dynamic changes of OSS testing, debugging and development, we will consider various random changes of fault introduction to develop the corresponding OSS reliability model in the future.

Acknowledgements

We would like to thank anonymous reviewers and associate editor for their valuable comments. This work is supported by Fundamental Research Program of Shanxi Province of China and Natural Science Foundation of Shandong Province of China under Grant Nos. 201801D121120, ZR2021MF067.

Funding

Funding was provided by Fundamental Research Program of Shanxi Province of China (Grant No. 201801D121120) and Natural Science Foundation of Shandong Province of China (Grant No. ZR2021MF067).

Declarations

Conflict of interest

Conflict of interest on behalf of all authors, the corresponding author states that there is no conflict of interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

1. Tamura Y, Yamada S (2007) Software reliability growth model based on stochastic differential equations for open source software. In: 2007 IEEE international conference on mechatronics. IEEE pp 1–6

2. Li, X; Li, YF; Xie, M; Ng, SH. Reliability analysis and optimal version-updating for open source software. Inf Softw Technol; 2011; 53, 9 pp. 929-936. [DOI: https://dx.doi.org/10.1016/j.infsof.2011.04.005]

3. Yang, J; Liu, Y; Xie, M; Zhao, M. Modeling and analysis of reliability of multi-release open source software incorporating both fault detection and correction processes. J Syst Softw; 2016; 115, pp. 102-110. [DOI: https://dx.doi.org/10.1016/j.jss.2016.01.025]

4. Lin, CT; Li, YF. Rate-based queueing simulation model of open source software debugging activities. IEEE Trans Softw Eng; 2014; 40, 11 pp. 1075-1099. [DOI: https://dx.doi.org/10.1109/TSE.2014.2354032]

5. Huang, CY; Kuo, CS; Luan, SP. Evaluation and application of bounded generalized pareto analysis to fault distributions in open source software. IEEE Trans Reliab; 2014; 63, 1 pp. 309-319. [DOI: https://dx.doi.org/10.1109/TR.2013.2285056]

6. Singh, VB; Sharma, M; Pham, H. Entropy based software reliability analysis of multi-version open source software. IEEE Trans Softw Eng; 2018; 44, pp. 1207-1223. [DOI: https://dx.doi.org/10.1109/TSE.2017.2766070]

7. Wang, J; Mi, X. Open source software reliability model with the decreasing trend of fault detection rate. Comput J; 2019; 62, 9 pp. 1301-1312. [DOI: https://dx.doi.org/10.1093/comjnl/bxy111]

8. Erto, P; Giorgio, M; Lepor, A. The generalized inflection s-shaped software reliability growth model. IEEE Trans Reliab; 2018; 69, pp. 1-17.

9. Raymond, ES. The cathedral and the bazaar: musings on Linux and open source by an accidental revolutionary; 2001; Sebastopol, O’Reilly:

10. Zhou Y, Davis J (2005) Open source software reliability model: an empirical approach. In: Proceedings of the fifth workshop on open source software engineering, ACM, New York. pp 1–6

11. Anbalagan P, Vouk M (2008) On reliability analysis of open source software-fedora. In: 2008 19th international symposium on software reliability engineering (ISSRE). IEEE, pp 325–326

12. Rossi B, Russo B, Succi G (2010) Modelling failures occurrences of open source software with reliability growth. In: IFIP international conference on open source systems. Springer, Berlin, Heidelberg pp 268–280

13. Chiu, KC; Huang, YS; Lee, TZ. A study of software reliability growth from the perspective of learning effects. Reliab Eng Syst Saf; 2008; 93, 10 pp. 1410-1421. [DOI: https://dx.doi.org/10.1016/j.ress.2007.11.004]

14. Okamura, H; Dohi, T; Osaki, S. Software reliability growth models with normal failure time distributions. Reliab Eng Syst Saf; 2013; 116, pp. 135-141. [DOI: https://dx.doi.org/10.1016/j.ress.2012.02.002]

15. Utkin, LV; Coolen, FPA. A robust weighted SVR-based software reliability growth mode. Reliab Eng Syst Saf; 2018; 176, pp. 93-101. [DOI: https://dx.doi.org/10.1016/j.ress.2018.04.007]

16. Wang, J; Zhang, C. Software reliability prediction using a deep learning model based on the RNN encoder-decoder. Reliab Eng Syst Saf; 2018; 170, pp. 73-82. [DOI: https://dx.doi.org/10.1016/j.ress.2017.10.019]

17. Ke, S-Z; Huang, C-Y. Software reliability prediction and management: a multiple change-point model approach. Qual Reliab Eng Int; 2020; 36, pp. 1678-1707. [DOI: https://dx.doi.org/10.1002/qre.2653]

18. Tamura Y, Yamada S (2005) Comparison of software reliability assessment methods for open source software. In: 11th international conference on parallel and distributed systems (ICPADS'05). IEEE vol 2, pp 488–492

19. Gyimothy, T; Ferenc, R; Siket, I. Empirical validation of object-oriented metrics on open source software for fault prediction. IEEE Trans Softw Eng; 2005; 31, 10 pp. 897-910. [DOI: https://dx.doi.org/10.1109/TSE.2005.112]

20. Tamura Y, Yamada S (2007) Software reliability assessment and optimal version-upgrade problem for open source software. In: 2007 IEEE international conference on systems, man and cybernetics. IEEE pp 1333–1338

21. Syed-Mohamad MS, McBride T (2008) A comparison of the reliability growth of open source and in-house software. In: 2008 15th Asia-Pacific software engineering conference. IEEE pp 229–236

22. Tamura Y, Yamada S (2008) A method of reliability assessment based on deterministic chaos theory for an open source software. In: 2008 Second international conference on secure system integration and reliability improvement. IEEE pp 60–66

23. Singh BV, Singh PG, Kumar R et al (2010) A generalized reliability growth model for open source software. In: 2010 2nd international conference on reliability, safety and hazard-risk-based technologies and physics-of-failure methods (ICRESH). IEEE pp 523–528

24. Singh, BV; Kapur, KP; Basirzadeh, M. Open source software reliability growth model by considering change–point. Int J Inf Technol (IJIT); 2012; 4, 1 405.

25. Singh V, Kapur KP, Tandon A (2010) Measuring reliability growth of open source software by applying stochastic differential equations. In: 2010 second world congress on software engineering. IEEE vol 2, pp 115–118

26. Tamura, Y; Yamada, S. Optimisation analysis for reliability assessment based on stochastic differential equation modelling for open source software. Int J Syst Sci; 2009; 40, 4 pp. 429-438. [DOI: https://dx.doi.org/10.1080/00207720802556245]

27. Yamada, S; Tamura, Y. OSS reliability measurement and assessment; 2016; Geneva, Springer International Publishing: [DOI: https://dx.doi.org/10.1007/978-3-319-31818-9]

28. Zhu, M; Pham, H. A multi-release software reliability modeling for open source software incorporating dependent fault detection process. Ann Oper Res; 2018; 269, 1–2 pp. 773-790.MathSciNet ID: 3848504[DOI: https://dx.doi.org/10.1007/s10479-017-2556-6]

29. Aggarwal, AG; Dhaka, V; Nijhawan, N. Reliability analysis for multi-release open-source software systems with change point and exponentiated Weibull fault reduction factor. Life Cycle Reliab Saf Eng; 2017; 6, 1 pp. 3-14. [DOI: https://dx.doi.org/10.1007/s41872-017-0001-0]

30. Ahmadi, M; Mahdavi, I; Garmabaki, SHA. Multi up-gradation reliability model for open source software. Current trends in reliability, availability, maintainability and safety; 2016; Cham, Springer: pp. 691-702. [DOI: https://dx.doi.org/10.1007/978-3-319-23597-4_51]

31. Tamura Y, Miyahara H, Yamada S (2012) Reliability analysis based on jump diffusion models for an open source cloud computing. In: 2012 IEEE international conference on industrial engineering and engineering management. IEEE, pp 752–756

32. Tamura, Y; Kawakami, M; Yamada, S. Reliability modeling and analysis for open source cloud computing. Proc Inst Mech Eng Part O; 2013; 227, 2 pp. 179-186.

33. Tandon, A; Singh, VB; Sharma, M et al. Entropy based software reliability growth modelling for open source software evolution. Tehnicki Vjesnik; 2020; 27, 2 pp. 550-557.

34. Kapur, PK; Panwar, S; Kumar, V et al. Entropy-based two-dimensional software reliability growth modeling for open-source software incorporating change-point. Int J Reliab Qual Saf Eng; 2020; 27, 05 pp. 2040009-1-2040009-19.

35. Ivanov, V; Reznik, A; Succi, G. Comparing the reliability of software systems: a case study on mobile operating systems. Inf Sci; 2018; 423, pp. 398-411. [DOI: https://dx.doi.org/10.1016/j.ins.2017.08.079]

36. Barack, O; Huang, L. Assessment and prediction of software reliability in mobile applications. J Softw Eng Appl; 2020; 13, 9 pp. 179-190. [DOI: https://dx.doi.org/10.4236/jsea.2020.139012]

37. Wang, J. Model of open source software reliability with fault introduction obeying the generalized pareto distribution. Arab J Sci Eng; 2021; 46, pp. 3981-4000. [DOI: https://dx.doi.org/10.1007/s13369-021-05382-4]

38. Yang J, Wang X, Huo Y, Cai J (2020) Change point reliability modelling for open source software with masked data using expectation maximization algorithm. pp 1–6

39. Saraf, I; Shrivastava, AK; Iqbal, J. Generalised fault detection and correction modelling framework for multi-release of software. Int J Ind Syst Eng; 2020; 34, 4 464.

40. Sun X, Li J (2021) Simulation of software reliability growth model based on fault severity and imperfect debugging. pp 140–152

41. Ullah, N; Morisio, M; Vetrò, A. Selecting the best reliability model to predict residual defects in open source software. Computer; 2014; 48, 6 pp. 50-58. [DOI: https://dx.doi.org/10.1109/MC.2013.446]

42. Raghuvanshi, KK; Agarwal, A; Jain, K et al. A time-variant fault detection software reliability model. SN Appl Sci; 2021; 3, 1 pp. 1-10. [DOI: https://dx.doi.org/10.1007/s42452-020-04015-z]

43. Saraf, I; Iqbal, J; Shrivastava, AK; Khurshid, S. Modelling reliability growth for multi-version open source software considering varied testing and debugging factors. Qual Reliab Eng Int; 2021; 38, pp. 1-12.

44. Kapur, PK; Pham, H; Anand, S; Yadav, K. A unified approach for developing software reliability growth models in the presence of imperfect debugging and error generation. IEEE Trans Reliab; 2011; 60, 1 pp. 331-340. [DOI: https://dx.doi.org/10.1109/TR.2010.2103590]

45. Goel, AL; Okumoto, K. Time-dependent error-detection rate model for software reliability and other performance measures. IEEE Trans Reliab; 1979; 28, 3 pp. 206-211. [DOI: https://dx.doi.org/10.1109/TR.1979.5220566]

46. Goel, AL. Software reliability models: assumptions, limitations and applicability. IEEE Trans Softw Eng; 1985; 11, 12 pp. 1411-1423. [DOI: https://dx.doi.org/10.1109/TSE.1985.232177]