Content area
In recent years, the Department of Commerce (DOC) has managed more than 335 software development project activities, incurring actual costs of $604 million. Over one-third of these investments have experienced cost variances surpassing 30%, highlighting persistent difficulties in accurately estimating and overseeing federal software investments.
Unanticipated cost variances can hinder IT portfolio modernization efforts and compromise stakeholder objectives. At the root of these challenges are unreliable cost and forecasting estimation methodologies, insufficient oversight mechanisms, and challenges in adapting to rapidly evolving technological demands.
To address these concerns, this research aims to improve existing DOC software development governance by identifying the most significant predictors of cost overruns and cost variances and developing statistical models capable of forecasting these overruns in advance.
Drawing upon project activity data obtained from the IT Dashboard, three predictive models were developed: two binary logistic regression models that estimate the probability of exceeding 10% and 30% cost variance thresholds, and a multiple linear regression model designed to quantify the extent of cost growth.
The binary logistic regression models demonstrated strong accuracy enabling proactive interventions in high-risk initiatives. The proposed binary logistic regression model for predicting cost variances ≥ |10%| was the most effective, as demonstrated by the 10-fold cross-validated AUC-ROC of 0.8620. This was further validated against actual project outcomes using a confusion matrix that showed precision, recall, specificity, and accuracy all above 80%.
The proposed binary logistic regression model for predicting cost variances ≥ |30%| produced an effective model, as demonstrated by the 10-fold cross-validated AUC-ROC of 0.8323. However, the resulting accuracy (77.9%), precision (69%), and recall (66.7%) reveal a moderate tradeoff between identifying true positive cases and avoiding false positive/negative predictions.
The multiple linear regression model proved less effective when compared to logistic regression models but provides insights on the most significant drivers of cost growth, pointing to areas in need of further methodological refinement. The proposed multiple linear regression model using the dependent variable cost growth (%) did not produce a model that met the target for R2 of 0.8.
By providing insights into the determinants of cost variances, this research aims to equips DOC project managers with the tools to allocate resources more effectively, improve cost estimation techniques, and strengthen oversight practices.