An Empirical Evaluation of Two COSMIC Early Estimation Methods

Background. Under specific circumstances-especially in the early phases of software development projects-a thorough application of the COSMIC method may require more time and effort than available. Thus, early approximate estimation methods have been proposed for estimating the functional size of a given application, instead of properly measuring it.Objective. This paper aims at empirically evaluating the accuracy of two COSMIC early size estimation methods. The goal is to provide practitioners with some empirical evidence on the accuracy that can be expected from these methods.Method. We evaluated the Average Functional Process and the Equal Size Bands methods by applying them to a set of applications that were previously measured according to the standard COSMIC method, and for which the data necessary to perform estimations were readily available. The application conditions and performance of the Equal Size Bands method were also evaluated from a theoretical point of view.Results. Our analyses show that in a few cases the Average Functional Process method features estimation errors that are too large to be acceptable, while on average it provides reasonable estimates. On the contrary, the Equal Size Bands method can provide quite accurate estimates, but only if the human measurer is sufficiently good at classifying each functional process in the correct band. From a theoretical point of view, it is shown that-perhaps counterintuitively-the Average Functional Process and the Equal Size Bands methods provide essentially equivalent estimates when the distribution of Functional Processes across the bands is the same in the historical datasets and in the new software to be estimated. When such distributions are quite different, experimental results show that the Equal Size Bands method performs much better than the Average Functional Process method.Conclusions. Our results show that in a few cases the Average Functional Process method fails to provide acceptably small estimation errors. On the contrary, the Equal Size Bands method is sufficiently accurate to provide good size estimates. However, organizations that plan to use it need to properly train measurers that are able to identify the correct size band in which every functional process belongs.