Background. Using accurate effort estimation models can help software companies plan, monitor, and control their development process and development costs. It is therefore important to define sound accuracy indicators that allow practitioners and researchers to assess and rank different effort estimation models so that practitioners can select the most accurate, and therefore useful one. Several accuracy indicators exist, with different advantages and disadvantages. Objective. We propose a general framework for building sound accuracy indicators for effort estimation models. Method. The accuracy indicators that comply with our proposal are built by means of a comparison between a reference effort estimation model and the specific model whose accuracy we would like to assess. Several existing indicators are built this way: we develop a framework so new indicators can be defined in a sound way. Results. From a theoretical point of view, we applied our approach to accuracy indicators based on the square of the residuals and the absolute value of the residuals. We show that using a random model as a reference model, as done in some recent literature, sets too low a bar in terms of what may be acceptable. Instead, we use reference models that are built based on constant functions. From a practical point of view, we applied our approach to datasets containing measures of industrial software development projects. With the proposed method we were able to derive indications both according to criteria already proposed in the literature and according to new criteria. Conclusions. Our method can be used to define sound accuracy indicators for effort estimation models.

On the evaluation of effort estimation models

LAVAZZA, LUIGI ANTONIO;MORASCA, SANDRO
2017-01-01

Abstract

Background. Using accurate effort estimation models can help software companies plan, monitor, and control their development process and development costs. It is therefore important to define sound accuracy indicators that allow practitioners and researchers to assess and rank different effort estimation models so that practitioners can select the most accurate, and therefore useful one. Several accuracy indicators exist, with different advantages and disadvantages. Objective. We propose a general framework for building sound accuracy indicators for effort estimation models. Method. The accuracy indicators that comply with our proposal are built by means of a comparison between a reference effort estimation model and the specific model whose accuracy we would like to assess. Several existing indicators are built this way: we develop a framework so new indicators can be defined in a sound way. Results. From a theoretical point of view, we applied our approach to accuracy indicators based on the square of the residuals and the absolute value of the residuals. We show that using a random model as a reference model, as done in some recent literature, sets too low a bar in terms of what may be acceptable. Instead, we use reference models that are built based on constant functions. From a practical point of view, we applied our approach to datasets containing measures of industrial software development projects. With the proposed method we were able to derive indications both according to criteria already proposed in the literature and according to new criteria. Conclusions. Our method can be used to define sound accuracy indicators for effort estimation models.
2017
Proceeding EASE'17
9781450348041
21st International Conference on Evaluation and Assessment in Software Engineering
Karlskrona, Sweden
15 - 16 June 2017
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11383/2064390
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 15
  • ???jsp.display-item.citation.isi??? ND
social impact