The evaluation of regression QSAR model performance, in fitting, robustness and external prediction, is of pivotal importance. Over the past decade different external validation parameters have been proposed: Q2F1, Q2F2, Q2F3, and the Golbraikh–Tropsha method. Recently, the concordance correlation coefficient (CCC, Lin), which simply verifies how small the differences are between experimental data and external data set predictions, independently of their range, was proposed by our group as an external validation parameter for use in QSAR studies. In our preliminary work we demonstrated, by thousands of simulated models, that CCC is in good agreement with the compared validation criteria (except ) using the cut-off values normally applied for the acceptance of QSAR models as externally predictive. In this new work, we have studied and compared the general trends of the various criteria relative to different possible biases (scale and location shifts) in external data distributions, using a wide range of different simulated scenarios. This study, further supported by visual inspection of experimental vs. predicted data scatter plots, has highlighted problems related to some criteria. Indeed, if based on the cut-off suggested by the proponent, could also accept not predictive models in two of the possible biases (location, location plus scale), while in the case of scale shift bias it appears to be the most restrictive. Moreover, Q2F1 and Q2F2 showed some problems in one of the possible bias (scale shift). This analysis allowed us to also propose recalibrated, and inter-comparable for the same data scatter, new thresholds for each criterion in defining a QSAR model as really externally predictive in a more precautionary approach. An analysis of the results revealed that the scatter plot of experimental vs. predicted external data must always be evaluated to support the statistical criteria values: in some cases high statistical parameter values could hide models with unacceptable predictions.
Real External Predictivity of QSAR Models. Part 2. New inter-comparable thresholds for different validation criteria and the need for scatter plot inspection
CHIRICO, NICOLA;GRAMATICA, PAOLA
2012-01-01
Abstract
The evaluation of regression QSAR model performance, in fitting, robustness and external prediction, is of pivotal importance. Over the past decade different external validation parameters have been proposed: Q2F1, Q2F2, Q2F3, and the Golbraikh–Tropsha method. Recently, the concordance correlation coefficient (CCC, Lin), which simply verifies how small the differences are between experimental data and external data set predictions, independently of their range, was proposed by our group as an external validation parameter for use in QSAR studies. In our preliminary work we demonstrated, by thousands of simulated models, that CCC is in good agreement with the compared validation criteria (except ) using the cut-off values normally applied for the acceptance of QSAR models as externally predictive. In this new work, we have studied and compared the general trends of the various criteria relative to different possible biases (scale and location shifts) in external data distributions, using a wide range of different simulated scenarios. This study, further supported by visual inspection of experimental vs. predicted data scatter plots, has highlighted problems related to some criteria. Indeed, if based on the cut-off suggested by the proponent, could also accept not predictive models in two of the possible biases (location, location plus scale), while in the case of scale shift bias it appears to be the most restrictive. Moreover, Q2F1 and Q2F2 showed some problems in one of the possible bias (scale shift). This analysis allowed us to also propose recalibrated, and inter-comparable for the same data scatter, new thresholds for each criterion in defining a QSAR model as really externally predictive in a more precautionary approach. An analysis of the results revealed that the scatter plot of experimental vs. predicted external data must always be evaluated to support the statistical criteria values: in some cases high statistical parameter values could hide models with unacceptable predictions.
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
