To evaluate how well the models predict, we ran a model using observed data from 1970 to 2000 and compared the predictions for 2007-2009 to the observed conflicts for the same three years. We are faced with a trade-off when deciding what constitutes a prediction. We can set a threshold that maximizes the ‘true positive rate’ (TPR), or the proportion of all observed conflicts that figure in our list of predictions. Alternatively, we can minimize the ‘false positive rate’ (FPR), or the proportion of observed non-conflicts that were among our predictions. The trade-off is that the higher the TPR, the higher the FPR.
To be more concrete, we might define a positive prediction as a country year that had a simulated conflict in more than half of the simulations we ran. 20 out of 169 countries were predicted to have conflict in 2009 according to this metric. 16 of these predicted conflict countries were among the 26 countries that really had a conflict in 2009. The TPR in that case is 16/26=0.63, and the FPR is 4/142=0.03. Alternatively, we might define a positive prediction as a country year with conflict in 30% of simulations. Then, TPR is 21/26=0.79 and FPR is 11/143=0.08.
Another metric is the ‘precision rate’ – the proportion of country years with positive predictions that were correct. This metric was 0.79 for the 2007-09 predictions when we set the threshold at 50% of the simulations. Assuming this also holds for our predictions into the future, 11 of the 14 counties we predict to be at conflict in 2017 will be at war according to our out-of-sample evaluation. These 14 countries are, sorted according to predicted risk with the most risky first: Ethiopia, India, Philippines, Myanmar, Thailand, Afghanistan, Algeria, Sudan, Pakistan, DR Congo, Somalia, Angola, Uganda, and Russia.
It might be more instructive to compare the actual experience of countries in 2011 to our predictions for 2017 based on our data from 1970-2009. 14 countries turn up with conflicts in more than 50% of our simulations for 2017. Only two of these did not have conflict in 2011, namely the Democratic Republic of Congo and Angola. All nine countries with conflict in more than 60% of simulations had conflict.
Among the 143 countries with conflict in less than 30% of simulations for 2017, the UCDP armed conflict dataset records conflict in eight: Cote d’Ivoire, Israel/Palestine, Libya, Mauritania, Senegal, Syria, Tajikistan, and Yemen. This is close to the expected false positive rate two years into the future. The incorrect predictions, however, do point to some of the constraints of our model.
Håvard Hegre 