Selective Evidence

Professor Paul Senior has on several recent occasions drawn attention to the governments' tendency to prefer policy-based evidence rather than the more usual and sensible evidence-based policy approach when determining action and legislation.

On Friday, in a somewhat surprising policy u-turn in relation to the introduction of plain packaging for cigarettes, the government suddenly appeared keen to espouse the need for evidence before proceeding with change. I guess cynics would point to the policy change only coming about because of the lobbying power of the tobacco industry, the likelihood of the change having electoral consequences in relation to UKIP and the arrival of David Cameron's new policy adviser. 

Cynics would say that the governments' sudden conversion to the idea that evidence should influence policy is not likely to hold water in other areas, like privatising the probation service by means of the as-yet untested Payment by Results method. And they are absolutely right.

Regular readers will recall that there are two pilot PbR schemes running at HMP Peterborough and HMP Doncaster, but despite full evaluation of their effectiveness being some years off, the government has never-the-less decided to roll out the method nationally. 

Some weeks ago the MoJ published some interim statistics that Chris Grayling trumpeted as 'promising' but others said were disappointing and inconclusive. Yet others, including Jon Harvey, felt there were serious problems with the figures and as a result he submitted an FoI request with 21 rather pointed questions. Well, he's had a response and only two questions were refused answers. 

As before, I will be honest in saying that I don't fully understand the ins-and-outs of statistical analysis, but others will and we must look to such experts for informed comment. However, I agree with Jon that the key and extremely illuminating answer appears to be that given to question 19:-

You say 'these interim figures show a fall in the frequency of re-conviction events at Peterborough' which is a drop from 41.6% to 39.2%. at what threshold of probability is this statistically significant?

The answer is as follows:-

We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change."

As Jon Harvey says, 'is that a solid basis for the distribution of taxpayers' money?' I'd go further and say is it any basis on which to take risks and destroy a well-performing public service like probation? Lets hope HM Treasury and No10 are taking a keen interest in this unfolding omnishambles.