Monday 15 July 2013

Selective Evidence

Professor Paul Senior has on several recent occasions drawn attention to the governments' tendency to prefer policy-based evidence rather than the more usual and sensible evidence-based policy approach when determining action and legislation.

On Friday, in a somewhat surprising policy u-turn in relation to the introduction of plain packaging for cigarettes, the government suddenly appeared keen to espouse the need for evidence before proceeding with change. I guess cynics would point to the policy change only coming about because of the lobbying power of the tobacco industry, the likelihood of the change having electoral consequences in relation to UKIP and the arrival of David Cameron's new policy adviser. 

Cynics would say that the governments' sudden conversion to the idea that evidence should influence policy is not likely to hold water in other areas, like privatising the probation service by means of the as-yet untested Payment by Results method. And they are absolutely right.

Regular readers will recall that there are two pilot PbR schemes running at HMP Peterborough and HMP Doncaster, but despite full evaluation of their effectiveness being some years off, the government has never-the-less decided to roll out the method nationally. 

Some weeks ago the MoJ published some interim statistics that Chris Grayling trumpeted as 'promising' but others said were disappointing and inconclusive. Yet others, including Jon Harvey, felt there were serious problems with the figures and as a result he submitted an FoI request with 21 rather pointed questions. Well, he's had a response and only two questions were refused answers. 

As before, I will be honest in saying that I don't fully understand the ins-and-outs of statistical analysis, but others will and we must look to such experts for informed comment. However, I agree with Jon that the key and extremely illuminating answer appears to be that given to question 19:-

You say 'these interim figures show a fall in the frequency of re-conviction events at Peterborough' which is a drop from 41.6% to 39.2%. at what threshold of probability is this statistically significant?

The answer is as follows:-

We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change."

As Jon Harvey says, 'is that a solid basis for the distribution of taxpayers' money?' I'd go further and say is it any basis on which to take risks and destroy a well-performing public service like probation? Lets hope HM Treasury and No10 are taking a keen interest in this unfolding omnishambles.        

6 comments:

  1. Hi Jim

    I know these stories aren't necessarily keeping in flavour with your last blog, but I hope they make for interesting reading for the regular readers:

    http://www.theboltonnews.co.uk/news/10547834.Police_commissioner_calls_for_government_to_rethink_probation_service_plans/

    http://www.dailyecho.co.uk/news/district/newforest/10548017.Probation_staff_stage_protest_march_over_controversial_privatisation_proposals/

    http://www.walesonline.co.uk/news/wales-news/leanne-wood-women-treated-afterthought-5104216

    ReplyDelete
  2. Well it's always good to hear news from Bolton and particularly that the Crime and Police Commissioner thinks the privatisation plans for probation is a bad idea.

    Similarly good to hear that Hampshire and Isle of Wight Napo branch are very active campaigning out in the sun.

    The news from Wales regarding women being very much an afterthought in the Transforming Rehabilitation privatisation plans could well be a subject of a post quite soon.

    Thanks for highlighting them,

    Cheers,

    Jim

    ReplyDelete
  3. "We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change."

    I'm not hot on stats either, but surely the above statement simply means that neither pilot will be assessed on the basis of whether they actually worked. In which case, what was the blinking point?

    ReplyDelete
    Replies
    1. Indeed, what was the point? It's no wonder politicians are held in such low esteem is it?

      Delete
  4. What that statement really translates to is;
    "we dont really give a shite if it works or not. Nor do we care what anyone thinks about it either. It's our baby, we say whats going on because we can and thats that".

    ReplyDelete
    Replies
    1. Yes it's a perfect example of politically-driven policy-based evidence if it works and if it doesn't, well as you say, we're doing it anyway! What a shit way to run things!

      Delete