Friday 3 November 2017

Lies, Damned Lies and Statistics

Last week the MoJ published some important and much-expected figures relating to the performance of the failing CRCs and which ones get a payout under the PbR part of their contracts. Russell Webster summed-it-up in less than 140 characters:- 
"Today's probation stats 13 CRCs have reduced binary reoffending - get PbR payment 2 CRCs have increased & have money deducted."
We can't put off talking about them any longer, but what the hell do they mean? It's all devilishly complicated and I notice Russell turned to a statistical expert Jack Cattell to try and make sense of it:-

PbR figures for first cohort

Last week (27 October 2017) saw the release of the Transforming Rehabilitation (TR) Payment by Results figures for the October to December 2015 cohort. The overall result was encouraging, and defy the view that TR’s radical changes to probation, and the ensuing problems, would result in increased reoffending – though it is very important that I point out that this is just the first set of results of many and overall judgement should be reserved for at least a year. The reoffending rate for all CRCs was 45.6% compared to a 2011 baseline rate of 47.5%. I had to make some (conservative) assumptions to estimate the baseline rate but I think it is also safe to say that the difference was statistically significant, suggesting reoffending rates have reduced under TR. Please see the the note at the end of this blog to understand better how I completed the analysis.


CRC Performance

The chart below describes each CRC’s reoffending rate in relation to the baseline 2011 rate. The grey line represents the range of reoffending rates that would indicate no change from 2011 (the baseline confidence interval). If the CRC’s rate is outside this range, we are confident in statistically terms to state that the CRC’s performance was either better or worse than the reoffending rate achieved in 2011. The green bars represent the reoffending rates of CRCs that outperformed 2011, the orange bars represent those that performed the same as 2011 and the red bars present those that performed worse than 2011.


Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

Thirteen of the CRCs beat the baseline rate. The best performing CRC was Cumbria and Lancashire, which beat the baseline rate by 8.2% (49.9% to 41.7%). [Editor’s note: Cumbria was the subject of a recent, very positive Probation Inspector’s report.] The nest best was Hampshire and the Isle of Wight which beat the baseline by 5.4% and the third best was Northumbria with a better rate by 4.3%. Two of the CRCs performed worse than the 2011 baseline. Warwickshire and West Mercia recorded a reoffending rate 3% worse than the baseline rate, and South Yorkshire’s rate was 2.8% worse. With most CRCs, however, outperforming the reoffending rate form 2011, the figures are a promising set of results.
Comparing CRC performance

Now that the baseline rates have been published, we can better understand how well each area was performing in 2011 and whether a CRC is now being asked to better good or bad performance achieved in that year. The chart below describes the difference between the actual baseline rate and the 2011 baseline’s OGRS score (in other words their expected rate of reoffending). A negative result in the chart means the area performed better in 2011 than the OGRS score expected.

2011 Baseline Performance




Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

The charts highlights that six of the CRCs are being asked to beat better than expected performance in 2011 (in other words to be better than good). Whereas other CRCs, notably London and Wales, are being asked to outperform potentially poor performance in 2011. It it interesting that South Yorkshire and, Warwickshire & West Mercia – the two areas that recorded poor performance for TR – are being asked to beat good performance from 2011.

Merseyside and Cheshire & Greater Manchester, however, are equally being asked to beat good performance from 2011 and were able to do so for the October to December 2015 cohort. The OGRS score does not allow for area effects, which will exist and could explain the differences between the OGRS score and the baseline rate. It not possible now to conclude whether payment by results will be easier in some areas than others, but, going forward, I will monitor the impact of whether a CRC is being asked to perform better than good or poor performance from 2011 on their ability to achieve payment by results bonuses.

Editor’s Conclusion

I am very grateful for Jack Cattell’s technical skills in analysing the first cohort of offenders supervised under the government’s new private/public probation service created by the Transforming Rehabilitation (TR) initiative. Jack will be following the performance of the new private Community Rehabilitation Companies via the publication of reoffending rates every three months.

Many will be surprised at the apparent good performance of the CRCs (and it is important to remember that reoffending performance is linked to contract remuneration via the TR payment by results contracts).

However, I would like readers to bear a key fact in mind: These figures relate to the first three months of the CRCs operation when, by general consent (confirmed by HMI Probation reports) there was considerable chaos and confusion, as would be expected in moving from public to private ownership. Most CRCs have taken the best part of two years to implement fully their new models of operation.

There are two consequences to this fact:
  1. It is inappropriate to judge CRCs (either way) on this first set of figures and, in truth, we may have to wait another two years to get a reliable indicator of reoffending performance.
  2. It forces us to consider whether reoffending rates are a reliable way of measuring probation performance or whether other factors (such as policing) have too great an influence.
Nonetheless, the comparisons between CRCs will certainly be valid over time and I look forward to reading Jack’s analysis of the next set of figures at the end of January 2018.

Jack Cattell

On Nov 28th, Jack will be talking more about TR’s PbR results and how rigorous analysis can help probation services and delivery chains to identify how best to reduce reoffending. See here for more details.

--oo00oo--

It's a free event designed, I suspect, for the serious enthusiast:-

Tue 28 November 201714:00 – 16:30 GMT

Transforming Rehabilitation: Learning from the PbR results

Transforming Rehabilitation challenged Community Rehabilitation Companies (CRCs) to reduce reoffending significantly and October will see the publication of the reoffending rates of the first Transforming Rehabilitation cohort.

These results are an important test of the government’s TR agenda and the success of the new CRCs. Some CRCs will be happy with their results, while others will need to improve. The published results also provide opportunities to commission new services for offenders.

Besides the implications for policy and practice, the results also present an opportunity for your organisation to learn how best to reduce reoffending with the tools at your disposable. You might want to increase a CRC’s performance, identify “what works?” and roll that out across all your CRCs, or prove that your specific service will make the difference.

Come join our experts at our TR PbR event, to discuss what needs to be learnt, what is within your control, and how embedding impact analytics in your CRC or service delivery organisation will help you to reduce reoffending.

Speakers:

Professor Darrick Jolliffe, University of Greenwich

“How to measure the quality of relationships between offenders and probation officers.”

Darrick is Professor of Criminology at the University of Greenwich. Darrick is interested in the broad areas of developmental life-course criminology, programme evaluation, prison research and psychology, individual differences and offending. He has developed tools to measure the satisfaction of offenders and the quality of their relationships with probation officers, and has estimated the association between relationship quality and reoffending. Darrick will also chair a discussion after the presentations.

Dr Sam King, University of Leicester

“What is high quality offender management?”

Sam is an expert on offender management and rehabilitation, and has contributed significantly to developments in desistence theory. His talk will discuss the latest evidence on what makes high quality offender management and what can be implemented to reduce reoffending. Sam has published widely on probation work and desistence theory, and has helped CRCs to implement innovative offender management tools to measure offender motivation.

Jack Cattell, Get the Data

“Predict your PbR results and continuously learn how to improve them before they happen.”

Jack is an expert on the prediction of reoffending rates, and the analysis of how a probation service can audit and improve its reoffending rate. He has worked for the Ministry of Justice, former probation trusts and Sodexo Justice Service’s six CRCs on research and analysis projects. He heads up GtD’s thought leadership on the use of predictive analysis to improve adult and youth offender management.

--oo00oo--

I was struck by the following tweet from Sally Lewis, former probation CEO:-
"As has always been the problem it's actually reconviction that is measured, not reoffending. Flawed measure. Police activity has huge impact."
A view endorsed by Russell Webster:-
"Agreed (although reoffending obviously unmeasurable). I'm increasingly convinced about impact of policing."
Another contributor put it rather more graphically thus:-
"Have to understand police performance alongside this. Run a CRC in an area where police aren't catching offenders and £££ to be made."
In trying to make sense of all this, factors such as contained in this policy document by West Midlands Police become increasingly relevant:-  

The Purpose and Benefits of Community Resolutions 

1.1. It is a long-established principle of British law that justice does not always demand a court prosecution. In many cases, the public interest is best served by using alternatives to prosecution, especially where such alternatives can provide swift redress for victims; and where we can intervene at an early juncture to change offending behaviour before it becomes entrenched. Peelian Principles emphasise the role of the police in preventing crime: Community Resolutions offer an opportunity to reduce the likelihood of future offending, and therefore, crime. 

1.2. Many of the crimes the police investigate are not committed by prolific, dangerous offenders. Rather, they represent momentary lapses in judgement by otherwise lawabiding citizens. In such situations, a formal sanction would often be a disproportionate response: the decision to give someone a criminal record is one that should never be taken lightly, and neither should the police underestimate how daunting the judicial system can be for victims and offenders alike. 

1.3. Community Resolutions are designed to deal with such situations. The use of a Community Resolution offers victims an informal, flexible response to the matter they have reported. At the same time, they offer offenders a ‘second chance’ - an opportunity to make amends for their mistakes without suffering consequences which can dramatically alter future life chances. 

1.4. Due to the lack of a criminal record, it is not appropriate to issue multiple Community Resolutions. However, the updated policy makes clear that an offender may still receive a Community Resolution in some circumstances where they have already received one previous disposal. If however, a Community Resolution is to be issued to an offender who has already received a previous disposal; more will be expected from the suspect. Specifically, it is expected that the suspect will be required to participate in activities which will directly address the underlying cause of their offending behaviour. 

1.5. The distinction between offenders who have no previous offending history and those with one previous disposal reflects a desire to intervene in offending behaviour at the earliest possible juncture, but without limiting the discretion of officers where it appears that the offence in question genuinely appears to be an isolated incident. 

1.6. Community Resolutions also offer the opportunity to deal with matters in the quickest, simplest fashion. In doing so, they help to ensure that the officers who serve the public are free to do more with their time. However, the need for efficiency must always be balanced against - and should never outweigh – the overriding goals of protecting and serving our communities. 

1.7. A Community Resolution is not the same as Restorative Justice, but they bear many similarities. Community Resolutions are founded on a commitment to empower victims by allowing them to have a say in how their matter is dealt with. They provide an opportunity for offenders to better understand the impact they have had on their victims, and to make amends for the harm they have caused. 

1.8. Fundamental to the success of every Community Resolution is the professionalism of the officer who administers it. Community Resolutions allow officers to exercise the greatest discretion, and with this discretion comes a responsibility to ensure that we can maintain the faith and confidence of the public.

--oo00oo--

Footnote

A glance at the archives reveals that I've only dared raise the subject of statistics once before Lies, Damned Lies and Figures published on 6th October 2012 and viewed just 76 times and with no comments...... 

23 comments:

  1. Worth remembering that some of these private companies have previous for submitting great sets of data for payment for result.

    In May 2011 a former auditor of Working Links claimed that the level of fraud at Working Links escalated to "a farcical situation" and was "endemic" but that he faced a "stonewall" from managers. Mr Hutchinson said he had encountered "a multi-billion-pound scandal", after working for Working Links and A4e in the welfare-to-work industry.[13] Working Links said: "We firmly reject any assertion of widespread fraud within our business."[14] The Department for Work and Pensions ruled that all allegations had been investigated at the time and no further action was needed.

    From Wikipedia Working Links. Press the link that leads to an explanation of why no further action was taken and you get a big 404.

    ReplyDelete
  2. Well spotted 7.09 but if you go further down there is another link to a 'Wales online'article that lists more details about the fraud allegations and treatment of the whistleblower. Seriously! MOJ knew this and they STILL awarded them the contract! MOJ we accuse you of calculated negligence. MOJ should be taken to an industrial tribunal for their gross negligence and stripped of their rights to make any further decisions in relation to criminal justice. Anyone with a shred of integrity should be standing up against this complete farce or face being accused of complicity.

    ReplyDelete
  3. Albeit early days, but this reconviction data will allow the flag-wavers for TR to gloat. This is how the effectiveness of probation will be measured in the public arena and by this measure TR is performing well.

    ReplyDelete
  4. Thank you Jim: I am slightly less bemused by the Cattell report than I was. I don't think the stats are supplied by the CRCs, although the original data on which they are based come from OASys I think.

    ReplyDelete
  5. I don't get the statistics. I don't understand how they are worked out. My impression from working in London CRC is that we do everything worse than before. We work in a knee jerk fashion, attentding to our tasks in a piecemeal fashion. It is all platespinning stuff, Ill thought out, badly planned if at all planned, no investment, no continuity, staff moving perpetually from pillar to post, service users changing officers every 5 minutes. It can't help desistance, and sure enough I see service users going in and out of prison at an ever faster pace and not just for recalls. So yes, I do think the statistics lie. And the presenters of the statistics lie. They would wouldn't they. This lie is totally in keeping with all the other lies we are constantly being fed.

    ReplyDelete
  6. Since we have had the IT to collect data, it has been used and abused by managers and politicians. "Gaming" performance has been around longer than TR, although it has now become the priority activity of both CRC and NPS, the former to secure profit, the latter to justify itself. OASys and Delius are both designed primarily to generate data: it is abundantly obvious that their function as assessment and recording tools for practitioners is secondary.
    Suffice to note that the event advertised has a statistician advising attenders how to improve performance, cynical me assumes this may be a tutorial on how to finesse your gaming skills

    ReplyDelete
  7. Increased interest in your coverage of statistics, then Jim! Way back in the early days of the campaign against TR I suggested that in lieu of striking, we should hit our employers where it really hurts them, and negatively game performance on given days. That is, perform the exact same work, but game our recording in order to fail the targets. It is within the gift of all workers beaten with the performance target stick: just turn "good performance" on and off like flicking a light switch. It would have been extraordinarily therapeutic too.
    I think we could, for instance, knacker a key performance figure by underscoring OASys sections. Leave the commentary box for actual practitioners use, the people who have genuine curiosity and interest in the client, but ensure the boxes are set to zero at the start of an order, and maximum at the end. Two years down the line: Catatrophic increase in reoffending as measured in these stats

    ReplyDelete
    Replies
    1. Funnily enough our unreliable IT systems seem to do a great deal of this for us. One minute we are jogging along at a steady rate, a little under what we should be, as always, this goes on comfortably for a few weeks. Suddenly there is a surge of something, we suddenly have well over 100 incomplete outcomes, no next appointments, from one day to the next. The SPO goes into a flat spin, emails urgent urgent urgent, attend to your stats. We abandon our plans for the day and settle down to a few hours of frantic mouse clicking. An hour or so into the exercise the SPO appears after having queried the stats with the IT, with the message that the 100 + have now plunged down by 115. Her question to us was: had we her team reduced that many between yesterday and today? The answer of course was no. My own hour's furious activity had only removed 3 or 4. I continued my exercise, finally eliminating all the statistical disasters I had been responsible for. Towards the end of the day the SPO appeared to let us know that the stats for our team was well above 100 again. It was an emotional rollercoaster of a day. The powers that be know full well that their own gaming equipment is as unreliable as their own moral compas. They will lie whatever. We need something bigger.

      Delete
  8. Valid the stats may be, but if it's going to take another two years to determine their reliability it just leaves questions hanging.
    If it emerges that they're not reliable in two years time there's going to be a protracted discussion about what went wrong, who's to blame, what should happen next. Won't that bring the contacts for TR to just about the end of their life?
    Will new contracts be negotiated with new criteria and payment mechanisms, or will the privateers just close shop and hand back the keys?
    I don't trust statistics anyway, but I'm struck by these ones for a couple of reasons.
    Firstly the PbR mechanism was changed, and I don't really know in what way.
    Secondly, it was only in February this year that MTCNova and Interserve appeared before the justice select committee threatening to walk away from their contracts because the PbR model was giving them such a pitiful return that it really wasn't worth having.
    How good are your results if the payment you get for them isn't worth having?
    Since February there's been an unexpected spike in the prison population, which was already at an all time high?
    These statistics seem not to corralate with the national conversation in the media, those that work on the front line, or what I see with my own eyes.
    I simply don't believe them.

    'Getafix

    ReplyDelete
  9. To put the 'reoffending rates' into context it is right & proper that Police policy & data for the corresponding areas are also examined over the same period.

    Which Forces are using more Cautions where previously there were arrests & charges?
    What have been the reduction of staff figures for each Force area?
    What are the arrest/charging figures for each Force over the same period?
    What criteria define 'reoffending'', e.g. is it arrest; charge; court appearance; conviction?

    A cynic's cynic might take the view that Policing has been intentionally underfunded in order to reduce the numbers of arrests/convictions as Police neccessarily prioritise the more serious aspects of their work, e.g. the massive demands on Police due to counter-terrorism must have affected levels of intervention at the lower end of seriousness?

    ReplyDelete
    Replies
    1. Here's one brief snippet from Cumbrian Newspapers based in the alleged best-performing area:

      "18 August 2017 11:25AM

      Budget cuts have led to the closure of 18 police stations or bases in Cumbria in just over five years.

      It can be revealed today how cash constraints are taking their toll on a major part of the public face of law and order - with fears raised over the impact on tackling trouble in some of the places affected.

      Since 2011, law-keeping outposts that stood for generations have vanished from towns and communities."

      Delete
    2. UKCrimeStats.com website shows the following annual recorded crime statistics for Police areas. Using Cumbria as an example:

      2011 - 54246
      2012 - 49465
      2013 - 49485
      2014 - 47007
      2015 - 40498
      2016 - 38372

      to Aug 2017 - 25809; est to Dec'17 = c.39,000

      Budget figures & staffing numbers would also be useful to match against this data...

      Delete
    3. Presumably the 'prison works' brigade will argue that increasing prison populations and reducing crime stats & reoffending figures proves that they should all be banged up, keys thrown away & we'll all be safer & happier!!!

      Delete
  10. Just been looking at On Probation Blog January 18 2116.
    If you operate a cohort model then the variables available to present data/outcomes is almost endless.

    ReplyDelete
  11. For a proper assessment of the "success" of TR we also need a comparison with how the Trusts were performing in relation to reoffending rates. No doubt the MoJ will say they can't provide that because ORA changed the cohort.

    ReplyDelete
  12. Even if the statistics could be proved to be absolutely true and correct, it's quantative only.
    Before the bugles hereld the success of TR the qualitive aspect needs consideration also.
    Working conditions, caseloads, service practices and service delivery should form part of the picture.
    If TR has by some unbelievable method reduced reoffending, then at what cost in human and monetary terms has been paid to effect that reduction?
    Are the costs really good value?

    'Getafix

    ReplyDelete
  13. I recall my last couple of years before retirement in 2011, when the Trust was firing targets at us. We were told that if we didn't meet targets we would lose precious money. So, on Crams, we had to indicate at commencement of Order, then every month, then most importantly, at termination of Order, if someone had got a job because of the officer's efforts, by the end of the Order. That included a tick if clients found employment with no intervention from us, or were on very short training courses, had employment for a few days, or did voluntary work, from their own efforts, and the most annoying- we had to give a tick if they already had work when they started their Order!

    This fibbing continued onto Oasys, when at the termination of the Order, the score had to indicate that a homeless person had found accommodation if they were in a temporary shelter, or if they slept the odd night on friend's settees, or even floors. I, and several other officers, ignored this instruction. We found this disgusting, but it has no comparison with the fraudulent instructions which are coming out of the sleazy CRCs, which include being told not to breach, even if they are a risk to the public. I appreciate that this has not been happening everywhere, before I get angry comments but it is certainly not uncommon, as the Blog has revealed many times.

    ReplyDelete
  14. I can safely say that I haven't reoffended not due to anything probation did or didn't do but solely as a result of my own actions and decisions. To be blunt, probation was so unhelpful and pretty much completely absent from the equation I may as well not have been "under supervision" for 3.5 years in respect of anything they had to do with my reoffending or not.

    ReplyDelete
    Replies
    1. This also means that Purple Futures got a ton of money due to my not reoffending even though they did diddly squat to earn it in any way shape or form. The tax payer got royally ripped off

      Delete
  15. If you are a critic of TR you won't like these statistics. You can add health warnings - the human toil and low morale - but these will be trumped by the headline figures. If the statistics had been damning of TR performance, we, too, would be focusing on the headlines to bash TR. As the statistician cautions, it's too early for any firm conclusions, so it's a waiting game.

    ReplyDelete
  16. Save Guido Fawkes. Boycot November 5th. He was innocent.

    ReplyDelete
  17. www.devonlive.com/news/devon-news/20-people-die-being-supervised-698867.amp

    ReplyDelete
    Replies
    1. Across England and Wales, there were 41 deaths of women, who were being supervised by probation services, either post-release from prison or while on a community sentence, in 2016/17 that were self inflicted, up from 35 in 2015/16 and the highest number in a year since at least 2010/11.

      Self-inflicted deaths were the most common cause of death for female offenders in the community.

      For men, the number of deaths due to homicide has hit its highest number for at least seven years, with 30 deaths in 2016/17, up from 20 in 2015/16.

      Overall, there were 748 deaths of offenders in the community in 2016/17, up 7% from 2015/16 when adjusted for the missing returns from two CRCs.

      There were 233 self-inflicted deaths, down 9% from the previous year.

      During this period natural-cause deaths increased by 10% to 258 and apparent homicides increased from 22 to 33 deaths in 2016/17.

      The increase in deaths in the community is due to the rise in number of offenders who died under post-release supervision, while deaths of other offenders, such as those subject to community sentences, have been falling.

      There were 372 deaths during post-release supervision, up 28% from the previous year.

      This increase corresponds with the introduction of the Offender Rehabilitation Act (ORA) on 1 February 2015, and the subsequent increase in the number of offenders on post-release supervision since.

      Under ORA, all offenders given custodial sentences are now subject to a minimum of 12 months’ supervision in the community upon release from prison.

      In 2016/17, there was a 16% increase in deaths of offenders in the community supervised by the NPS and a 2% increase in those supervised by CRCs, private sector companies than manage low to medium risk offenders, compared with 2015/16.

      Natural causes were the predominant cause of deaths in the NPS (44%), while self-inflicted predominated in the CRCs (34%).

      This is only partly explained by the different age distributions of the supervised offenders. When comparing on a like-for-like basis, CRCs had a drop in the number of self-inflicted deaths compared to the previous year, whereas the NPS saw an increase.

      There are notable differences in the distribution of age at the time of death between the NPS and CRCs. A larger proportion of deaths in the NPS occurred in the 50-65 years and over 65 years age categories, while the largest number of deaths in CRCs occurred among those aged between 36 and 45 years.

      Delete