Unconvincing

"Lies, damned lies, and statistics" is a phrase describing the persuasive power of numbers, particularly the use of statistics to bolster weak arguments. It is also sometimes colloquially used to doubt statistics used to prove an opponent's point. The term was popularised in United States by Mark Twain (among others), who attributed it to the British Prime Minister Benjamin Disraeli: "There are three kinds of lies: lies, damned lies, and statistics." However, the phrase is not found in any of Disraeli's works and the earliest known appearances were years after his death. Several other people have been listed as originators of the quote, and it is often erroneously attributed to Twain himself.

So says Wikipedia. I notice that Russell Webster posted yesterday about the interim CRC reoffending rates, re-publishing a post by Jack Cattell of Get the Data from February. 

Ever since this TR omnishambles got going, many of us have suspected that trying to apply the smoke and mirrors magic of PbR or Payment by Results to our work was always going to be intriguing to say the least, and more so given that hard cash rewards are involved so the privateers can make their profit from the CRCs. 

Here are some extracts that go some way, in my mind at least, to highlight the nonsense of it all:-

The payment mechanism is complex – see here for the important documents. The basics are that if the CRC’s reoffending rate is lower than the 2011 baseline rate, having adjusted any differences in offenders’ likelihood of reoffending, then the CRC will receive a payment. Being just 0.1% lower would not be good enough to trigger a payment – the CRC’s rate must be statistically significantly lower (I have not found in the public domain information on how significant the reduction must be – do contact me if you know of a source).

Not enough data have been released to estimate which CRC will receive a payment (the 2011 baseline rates have not been published for example) but there are enough published data to assess the relative performance of each CRC. Each CRC’s published reoffending rate cannot be directly compared because the CRCs’ offenders will present with varying likelihoods of reoffending. Fortunately the OGRS4 expected one year reoffending rates were published. With this information, I estimated what each CRC’s proven reoffending rate would be if likelihood of reoffending was exactly the same in company (the adjusted rate – I used the average OGRS4 rate across all CRCs which was 45.7%).

There is a large 17.2% spread in interim proven reoffending rate in the MoJ data. This is reduced to a 9.5% spread in the adjusted reoffending rate. The lowest adjusted rate is in Merseyside (34.4%) and the highest is in Warwickshire & West Mercia (43.9%). There appears to be regional clustering at the top and bottom of the chart. Three of the four CRCs with the lowest adjusted rates are in the North West – Merseyside, Cumbria and Lancashire and Cheshire & Greater Manchester – and the neighbouring areas South Yorkshire, Humberside, Lincolnshire & North Yorkshire, and Durham Tees Valley make up three of the four areas with the highest adjusted rates. This clustering could be due to police practice, the CRCs’ effective work, efficient court processes or in or an unknown factor altogether.

These results do not mean the providers in the North West can expect a payment to come their way. We do not know the baseline to which they will be compared and there is another 6 months of convictions data to be included (the proven reoffending measure includes offences within 12 months that are convicted within 18 months). Also there is likely to be an area effect that is not included in the OGRS4 measure but will be in the 2011 baseline rate. However, the differences do suggest that persons interested in how to reduce reoffending rates should visit the North West to understand if they are doing anything different there.

--oo00oo--

Get the Data or GtD is clearly run by experts and they explain what they offer thus:-

Services

Our services are underpinned by our robust method, cost effective data collection, rigorous analysis skills, flexible project management approach and communication which add real value to social impact management.

Below you’ll find more information about each of our social impact analytics services along with details of how organisations are using them to carry out evaluations to monitor, learn how to improve and prove definitively their social impact.

Measuring

Our Impact Management analytics are helping organisations to work, understand and be proud of the impact they are having.

• Theory Of Change
• Social Impact Framework
• Cost Measurement Framework
• Data Collection and Report System
• Data Cleaning, Validation and Administration

Our Impact Management is helping practitioners, managers and board members think about what impact they are seeking to achieve, how to measure it and how they will achieve it with the resources at their disposal. We understand that practitioners, managers, board members and stakeholders’ value different impact information and our reporting and analysis systems ensure relevant information is available to all.

Learning

Our Predictive Analytics are assisting organisations to understand how they can increase their impact.

• Statistical Models
• Simulation and Scenario Analysis
• Decision Support Tools

Our Predictive Analyses are helping organisations to deliver more effective services. Our analyses are helping practitioners to identify what will work best for their clients, and managers are using the information to improve interventions and predict future impact, and in the case of social impact bonds, future income.

Proving

Our Impact Evaluation service has provided clients with a definitive and trusted assessment of their impact.

• Control Group Data Collection
• Quasi-Experiments
• Longitudinal Analysis
• Statistical Modelling of Impact

Organisations that commission GtD are working with some of the most vulnerable people in society. We value their work and are committed to using quantitative methods of evaluation to determine their impact. We are proud that our Impact Evaluations are not only delivering definitive reports on the impact of their work, but are used to provide a highly persuasive business case to continue an intervention and source income or a press release as part of a media or funding campaign.

--oo00oo--

I'll end with highlighting the criminal justice case study and I know I'm a fairly cynical kinda guy, but I just don't feel that convinced......

Predictive Analyses for Sodexo Justice Services

Sodexo Justice is one of the leading providers of justice services in the UK and are committed to reducing reoffending rates of the people they support. In creating a continuous improvement culture, Sodexo has commissioned GtD’s Predictive Analyses to identify who is most likely to reoffend and why, and to understand what high quality support services are needed to reduce reoffending.

Our cutting-edge Predictive Analyses are providing Sodexo with Closeness to Change reports for each of their service users which quantify who has expressed a desire not to reoffend. Our analyses can also help Sodexo identify what package of support is required to bring about desistance of offending in the service user.

In supporting Sodexo’s continuous improvement culture, our predictive analyses are also helping Sodexo to “learn quickly”. We have developed a predictive algorithm that estimates how likely someone is to reoffend and this is updated as we learn more about an individual over the duration of their sentence. This work also provides an estimate of the overall reoffending rate a year or more before it is actually realised.

By using our analyses to “learn quickly”, Sodexo have the tools not only to identify the most cost-effective ways of supporting their service users, but also to manage their resources to ensure they are focused on interventions and activities that bring about desistance. This dynamic approach is realising the continuous improvement culture by enhancing the relationship between practitioner and service user.