Thursday 9 March 2017

Unconvincing

"Lies, damned lies, and statistics" is a phrase describing the persuasive power of numbers, particularly the use of statistics to bolster weak arguments. It is also sometimes colloquially used to doubt statistics used to prove an opponent's point. The term was popularised in United States by Mark Twain (among others), who attributed it to the British Prime Minister Benjamin Disraeli: "There are three kinds of lies: lies, damned lies, and statistics." However, the phrase is not found in any of Disraeli's works and the earliest known appearances were years after his death. Several other people have been listed as originators of the quote, and it is often erroneously attributed to Twain himself.
So says Wikipedia. I notice that Russell Webster posted yesterday about the interim CRC reoffending rates, re-publishing a post by Jack Cattell of Get the Data from February. 

Ever since this TR omnishambles got going, many of us have suspected that trying to apply the smoke and mirrors magic of PbR or Payment by Results to our work was always going to be intriguing to say the least, and more so given that hard cash rewards are involved so the privateers can make their profit from the CRCs. 

Here are some extracts that go some way, in my mind at least, to highlight the nonsense of it all:-

The payment mechanism is complex – see here for the important documents. The basics are that if the CRC’s reoffending rate is lower than the 2011 baseline rate, having adjusted any differences in offenders’ likelihood of reoffending, then the CRC will receive a payment. Being just 0.1% lower would not be good enough to trigger a payment – the CRC’s rate must be statistically significantly lower (I have not found in the public domain information on how significant the reduction must be – do contact me if you know of a source).

Not enough data have been released to estimate which CRC will receive a payment (the 2011 baseline rates have not been published for example) but there are enough published data to assess the relative performance of each CRC. Each CRC’s published reoffending rate cannot be directly compared because the CRCs’ offenders will present with varying likelihoods of reoffending. Fortunately the OGRS4 expected one year reoffending rates were published. With this information, I estimated what each CRC’s proven reoffending rate would be if likelihood of reoffending was exactly the same in company (the adjusted rate – I used the average OGRS4 rate across all CRCs which was 45.7%).

There is a large 17.2% spread in interim proven reoffending rate in the MoJ data. This is reduced to a 9.5% spread in the adjusted reoffending rate. The lowest adjusted rate is in Merseyside (34.4%) and the highest is in Warwickshire & West Mercia (43.9%). There appears to be regional clustering at the top and bottom of the chart. Three of the four CRCs with the lowest adjusted rates are in the North West – Merseyside, Cumbria and Lancashire and Cheshire & Greater Manchester – and the neighbouring areas South Yorkshire, Humberside, Lincolnshire & North Yorkshire, and Durham Tees Valley make up three of the four areas with the highest adjusted rates. This clustering could be due to police practice, the CRCs’ effective work, efficient court processes or in or an unknown factor altogether.

These results do not mean the providers in the North West can expect a payment to come their way. We do not know the baseline to which they will be compared and there is another 6 months of convictions data to be included (the proven reoffending measure includes offences within 12 months that are convicted within 18 months). Also there is likely to be an area effect that is not included in the OGRS4 measure but will be in the 2011 baseline rate. However, the differences do suggest that persons interested in how to reduce reoffending rates should visit the North West to understand if they are doing anything different there.


--oo00oo--

Get the Data or GtD is clearly run by experts and they explain what they offer thus:-

Services


Our services are underpinned by our robust method, cost effective data collection, rigorous analysis skills, flexible project management approach and communication which add real value to social impact management.

Below you’ll find more information about each of our social impact analytics services along with details of how organisations are using them to carry out evaluations to monitor, learn how to improve and prove definitively their social impact.

Measuring

Our Impact Management analytics are helping organisations to work, understand and be proud of the impact they are having.

  • Theory Of Change
  • Social Impact Framework
  • Cost Measurement Framework
  • Data Collection and Report System
  • Data Cleaning, Validation and Administration
Our Impact Management is helping practitioners, managers and board members think about what impact they are seeking to achieve, how to measure it and how they will achieve it with the resources at their disposal. We understand that practitioners, managers, board members and stakeholders’ value different impact information and our reporting and analysis systems ensure relevant information is available to all.

Learning

Our Predictive Analytics are assisting organisations to understand how they can increase their impact.
  • Statistical Models
  • Simulation and Scenario Analysis
  • Decision Support Tools
Our Predictive Analyses are helping organisations to deliver more effective services. Our analyses are helping practitioners to identify what will work best for their clients, and managers are using the information to improve interventions and predict future impact, and in the case of social impact bonds, future income.

Proving

Our Impact Evaluation service has provided clients with a definitive and trusted assessment of their impact.
  • Control Group Data Collection
  • Quasi-Experiments
  • Longitudinal Analysis
  • Statistical Modelling of Impact
Organisations that commission GtD are working with some of the most vulnerable people in society. We value their work and are committed to using quantitative methods of evaluation to determine their impact. We are proud that our Impact Evaluations are not only delivering definitive reports on the impact of their work, but are used to provide a highly persuasive business case to continue an intervention and source income or a press release as part of a media or funding campaign.

--oo00oo--

I'll end with highlighting the criminal justice case study and I know I'm a fairly cynical kinda guy, but I just don't feel that convinced......

Predictive Analyses for Sodexo Justice Services

Sodexo Justice is one of the leading providers of justice services in the UK and are committed to reducing reoffending rates of the people they support. In creating a continuous improvement culture, Sodexo has commissioned GtD’s Predictive Analyses to identify who is most likely to reoffend and why, and to understand what high quality support services are needed to reduce reoffending.

Our cutting-edge Predictive Analyses are providing Sodexo with Closeness to Change reports for each of their service users which quantify who has expressed a desire not to reoffend. Our analyses can also help Sodexo identify what package of support is required to bring about desistance of offending in the service user.

In supporting Sodexo’s continuous improvement culture, our predictive analyses are also helping Sodexo to “learn quickly”. We have developed a predictive algorithm that estimates how likely someone is to reoffend and this is updated as we learn more about an individual over the duration of their sentence. This work also provides an estimate of the overall reoffending rate a year or more before it is actually realised.

By using our analyses to “learn quickly”, Sodexo have the tools not only to identify the most cost-effective ways of supporting their service users, but also to manage their resources to ensure they are focused on interventions and activities that bring about desistance. This dynamic approach is realising the continuous improvement culture by enhancing the relationship between practitioner and service user.

8 comments:

  1. I have looked at article by Russell Webster. One of the problems with these statistics, I immediately went back and checked on South Yorkshire Probation Trusts reoffending statistics prior to split (which used to be the best), is the quick realisation that there will need to be some significant extrapolation to make a like for like comparison. The stats are meaningless at this point other than comparison between different CRCs, some (for reasons that are unclear) whose reoffending rates are markedly different. Conclusion, these stats are meaningless. I think the North West of the country whose rates are the lowest will not be rushing to make any sweeping claims about the secret of their success.

    ReplyDelete
    Replies
    1. As an afterthought, there are so many regional and fluctuating regional variables as to make payment by results set against crude reoffending rates very questionable. I would not bank my business strategy set against PbR.

      Delete
  2. "We have developed a predictive algorithm that estimates how likely someone is to reoffend and this is updated as we learn more about an individual over the duration of their sentence."

    Ah, but if it can't predict the lottery numbers, make tea & provide intimate relief then I'm not interested.

    "This work also provides an estimate of the overall reoffending rate a year or more before it is actually realised... This dynamic approach is realising the continuous improvement culture by enhancing the relationship between practitioner and service user."

    Now we're getting warmer...! Oooh, I never knew selling snake oil could be such a sensual experience.

    ReplyDelete
  3. Statistics? Try these...

    "Former Chancellor George Osborne was the highest earning MP of 2016, raking in £628,000 on top of his MP’s salary of £74,962 - chiefly from lucrative public speaking engagements - latest figures have revealed."

    Since 2001, when first elected to seat, Osborne has trousered over £1M in salary as a MP plus many, many times more in expenses. Taxpayer money funded his professional development yet he now doubles-up his cashing-in on a personal level - nothing for the taxpayer - by utilising his experiences paid for by the taxpayer to bank a further £650k annually for 4 days a month on the books of Blackrock. But isn't he being paid by the taxpayer as a Member of Parliament? Shouldn't he be paying back 48 days' worth of his £76k salary, because he isn't available on those days? And what about all of the additional work referred to above? What does he actually do as MP for somewhere in Cheshire?

    ReplyDelete
  4. It brings back happy memories of TOM, PAM and all that other totally indispensable, hi-tec & very costly bollocks from yet another "specialist" not so many years ago.

    ReplyDelete
    Replies
    1. 2011 - "Humberside Probation Trust (HPT) has decided to use pam for its internal business management, employee engagement and external commissioning & partnering activity. Steve Hemming, the CEO, and his team decided to sign a long term agreement after seeing increasing successes enabled by pam elsewhere in Probation. HPT like other Probation Trusts and wider Public Services have big challenges as well as opportunities ahead. We are delighted pam is going to help enable them to achieve their goals. pam is now deployed and supporting delivery of various customer initiatives in almost 50% of the Probation Trusts. Coupled with its support of the Probation Chiefs Association, pam is now represented in 100% of the English and Welsh Probation Trusts, and increasingly being adopted by its statutory and collaborative partners across the Criminal Justice System and wider Public Services."

      Alliantist, the snake handler for 'pam', had also been awarded a "Cyber Essentials" award.

      Delete
    2. Why haven't we heard any more from Alliantist? "We live by what we preach. Much of our public sector work is also now of a sensitive nature and not allowed to be promoted."

      So if they told you they'd have to kill you.

      Delete
    3. pam is brought to you by Alliantist

      Our mission is to make the world a better place by equipping people to get their work done well.

      The world of work is complex. More than ever before, colleagues and partners must collaborate to achieve better results. But you can’t expect people to act differently and work well together if you don’t give them the tools and systems to do so.

      Alliantist has a heritage in partnering and alliances, and our services have evolved over time to reflect the needs of our customers. Alongside pam we also offer specialist cloud services delivered over the internet, including an internet delivered pam service, and ISMS.Online a solution for information security management and compliance. We also work with leading consultants and organisations on whitelabel solutions such as Track8, a collaboration with Future Purchasing. Read more about our people and the Alliantist journey below.

      Delete