Targets and Measures

I can't remember how or where, but I was struck by this reminder the other day:-

Goodhart's Law is expressed simply as: “When a measure becomes a target, it ceases to be a good measure.” In other words, when we set one specific goal, people will tend to optimize for that objective regardless of the consequences.

And reader FranK highlighted the following:-

"Also on HMPPS website: The critical Community Performance Quarterly updates, those targets that management make you break your backs to meet, will be hidden from public view until after "reunification" in July 2021.

Yet another disgraceful, cynical sleight of hand to mask, manipulate & otherwise massage the data to suit the narrative of the Bullies & Liars at Her Majesty's Piss Poor Service. Statistics release cancelled."

From HMPPS website:-

"Since the introduction of the Offender Rehabilitation Act (ORA) as part of Transforming Rehabilitation, the National Probation Service (NPS) and Community Rehabilitation Companies (CRCs) have been monitored against performance framework to make sure their delivery of services is timely, consistent and of high quality.

The publication will cover all performance metrics from both frameworks, at a national level and broken down to lower levels of geography where appropriate.

As stated in the previous March 2020 release of this publication, and following the completed consultation period, the release schedule for this publication is moving to an annual cycle, with the next edition reporting full-year outcomes for 2020/21 in July 2021. The contents and structure of the publication will not change and the additional tables on accommodation and employment circumstances will continue to be included. From June 2021, the current performance frameworks for probation will be coming to an end. Our intention from this point onward is to produce a re-designed publication to better fit the new performance monitoring arrangements that will be in place under the Unified Probation Model."

--oo00oo--

Whilst we ponder exactly what shenanigans are going on at HMPPS HQ, I thought these extracts from a recent report on the Clinks website was instructive:-

Information is part of the cost-benefit cycle 

Sound cost-benefit assessment relies on the availability of robust, accurate, and reliable data and information. In most funding contexts, some form of independent verification of the claims of programme providers is required by funders as a condition of continued funding; this evidence is gathered through a combination of evaluations or inspection audits by external or official bodies. 

In principle, evaluations of programmes should primarily inform internal planning and longer-term strategisation. However, the purpose of commissioned studies can be distorted by commercial and policy priorities, where ‘evaluations’ and audit reports are mainly used for external consumption of funders, sponsors and the general public. Commissioned evaluative studies by consultants or university personnel can be very costly, while smaller charities find that the required investment in data gathering systems and activities may outweigh the potential resources they need for improving or expanding their services. In the sense that information is liable to be commoditised in such a manner, it might be argued that there is an inverse relationship between the value of such studies as a tool for strategic planning and degree to which they are valued for fundraising and publicity. Put another way, evaluations might not always serve the functions of both revealing weaknesses in performance while generating a public picture of success. Voluntary sector providers should be clear about their purpose when undertaking or commissioning a CBA and be aware that a more in-depth investigation may reveal areas for improvement as well as areas of success. (For a broader discussion of the value and pitfalls of evaluation for the voluntary sector, see Hedderman and Hucklesby, 2016).

CBAs are about more than reducing reoffending 

Inspection audits of a kind which weigh performance and outcomes against national targets – such as assessing the success of programmes in reducing reoffending against national or regional reoffending thresholds – present their own advantages and limitations. Whilst the ability to compare outcomes for service users against the national or regional picture adds value to assessing the strengths and weaknesses of programmes (itself a useful form of qualitative cost-benefit analysis), such comparisons should be treated with caution rather than certitude. 

The Ministry of Justice’s Justice Data Lab (JDL) gives voluntary organisations and other bodies working with offenders free access to central reoffending data. Provided you have service users’ consent to use their data and their names, date of birth, gender and date of community sentence or release from prison, and a significant enough number of service users to generate a statistically significant result, the JLD can calculate the reoffending rates of your service users against a matched cohort of offenders. You should be confident of your results though as the JDL stipulates that all findings are published in order to advance our knowledge of what works in reducing reoffending. 

Chris Fox and Kevin Albertson (2012; 125-126)2 usefully summarise the proper uses and misuses of cost-benefit analyses with a view to introducing some proportionality into commissioners’ expectations as to what is reliably measurable and relevant for providers, including small and medium sized VSOs. 

• The expectation that benefits (in terms of desistance or increasing pro-social behaviour) may outweigh costs (in financial terms) may not hold. In other words, there is a weak relationship between criminal justice interventions for reducing offending, and net financial savings to the public. 

• Properly conducted, cost-benefit analysis should consider all outcomes (social, financial, personal) and not just weigh reduced reoffending against savings. 

• Properly done, valuations of ‘soft’ outcomes (wellbeing, reduction in fear or insecurity) are as important to gaining an accurate analysis as comparison with ‘hard’ outcomes (reduced offending behaviour).

• The direct translation of outcome evaluative findings into monetary values is fraught with preferential bias, and any calculation of ‘savings’ where crime does not occur needs to be carefully interpreted. 

• Traditional cost-benefit analysis is poor at identifying externalities, so that attribution of factors contributing to success or failure are often omitted, thus giving an inaccurate picture of what contributed to success or failure in a project.

Be careful what you measure – and what you are required to measure 

The pressure to prove the efficacy of your programme (and understate problems) can be compounded when it is directly and narrowly linked to outcomes-based commissioning (in the form of payment by results or principal only payment systems). A major concern for the voluntary sector is the burden of evidence which is placed upon providers to establish both the cost-efficiency and social benefit of their work utilising instruments and measures which capture narrow financial and reducing reoffending parameters. In the current climate of austerity and the need for providers to ‘do more with less’, it can be a matter of politics, rather than social science, as to which ‘costs’ and which ‘benefits’ are actually utilised and calculated. The traditional view of commissioners and funders is that the costs of programmes or interventions should be favourable in relation to expenditure (i.e. should save money) when assessing the validity of programmes. This kind of calculation, for example, might offset the cost of intervention or treatment against what might notionally be saved had the service user otherwise been using health, policing, custodial or other public resources, reduced criminal activity, or any reduction in victim costs, for example. Net benefits might also be calculated for participants (engagement in education, treatment programmes), their families and communities, and the general public. 

-----//-----

At present, payment-by-results or lead provider (sometimes called principal contract) systems do not adequately capture the actual value of the work done because they reward the agent (provider) for the primary product, while little commercial reward is allocated for ‘by-products’ such as social justice, humanitarian intervention, personal transformation, or community benefit. In short, a great deal of voluntary sector work is disregarded as having economic viability. There is no inherent reason (other than narrow misconceptions of what is financially worthwhile) why these outputs are not regarded as appropriate for society to support via governmental or social funding. 

Should there be markets in the criminal justice system? 

In criminal justice, we cannot ignore the important moral imperative which conventional political economy overlooks, but which is intrinsic to humanitarian intervention. There remains the strong public belief that working with criminally sanctioned persons is not like any other public services because of the solemn penal function of the criminal justice system (powers which one might argue should be limited by law to democratic and legally constituted bodies). The obligatory duty to punish and sanction, one argument goes, means that criminal justice is non-marketable (if one accepts that involvement in criminal justice work equates with participating in punishment). However, this does not always translate into a consensus whether this makes the penal/criminal justice sphere merely distinctive, or makes this an exception from the normal conditions of humanitarian work. 

Classic commercial and policy cost-benefit models do not adequately reflect the contribution of the voluntary/ third sector. Charities must achieve commercial viability, but their activities cannot be determined by profit generation if the sector is also to claim public legitimacy and create meaningful distinctions between themselves and commercial businesses. In this sense, cost-benefit analysis means that charities need to employ tools which take into consideration of the different goals of charitable organisations – some of which conflict with A bolder cost-benefit approach to capture the contribution of the voluntary sector in criminal justice each other. The interaction between sectors is complex, and there is still plenty of scope for policymakers and commissioning bodies to work with different sectors to devise a framework for assistance provided by different actors. It ought to be possible to devise actuarial instruments for assessing the holistic costs and benefits of working with those affected by criminal harms. Such tools would assist strategic decision making, not just in terms of financial planning, but in factoring in the other significant elements of reputation (lending or transacting your good name), recognisability with crucial stakeholders, personnel and organisational ‘costs’, reach and legitimacy with beneficiary groups and communities. There are no intrinsic reasons why the value of externalities and outputs cannot be recognised and rewarded by funders and stakeholders alike.