Never Mind the Quality!

I wondered whether the claim that Justice Transcribe is “improving the quality and consistency of case records” was somebody’s interpretation of what MoJ had said. It isn’t. That’s the Justice AI Unit’s own wording. And I have a real problem with the word “quality”. What exactly are we measuring?

If quality means a record is grammatically better, neatly structured, comprehensive-looking and contains the appropriate headings, then yes, Justice Transcribe probably produces “better” records. But that’s quality of documentation. It tells us absolutely nothing about the quality of the probation work that took place. That’s precisely what worries me.

Scratch beneath the beautifully constructed record, get through the “non-structured intervention”, “pro-social manner”, strengths, needs, actions and whatever else has been generated, and what actually happened in the room?

It might have been an excellent 45-minute piece of work involving challenge, reflection, professional curiosity and meaningful discussion. Or it might have been a fairly meaningless ten-minute check-in, a bit of signposting and “anything changed since I last saw you?” Yet Justice Transcribe potentially makes both look remarkably impressive on the page.

That isn’t necessarily because the AI has recorded anything untrue. The problem is that structure and polished language can give relatively mundane interactions a significance and depth they didn’t possess.

There is actually pilot evidence behind the “quality” claim. HMIP reports that 96% of practitioners in one pilot said the quality of records improved. Fine. But again: quality of the record isn’t quality of the intervention.

And the latest MoJ transparency publication is revealing in another way. We know 1.6 million meetings have been summarised and we’re given an illustrative estimate of 266,667 hours saved. But MoJ itself says that calculation doesn’t account for variation in meeting length, type, user practice or summary quality. So I’d quite like to see somebody properly evaluate what we’re now producing before “quality” becomes another accepted fact through repetition. I’d also like HMIP to look beneath the sparkle when they inspect it.

Don’t just audit whether the contact contains the right information. Don’t assume a beautifully structured entry demonstrates meaningful engagement. Somehow, and I accept this is incredibly difficult, go back to the actual practice underneath it. Because we’re potentially removing one of the few clues inspectors previously had. A poor record might once have indicated poor recording. 

An AI-generated excellent record can now potentially conceal pretty mediocre practice. And if that happens, we’ve created something rather dangerous: a system capable of making probation look better without probation actually becoming better. That’s not an improvement in quality. It’s an improvement in presentation.

Anon

*****
I work in Court completing PSRs, and it now takes far longer than it should to review contact logs and work out what's actually been going on. Justice Transcribe generates dense, jargon-heavy transcriptions padded with phrases like "crisis management" and "stabilisation" standing in for what wasn't actually done. It reads as though far more is happening than really is. Whatever happened to CRISSA and the principle of concise, clear logs? Now we get paragraphs of dense text that sound professional on first read, but strip away the language and the actual content is often minimal. There's also no record of body language or presentation …. details that can say more about someone's state than anything they tell you.

It's genuinely infuriating. And I understand this approach is coming for court reports next, meaning reports will start to sound identical: the same phrases, the same flat, robotic analysis. I cannot bare the thought.

Meanwhile, the new Probation Service's training says nothing about the quality of supervision within appointments … the actual craft of the job, which has been neglected for years. Building rapport and trust, asking the right questions, challenging behaviour appropriately, working in a trauma-informed way. Instead it's all about papering over cracks, ticking boxes, and doing just enough to get by.

Everything right now is about saving time and seeing people less. It's transactional, not relational. And in the process, the relationship with offenders is being stripped out entirely, leaving a culture of learned helplessness in its place.

Then there's the online reporting rollout. What about people who lose their phones every few days? Who don't have a smartphone or internet access? A significant proportion of the offending population struggles with reading, writing, and basic admin. Why would Mr X openly disclose on an online form that he's struggling with his mental health, had a domestic incident, lapsed into crack use, or is behind on rent — if he's barely met his PP, or only had a few brief check-ins before being moved to online reporting because he's deemed "stable"? So many of these individuals have real difficulties with attachment, trust and authority. Building enough rapport for someone to start opening up can take weeks, months, sometimes years. Body language and engagement tell you so much about how someone is really doing, and what's normal for them. None of that comes through online.

We're going backwards. So much for "assist, befriend, advise" …. it's become skim, tick, and signpost. What about the deep dive of truly understanding someone and their lived experiences.

Sixteen years in, and I want so badly for the soul of this service to be found again — because I still care. A lot. And I hate that I do, because it feels like HMPPS doesn't. Get rid of the civil service bureaucracy. Bring back the trusts, where there was local autonomy and room for creativity.

Anon

*****
I think you’ve absolutely hit the nail on the head. This is probably the best explanation I’ve read of why my unease about Justice Transcribe isn’t really about Justice Transcribe at all.

It may produce faster, neater, more consistent records, but none of that addresses the fundamental problems facing probation. It doesn’t reduce caseloads. It doesn’t create experienced practitioners. It doesn’t improve the quality of supervision. It doesn’t build relationships, professional curiosity or trust. It doesn’t give somebody the time to understand the person sitting opposite them. And, as you demonstrate from the other end of the process, it may actually make it harder to see what meaningful work has really taken place.

That’s the bit I think we’re missing when we celebrate hundreds of thousands of “hours saved”. Saved for what? If that time is genuinely returned to practitioners to spend thinking, investigating, building relationships and doing better work, fantastic. I’ll champion JT alongside anyone.

But everything else currently seems to be travelling in the opposite direction: fewer contacts, online reporting, supervision packages, early termination, enormous caseloads and increasing standardisation. In that context, JT starts looking less like technology creating space for better probation and more like technology helping an overstretched service process the same impossible volume more efficiently.

And your point about the records themselves is crucial. A concise contact which accurately tells the next practitioner what actually happened is surely more useful than three paragraphs of beautifully structured professional language which they have to decode to discover that very little happened at all.

Probation doesn’t have a writing problem. It has a capacity problem and, increasingly, a relationship problem. Justice Transcribe might solve the first ten minutes of administration. It won’t solve either of those.

Anon

*****
"..,now we get paragraphs of dense text that sound professional on first read, but strip away the language and the actual content is often minimal. There's also no record of body language or presentation …. details that can say more about someone's state than anything they tell you.

It's genuinely infuriating. And I understand this approach is coming for court reports next, meaning reports will start to sound identical: the same phrases, the same flat, robotic analysis. I cannot bare the thought.

Meanwhile, the new Probation Service's training says nothing about the quality of supervision within appointments … the actual craft of the job, which has been neglected for years. Building rapport and trust, asking the right questions, challenging behaviour appropriately, working in a trauma-informed way. Instead it's all about papering over cracks, ticking boxes, and doing just enough to get by.

Everything right now is about saving time and seeing people less. It's transactional, not relational. And in the process, the relationship with offenders is being stripped out entirely, leaving a culture of learned helplessness in its place."

This! This is the bit that resonates with me more than anything else in your comment. We’re becoming increasingly sophisticated at documenting probation while simultaneously hollowing out the actual practice of it.

Justice Transcribe can produce paragraphs of professional-sounding language, but strip that language away and what actually happened? What was explored? What was challenged? What changed? What did the practitioner notice? What did the person not say? How did they present? Was there hesitation, agitation, avoidance, a change in demeanour? Did something simply feel different because you’ve known this person for months and know what “normal” looks like for them?

None of that comes from better prose. It comes from time, attention, experience and a relationship. And this is where I think the direction of probation becomes genuinely worrying. At exactly the point we’re developing technology capable of making the written evidence of practice look increasingly impressive, we’re reducing the conditions in which good practice actually happens.

Less contact. Shorter contact. Online reporting. Enormous caseloads. Supervision packages. Signposting. Standardised processes. And apparently very little attention in all this new training to the actual craft of supervision.

How do you conduct a really good probation interview? How do you build enough trust to challenge somebody without losing them? How do you recognise avoidance rather than simply record an answer? How do you explore ambivalence? How do you sit with silence rather than fill it? How do you recognise when the person you’ve supervised for six months isn’t behaving like themselves today? Those aren’t soft extras around probation practice. That is probation practice.

And the point about court reports genuinely worries me too. A good PSR shouldn’t simply contain the correct information arranged under the correct headings. It should contain analysis. It should demonstrate that somebody has thought about this particular person, this particular offence, these particular circumstances and what an appropriate sentence might actually achieve.

If AI eventually makes every contact, assessment and report beautifully structured, grammatically perfect and professionally phrased, while the human interaction underneath becomes increasingly brief and transactional, we won’t have improved probation. We’ll simply have become much better at writing about a poorer version of it.

Anon