Communicating the value of evidence

I presented at a couple of conferences over the last few weeks about my BERRI system. And I was struck, once again, by how little weight is given to evidence when it comes to services that are commissioned in the social care sector. Various glossy marketing claims and slick consultants were successfully persuading commissioners and service managers that it was equivalent to use their systems and “metrics” (in which people gave entirely subjective ratings on various arbitrarily chosen variables) to using validated outcome measures. By validated outcome measures, I mean questionnaires or metrics that have been developed through a methodical process and validated with scientific rigour that explores whether they are measuring the right things, whether they are measuring them reliably, whether those measures are sensitive to change, and whether the results are meaningful. A pathway that then leads to an established scientific process of critical appraisal when those studies are presented at conferences, published and made subject to peer review.

But outside of the academic/scientific community it is very hard to prove that having a proper process is worth the time and investment it takes. It means that you are running a much longer race than those who work without evidence. At one event last week, I asked a question of a consultancy firm making hundreds of thousands of pounds out of “improving children’s social care outcomes”, about their basis for what they chose to measure, how they measure it, and how they had validated their claims. The answer was that they were confident that they were measuring the right things, and that having any kind of scientific process or validation would slow down their ability to make impact (aka profit). My answer was that without it there was no evidence they were making any impact.

They couldn’t see that their process of skipping to the doing bit was equivalent to thinking that architects, structural drawings, planning permission and buildings regulation control slow down building houses, and selling houses they’d built without all that burdensome process. Thinking anyone can build a house (or a psychometric measure to track outcomes) feels like an example of the Dunning-Kruger effect, the idea that those with the least knowledge overestimate their knowledge the most. But the worst thing was that those commissioning couldn’t see the difference either. They find the language of evidence to be in the domain of academics and clinicians, and don’t understand it, or its importance. We are in an age where expertise is dismissed in favour of messages that resonate with a populist agenda, and it seems that this even applies when commissioning services that affect the outcomes of vulnerable population groups. I don’t know how we change this, but we need to.

For those who don’t know, I’ve been working on BERRI for 12 years now, on and off, with the goal of being able to map the needs of complex children and young people, such as those living in public care, in a way that is meaningful, sensitive to change and helps those caring for them to meet those needs better. For as long as I’ve worked with Looked After children, there has been a recognition of the fact that this population does worse in life along a wide range of metrics, and a desire to improve outcomes for them for both altruistic and financial reasons. Since Every Child Matters in 2003, there have been attempts to improve outcomes, defined with aspirations in five areas of functioning:

  • stay safe
  • be healthy
  • enjoy and achieve
  • make a positive contribution
  • achieve economic well-being

A lot of services, the one that I led included, tried to rate children on each of these areas, and make care plans that aimed to help them increase their chances in each area. Each was supposed to be associated with a detailed framework of how various agencies can work together to achieve it. However, whilst the goals are worthy, they are also vague, and it is hard to give any objective score of how much progress a young person is making along each target area. And in my specific area of mental health and psychological wellbeing they had nothing specific to say.

As with so much legislation, Every Child Matters was not followed up by the following government, and with the move of children’s social care and child protection into the remit of the Department for Education, the focus shifted towards educational attainments as a metric of success. But looking primarily at educational attendance and attainments has several problems. Firstly it assumes that children in Care are in all other ways equivalent to the general population with which they are compared (when in fact in many ways they are not, having both disproportionate socioeconomic adversity and disproportionate exposure to trauma and risk factors, as well as much higher incidence of neurodevelopmental disorder and learning disability). Secondly it limits the scope of consideration to the ages in which education is happening (primarily 5-18, but in exceptional circumstances 3-21) rather than the whole life course. Thirdly it doesn’t look at the quality of care that is being received – which has important implications for how we recruit, select and support the workforce of foster carers and residential care staff, and what expectations we have of placement providers (something I think critical, given we are spending a billion pounds a year on residential care placements, and more on secure provision, fostering agencies and therapy services that at the moment don’t have to do very much at all to show they are effective, beyond providing food, accommodation, and ensuring educational attendance). Finally, it masks how important attachment relationships, and support to improve mental health are in this population. I can see that strategically it makes sense for politicians and commissioners not to measure this need – they don’t want to identify mental health needs that services are not resourced to meet – but that is significantly failing the children and young people involved.

In my role as a clinician lead for children in Care and adopted within a CAMH service, I kept finding that children were being referred with behaviour problems, but underlying that were significant difficulties with attachment, and complex trauma histories. I was acutely aware that my service was unable to meet demand, leading us to need some system to prioritise referrals, and that there was a lot of ambiguity about what was in the remit of CAMHS and what was in the remit of social care. I wasn’t alone in that dilemma. There were a lot of defensive boundaries going on in CAMHS around the country, rejecting referrals that did not indicate a treatable mental health condition, even if the child had significant behavioural or emotional difficulties. The justification was that many children were making a normal response to abnormal experiences, and that CAMHS clinicians didn’t want to pathologise this or locate it like an organic condition inside the child, so it should best be dealt with as a social care issue.

On the other hand, I was mindful of the fact that this population have enormous mental health needs, having disproportionately experienced the Adverse Childhood Experiences that are known to lead to adverse mental and physical health outcomes. Research done by many of my peers has shown that two thirds to three quarters of Looked After children and young people score over 17 on the SDQ (the Strengths and Difficulties Questionnaire – the government mandated and CORC recommended measure for screening mental health need in children) meaning they should be eligible for a CAMH service, and various research studies have shown that 45% of LAC have a diagnosable mental health condition, but the resources are not available to meet that need. As The Mental Health Foundation’s 2002 review entitled “Mental Health of Looked After Children” put it:

Research shows that looked-after children generally have greater mental health needs than other young people, including a significant proportion who have more than one condition and/or a serious psychiatric disorder (McCann et al, 1996). But their mental health problems are frequently unnoticed or ignored. There is a need for a system of early mental health assessment and intervention for looked-after children and young people, including those who go on to be adopted.

My initial goal was to develop a new questionnaire to cover the mental health and psychological wellbeing issues that this population were experiencing, as well as considering attachment/trauma history and the child’s ability to trust others and form healthy relationships, and the behaviours that these often expressed through. I was also interested in what issues determined the type of placement given to a child, and the risk of placement breakdown, as well as what opened doors to specialist services such as therapy, and whether those services and interventions really made any difference. I therefore ran two focus groups to explore what concerns carers and professionals had about Looked After children and young people, and asked them about what they saw that might indicate a mental health problem, or any related concerns that led people to want my input, or that caused placements to wobble or break down. One group contained foster carers and the professional networks around them (link workers, children’s social workers, the nurse who did the LAC medicals, service managers) and one contained residential care workers and the professional networks around them (home managers, children’s social workers, the nurse who did the LAC medicals, service managers). I wrote their responses down on flip-charts, and then I sorted them into themes.

I had initially thought that it might cluster as behavioural and emotional, or internalising and externalising, but my items seemed more complex than that. In the end there were five themes that emerged:

  • Behaviour
  • Emotional wellbeing
  • Risk (to self and others)
  • Relationships/attachments
  • Indicators (of psychiatric or neurodevelopmental conditions)

The first letters gave me the name for the scale: BERRI. I then piloted the scale with various carers, and then with a group of clinical psychologists involved with CPLAAC (the national network within the British Psychological Society that contained about 300 Clinical Psychologists working with Looked After and Adopted Children that I was chair of for about six years). I then added a life events checklist to set the issues we were identifying in context.

The working group I chaired in 2007 on the state of outcome measurement for Looked After and adopted children (on the invitation of CORC) came to the conclusion that no suitable metrics were available or widely used. We therefore agreed to further develop and validate the various tools that members of the group had home-brewed, including my BERRI. There was acknowledgement that it takes a lot of work to develop a new psychometric instrument in a valid way, but a consensus that this needed to be done. So I resolved to find a way to follow that proper process to validate and norm BERRI, despite the lack of any funding, ring-fenced time or logistical support to do so. The first challenge was to collect enough data to allow me to analyse the items on the measure, and the five themes I had sorted them into. But I didn’t have the resources to run a research trial and then enter all the data into a database.

My way around this barrier was to get my peers to use the measure and give me their data. To do this I took advantage of some of the technically skilled people in my personal network and developed a website into which people could type anonymous BERRI scores and receive back a report with the scores and some generic advice about how to manage each domain. I tested this out and found my peers were quite enthused about it. We then had a formal pilot phase, where 750 BERRIs were completed by Clinical Psychologists about children and young people they were working with. I then talked about it with some young people and care leavers to check that they felt the areas we were covering were relevant and helpful to know about*. Then I started to use the system in a large pilot with residential care providers and developed tools to focus in on particular concerns as goals to work on, and track them day by day or week by week, as well as creating tools to give managers an overview of the progress of the children in their care. We’ve had a lot of feedback about how useful and game-changing the system is, and how it has the potential to revolutionise various aspects of commissioning and decision-making in children’s social care.

But I really wanted the process to be one in which we were truly scientific and based our claims on evidence. I’ve never marketed the BERRI or made claims about what it can do until very recently, when I finally reached a point where we had evidence to substantiate some modest claims**. But to me the process is critical and there is still a long way to go in making the data as useful as it can be. So from day one a process of iterative research was built in to the way we developed BERRI. As soon as it was being used by large numbers of services and we had collected a large data set we were able to look closely at how the items were used, the factor structure, internal consistency and which variables changed over time. We ran a series of validity and reliability analyses including correlations with the SDQ, Conners, and the child’s story – including ACEs, placement information and various vulnerability factors in the child’s current situation. But even then I worried about the bias, so a doctoral student is now running an independent study of inter-rater reliability and convergent/divergent validity across 42 children’s homes.

BERRI will always be developed hand in hand with research, so that there is an ongoing process of refining our outputs in light of the data. The first step in that is getting age and gender norms. But the data can also indicate what we need to do to improve the measure, and the usefulness of the output reports. For example, it seems that it might be meaningful to look at two aspects of “Relationships” being distinct from each other. If the evidence continues to show this, we will change the way we generate the reports from the data to talk about social skills deficits and attachment difficulties separately in our reports. We might also tweak which items fall into which of the five factors. We also want to check that the five factor model is not based on the a priori sorting of the items into the five headings, so we are planning a study in which the item order is randomised on each use to repeat our factor analysis. We also want to explore whether there are threshold scores in any factor or critical items within factors that indicate which types of placements are required or predict placement breakdown. We might also be able to model CSE risk.

The results to date have been really exciting. I have begun to present them at conferences and we are currently preparing them to submit for publication. For example, I am currently writing up a paper about the ADHD-like presentation so many traumatised children have, and how we have learnt from our BERRI research that this reflects early life ACEs priming readiness for fight-or-flight rather than proximal events or a randomly distributed organic condition. But the findings depend on all the groundwork of how BERRI was developed, our rigorous validation process and the data we have collected. It is the data that gives us the ability to interpret what is going on, and to give advice at the individual and organisational level.

So you’ll forgive me if I’m somewhat cynical about systems that request a subjective likert rating of five domains from Every Child Matters, or an equally subjective score out of 100 for twelve domains pulled from the personal experience of the consultant when working in children’s social care services, that then claim to be able to map needs and progress without any validation of their methodology, areas to rate, sensitivity to change or the meaning of their scores. Having gone through the process the long way might put me at a commercial disadvantage, rather than going straight to marketing, but I like my houses built on the foundations of good evidence. I can feel confident that the load bearing beams will keep the structure sound for a lifetime when they are placed with precision and underpinned by the calculations and expertise of architects, structural engineers, surveyors and buildings control, rather than cobbled together as quickly as possible, marketed with amorphous claims and sold on rapidly to anyone who will pay for them. After all, I’m not in it to make a quick buck. I know my work is a slow and cumulative thing, and BERRI still has a long way to go before it can create the greatest impact. But my goals are big: I want to improve outcomes for children and young people who have experienced adversity, and I want that impact to influence the whole culture of children’s social care provision in the UK and to continue to be felt through the generations. And to do that, I need to build the thing properly.

*I’m still intending to act on the advice to also have a strengths scale to recognise resilience and positive factors, so that it doesn’t feel like we see the children purely as a list of problems. However, I didn’t want to duplicate the work of others, so I am following up a potentially exciting lead in terms of a collaboration with the Mulberry Bush School, who have explored the positive factors they have seen as markers of progress in their environment.
** that carers, therapists and managers find it useful and easy to use, that using the BERRI pathway demonstrated an improvement of 14% over 6 months for the first 125 children placed on the system, and that BERRI has the basic statistical qualities that suggest sufficient validity for use. We also have some testimonials, including a commissioner who used BERRI to map the needs of 15 high tariff children and found four suitable to move to foster or family placements with support, saving nearly half a million pounds per year from his budget – a finding we would like to replicate with a much larger study, given the opportunity.

 

 

Giving psychology away – the positive ripples of training

When I left the NHS I wondered whether I would have enough work to keep me busy, and felt like the expert witness work I do the family court would probably be the main strand of my income for the foreseeable future. However, perhaps because of my book, I have increasingly been asked to speak at conferences, provide training to various groups of professionals, and consult to organisations – particularly on the topic of working with attachment, trauma and the impact of maltreatment on children. This has become an enjoyable sideline where I can cascade psychological thinking to a wider audience, meet lots of new people, learn from others, and earn a good daily rate without the same emotional weight as doing court work.

I also like any work that involves improving the quality of services, particularly for complex or marginalised client groups. It was therefore very interesting to me when I was asked to meet up with some of the directors of a children’s home company, Keys Childcare, about offering training and consultation to a 20 placement project they were running about an hour away from my base. After some initial meetings, we decided that I would offer some staff training sessions (on attachment, trauma and the impact of maltreatment on children’s development) to all the staff, and then help them to implement a more systematic program of care planning and outcome measurement. The aim was to make decisions about which young people needed therapy and what form of therapy they received more evidence-based, as well as to ensure that all of the staff were involved in making the care for each young person as therapeutic as possible.

This fitted in really well with an outcome measurement system I had been developing for several years, using my ‘BERRI’ checklist of Behaviour, Emotional well-being, Relationships, Risk and Indicators of something requiring further assessment (such as markers of neuro-developmental conditions, or more usual behaviours, perception or thinking which would merit more specialised interventions). I had already developed a system whereby data could be entered into the BERRI online and produce a feedback report, and scores could be compared to track progress over time. This could produce a visual representation of a young person’s needs across the five dimensions, and allow us to be more holistic in our thinking about how to support them to make optimum progress.

radar

After discussion with friends and colleagues, I had started to develop a system for targeting particular concerns to address in the care plan. This involves identifying around three specific concerns from the BERRI and tracking them with greater frequency, typically daily, whilst giving the staff various strategies to try to address them. The target behaviours can be entered into the online system, which can then produce graphs to show progress and to identify any triggers. For example, in the chart below I’d hope that staff would instantly wonder what triggered the big green spike of aggressive behaviour.

tracking

I spent a couple of days each month with Keys, completing the staff training and improving the online system so that it would meet their needs. We also developed a pathway in which there would be a psychological screening assessment when young people arrive in placement, and their paperwork would be properly digested to ensure we had all relevant information from their history as well as an initial BERRI checklist, and we would bring this together in a meeting where the professionals around the child, their care team, the therapist linked to the home and myself would put together a therapeutic care plan document and identify the targets to work on over the coming months. The care plan could contain a summary of the history, a formulation section giving insight into the meaning of the behaviours and concerns the child was showing, then the targets we had agreed to track, and the strategies we had identified to help work on those concerns.

This was seen as a refinement to the existing therapeutic plan the homes in the project were already using, and embraced very positively. We called the approach Psychologically Informed Care-planning and Intervention, with the acronym PICI (which Jonathan Stanley, Chair of ICHA, said was an acronym that led naturally to a marketing slogan: “if you want the best for your child, get PICI”).

After a few months of seeing the changes within the local project, and gathering feedback from staff, social workers and commissioners, Keys decided to expand my remit. They asked me to do training for staff outside of my local project, and then to do some consulting to the wider organisation. They subsequently offered me a job as Clinical Director of Keys, with a remit to train all of their staff and implement the pathway across the whole organisation. This will take me three days per week for the next two years! They have also agreed to employ a psychology team to implement the project in each region, and this week we appointed the first qualified clinical psychologists. In time this team will grow to cover all of Keys services across the UK. We will also be taking on 3.5 assistant psychologists and more therapist time, so that we have a consistent level of consultation to each home in the company (with additional therapy time in the attachment homes where every child placed gets direct therapeutic input). This is super exciting, and I hope that we are able to evaluate the impact on both staff and the young people we look after.

Keys have embraced the new pathway and outcome measurement system as part of a wider push to improve staff training (which now also includes a modular training program that every staff member will work through) and to be able to evidence the effectiveness of their care. This includes a desire to be more evidence-based in how they make decisions about placements and interventions. The changes also coincided with the tenth anniversary of the company moving into childcare and buying the initial ten Keys homes from their previous owners, and also with a rebranding exercise. When they revealed the new branding a few weeks ago in Belfast I was delighted to see that Keys now describe themselves as providing “psychologically informed care for children and young people” and see this as a way to improve the quality of their placements (and through this to gain better Ofsted inspection grades, and become the placement of choice when commissioners seek to place complex young people).

It seems like a natural progression from the kind of consultation I was doing in a CAMHS LAC team, but on an organisation wide scale. My small amount of time has the power to influence more children by working indirectly. By “giving psychology away” I hope I am giving increased insight to those doing the day to day care, and slightly changing their way of working with the young people. Instead of a child getting one hour per week with me, they are getting 38 hours per week with a slightly more psychologically minded caregiver, and my influence can reach 320+ children and nearly 1000 staff.

The staff within Keys have been really enthusiastic about the new pathway and have given me positive feedback for the training days, so the although there is a lot of travelling and a very ambitious timetable for implementation I am really enjoying the work so far. I also get surprisingly positive feedback from training elsewhere, which is so lovely to receive. I think it really stands out for me given how few positive feedback loops there are in most clinical roles nowadays (beyond the occasional client or colleague sending a thank you card). I get a real buzz from positive tweets about my training or keynotes, gaining high scores on conference feedback forms, positive reviews of my book, or complimentary endorsements on LinkedIn.

I went up to Glasgow last week to train some staff to run my group program for parents and carers of maltreated children (Managing Behaviour with Attachment in Mind) and really enjoyed that also. Glaswegians really are the friendliest people in the world, and the group I trained were clearly hugely empathic to the children they work with so they were a delight to train. As I was leaving someone said to me that “training is a way to change the lives of thousands of children”. I think I had made some comment to brush this off (perhaps that it was the clinicians working directly with the families whose relationships were vital, rather than a day from a person like me), but she wasn’t having any of it. She said “no I mean it. I will interact different with the families I see tomorrow and in the future because I spent the day with you, and each person who came to the training will do likewise”. She explained emphatically that she had literally meant that in one day I had done something that would ripple out and change huge numbers of lives. That was pretty humbling to think about.

I’ve also had some exciting discussions about research, service evaluation and different kinds of therapeutic work (for example, working in partnership with collaborative lawyers). So I have decided to have a break from doing expert witness work over the summer and really have a think about what I most want to do. I need to weigh up all the choices in front of me, prioritise the things that make the most impact and/or give me the most satisfaction, and put some energy into finding allies with shared goals (including seeing whether I can recruit another clinical psychologist to LifePsychol in Milton Keynes to pick up some of the therapy and court work). I also want to reclaim my life outside of working hours, and to consider where in the country it makes most sense to be located. But it feels like all the effort I have invested in various projects to date is finally nearing fruition, and there are lots of interesting things on the horizon.

Long ago, I had a task list written on the whiteboard in my office, and Gilly had made me laugh by adding to the bottom of this “build secret volcano base, take over the world”, to reflect my ambitions to change practise on a wide scale. Maybe it isn’t such an unrealistic aim after all.