This project or publication was produced before or during the merger of What Works for Children’s Social Care (WWCSC) and the Early Intervention Foundation (EIF).
Social workers make potentially life-changing decisions every day. These range from, choosing whether to accept a referral, undertaking a child protection investigation, pursuing care proceedings, and making judgements about the likelihood of future significant harm.
But people are very poor at predicting the future. In part, these difficulties are caused by our susceptibility to cognitive bias (systematic errors in thinking that typically occur when we interpret information). A recent review of the literature found that there are currently no well-evidenced interventions to help mitigate the effects of cognitive bias in social work. In other fields, such as politics and economics, various interventions have been developed and found to significantly improve prediction of outcomes (forecasting abilities).
This study examines the impact of a Checklist intervention through a randomised controlled trial, exploring the differences it made in relation to forecasting abilities and confirmation bias.
The Checklist Intervention aimed to help social workers avoid some of the negative effects of confirmation bias and improve forecasting accuracy using a low-cost and easy to administer tool.
The trials aimed to evaluate the effectiveness of the Checklist interventions on the difference it made in relation to forecasting abilities and a direct measure of confirmation bias.
Specifically, it aimed to answer the following questions:
This study involved a randomised controlled trial (RCT), where a total of 87 social workers and one student social worker took part in an online survey. Each participant read two baseline case studies and answered four questions for each case study about the likelihood of different outcomes. Participants were then randomly allocated to the control group or the intervention group. Participants in the control group were asked to complete two endline case studies and the Wason Selection Task before accessing the checklist intervention. Participants in the intervention group were asked to complete the checklist intervention first, before completing the two Endline case studies, and the Wason Selection Task.
To assess whether the Checklist intervention had an impact on social workers forecasting abilities, multiple regression was used on Endline Brier scores. There was a post-intervention reduction in Brier scores for participants in the intervention group (indicating improved forecasting accuracy). However, this difference between the intervention and control groups was small and not statistically significant. Thus, the checklist intervention did not have a significant impact on forecasting accuracy. Multiple regression was also used to determine the impact of the checklist intervention on confirmation bias. Findings indicated the checklist intervention did not have a significant impact on confirmation bias. The personal and professional characteristics of the participants were not associated with forecasting accuracy or confirmation bias.
Notably, Checklist interventions are typically designed to be used within actual decision-making settings and are evaluated accordingly. In this study, because of the Covid-19 pandemic, we had to design an intervention that could be used online. This limits our ability to generalise from these findings, to what might happen if the same (or a similar) checklist were used in real-life decision-making environments. This is because any similar tools would most likely be used differently in practice, for example in discussion with a supervisor in a supervision session. Other limitations of the study include the lack of a pre-intervention measure of confirmation bias, and the smaller-than-intended sample size, lowering the likelihood of detecting significant effects (or non-effects) of the intervention.
Overall, we found no evidence that the checklist intervention made a significant difference in relation to forecasting accuracy or confirmation bias. These results, especially when considered alongside those of a previous study, suggest that interventions to improve forecasting accuracy (or mitigate cognitive bias) in social work would need to be much more in-depth than the relatively brief online interventions we have tested here and previously. For example, through longer training courses.
Rated 1: Set up and delivery is low cost, equivalent to an estimated unit cost of less than £100.
Rated 2: Set up and delivery is medium-low cost, equivalent to an estimated unit cost of £100–£499.
Rated 3: Set up and delivery is medium cost, equivalent to an estimated unit cost of £500–£999.
Rated 4: Set up and delivery is medium-high cost, equivalent to an estimated unit cost of £1,000–£2,000.
Rating 5: Set up and delivery is high cost. Equivalent to an estimated unit cost of more than £2,000.
Set up and delivery cost is not applicable, not available, or has not been calculated.
Click here for more information.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.
Supporting children’s mental health and wellbeing: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Preventing child maltreatment: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Enhancing school achievement & employment: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Preventing crime, violence and antisocial behaviour: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Preventing substance abuse: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Preventing risky sexual behaviour & teen pregnancy: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Preventing obesity and promoting healthy physical development: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient.
Rated 2: Has preliminary evidence of improving a child outcome from a quantitative impact study, but there is not yet evidence of causal impact.
Rated 2+: Meets the level 2 rating and the best available evidence is based on a study which is more rigorous than a level 2 standard but does not meet the level 3 standard.
Rated 3: Has evidence of a short-term positive impact from at least one rigorous study.
Rated 3+: Meets the level 3 rating and has evidence from other studies with a comparison group at level 2 or higher.
Rated 4: Has evidence of a long-term positive impact through at least two rigorous studies.
Rated 4+: Meets the level 4 rating and has at least a third study contributing to the Level 4 rating, with at least one of the studies conducted independently of the intervention provider.
Rating has a *: The evidence base includes mixed findings i.e., studies suggesting positive impact alongside studies, which on balance, indicate no effect or negative impact.
Click here for more information.