Allegheny Family Screening Tool evaluation

Researchers from Stanford University (US) have evaluated the Allegheny Family Screening Tool (AFST), a child welfare predictive risk modelling tool developed by a research team led by Rhema Vaithianathan, and implemented by Allegheny County (US) in 2016.

Key findings are that implementing the AFST and accompanying policy changes improved the accuracy of maltreatment referral screening decisions (i.e., which referrals to ‘screen-in’ for investigation) and reduced racial disparities in case openings between black and white children.

The design and implementation of the AFST has been a multi-year project for Allegheny County and a research team led by Rhema Vaithianathan.  The project has included community and stakeholder meetings to solicit input, an intentional procurement process, an ethical review, a validation study, and independent and rigorous process and impact evaluations.

The Stanford research team studied the tool’s indirect and direct effects on the maltreatment screening process, including decision accuracy, workload, and consistency. They found that implementation of the tool saw no adverse consequences and increased the accurate identification of children who needed further intervention services, without increasing the workload on investigators. Researchers also documented reductions in racial disparities in case openings following investigations, resulting from modest but statistically significant increases in cases opened for white children coupled with declines in the rate at which black children were screened in for investigation.

“We implemented the AFST at the end of a lengthy community engagement process in the hope that we could improve accuracy of decision-making at our child welfare’s front door, when call screening staff decide whether to investigate a situation that has come to our attention,” said Marc Cherna, Director of the Allegheny County Department of Human Services (DHS). “We are encouraged that the AFST has shown positive results by increasing accuracy, while preserving clinical judgement, and we believe that it has great potential as we continuously strive to improve our ability to keep children safe.”

The findings of the impact evaluation will help to guide DHS in making additional improvements to the AFST and adaptations to its implementation.

“Allegheny County committed to an independent evaluation of this tool from the start, demonstrating their commitment to knowledge building and transparency,” said Rhema Vaithianathan.  “For the child welfare field at large, the evaluation offers invaluable information and guidance, as more and more agencies consider predictive analytics to support better decisions. We often hear about concerns that predictive analytics might worsen racial disparities, so we are pleased the evaluation finds that the AFST can support decisions that actually reduce racial disparities in the child welfare system.”

Read more

Impact Evaluation Summary of the Allegheny Family Screening Tool

Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County’s Child Welfare Office