How do we find the right fit when selecting an evidence based program?

Child welfare agencies have an interest in providing the most effective programs available for the children and families they serve. Toward that end, child welfare administrators are expanding their use of evidence based practices (EBP) and reliance on evidence when making programming and policy decisions.^1,2 Using programs with reliable research evidence can help child welfare leaders use resources more efficiently, increase client satisfaction, and improve safety and permanency outcomes for children and families involved with child welfare.^3,1 However, achieving these outcomes happens only if the program addresses an identified need and is compatible with, or “fits,” your agency context. Deciding what EBP to integrate into your agency or system requires analyzing existing data and clearly understanding the need or issue you are attempting to address.

Why is it important to find the right fit?

While many programs have been proven to be effective at improving safety and permanency for children and families involved with the child welfare system, the research may apply only to certain populations of children, to families with specific characteristics, or to a subset of specific agency contexts. Other factors that contribute to determining a program’s suitability include the specific issue the program addresses, cultural appropriateness for the population served, the fit of the program within the culture of your agency, logistical challenges that may be pertinent to delivering the intervention, and agency resources.⁴ Improving the fit between an EBP and the agency implementing it increases the likelihood of an EBP’s adoption.⁵

Key questions

• What data can we analyze/do we need to better understand the root cause of the issue?
• What are the core components that have been found to improve outcomes and must be implemented for a potential EBP to be successful?
• What components of the EBP can be adapted to better fit our context?

What strategies can be used to help select a program?

You can improve your selection of an EBP by taking the time to identify a specific problem you want to solve and the target population you want to serve, understand how different sources present evidence of effectiveness, and examine the fit between the agency and the EBP. The following strategies can help you make the best possible decision for your agency.

Identify a problem you want to solve or a service need to address, including what client population is in need. The first step is to identify a specific issue to address. As the administrator of an agency, you may have some sense of the outcomes you would like to improve, but it is important to analyze data to determine whether existing practices or programs are meeting agency goals and for which groups or clients. Possible data sources include internal agency monitoring and review data, federal monitoring or other external oversight data, and data obtained during an investigation.⁶ For example, recent studies in Illinois, Montana, Ohio, and Texas have identified specific sub-groups or clusters of youth with distinct patterns of need to enable those states to better match child needs with the right interventions.

Identify why the issue or service need exists. After identifying the issue, your next step is to figure out the root cause(s), or why the issue or service need exists. You must understand the root cause(s) of the issue before being able to address it. One element of this process of finding the root cause involves determining whether your agency or system can address that cause.⁶ You can perform this analysis by continuing to ask why an issue or service need exists and examining the data to figure out the factors associated with the problem. In addition to SACWIS data, possible data sources include literature reviews, focus groups, surveys, and case record reviews.

Understand how different sources present evidence of effectiveness. Selecting a program requires you to examine the evidence of effectiveness of various programs. To help you learn more about evidence based programs, there are multiple websites or clearinghouses designed to help you search for a program that addresses your identified issue.⁷ Searching for possible programs can be challenging, as each website presents and rates the evidence of effectiveness differently.^7,2 Some clearinghouses simply present the evidence that exists, while others have defined standards of evidence and provide a rating for the program based on those standards.  For example, some clearinghouses rate programs that were evaluated multiple times with rigorous evaluations as “evidence-based,” while those that have been evaluated using a less rigorous design are called “promising practices.”¹

Ask what the use of a particular EBP would mean in your jurisdiction. Does it fit? Using information about the effectiveness of programs to decide which EBP would work best in your agency or system is not enough. Additional information about the environmental context, the implementation methods, or adapting the program to diverse contexts and populations are also necessary to inform your decision.⁷

However, information about how different contextual variables affect outcomes is not often available from clearinghouses.⁷ As a result, you must take the additional step of examining how each program would work in your context.^2,4 This may entail interviewing the program developer to gather these details before committing to the implementation of an EBP. Another suggestion is to consult with a jurisdiction that has implemented the EBP you are considering.

Sample “Fit” questions

• Does the program address the identified issue?
• Does my staff meet program delivery qualifications? If not, could they be trained?
• Can our agency provide quality assurance oversight?
• Can the program be adapted for use with the population served by our agency?
• Do we have the money to implement this program? If not, can we get the money for implementation?
• Is the program sustainable? For example, how might the program be affected by leadership change and staff turnover?  And how much does it cost to keep the program running in terms of materials, staff coaching, fidelity assessment, and accreditation and certification?

Conclusion

There is no proven method that ensures every program you implement is a perfect fit for your agency. Likewise, there are no assurances that a chosen program will improve outcomes. But clearly identifying the issue and the root cause, using practices and programs with reliable research evidence, and ensuring the chosen program fits your agency are necessary steps when trying to improve safety and permanency for children and families.

1 Pew-MacArthur Results First Initiative. (2014). Evidence based policy making: A guide for effective government. Retrieved from The Pew Charitable Trusts website: http://www.pewtrusts.org/~/media/assets/2014/11/-evidencebasedpolicymakingaguideforeffectivegovernment.pdf
2 Tseng, V. (2012). Sharing child and youth development knowledge: The uses of research in policy and practice. Society for Research in Child Development, Social Policy Report 26(2). Retrieved from: https://www.srcd.org/sites/default/files/documents/spr_262_fiinal.pdf
3 DuMont, K., Wulczyn, F., Anderson, C., Samuels, B., Weiner, D., Danielson, D., Killos, L., & Roller White, C. (2016). Assembling the pieces: Research, policy, and practice in child welfare [Issue brief]. Chapin Hall, Casey Family Programs, and William T. Grant Foundation. Retrieved from: https://www.casey.org/media/assembling-the-pieces.pdf
4 Tseng, V., & Nutley, S. (2014). Building the infrastructure to improve the use and usefulness of research in education. In Using research evidence in education (pp. 163–175). Springer International Publishing.
5 Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.
6 Permanency Innovations Initiative Training and Technical Assistance Project. (2016). Guide to developing, implementing, and assessing an innovation. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. Retrieved from https://www.acf.hhs.gov/sites/default/files/documents/cb/guide_vol2_exploration.pdf
7 Horne, C. S. (2016). Assessing and strengthening evidence-based program registries’ usefulness for social service program replication and adaptation. Evaluation Review, http://journals.sagepub.com/doi/abs/10.1177/0193841X15625014