Public child welfare staff knowledge, attitudes, and referral behaviors for an evidence based parenting program

The goal of the current paper is to report on results from a survey of public child welfare staff in one state regarding their knowledge, attitudes, and behavior regarding SafeCare, an evidence-based mo-del that had been implemented for a few years in this particular state. SafeCare is a home-based, behavioral parenting program designed to address child neglect and physical abuse. SafeCare addres-ses three skill deficits that are proximal antecedents to child neglect and physical abuse: positive parenting skills, home safety, and child health care skills. The evidence base for SafeCare indicates that it can reduce child welfare recidivism (Chaffin, Hecht et al., 2012), improve parenting skills ( Carta, Lefever, Bigelow, Borkowski, & Warren, 2013 ), decrease parental depression ( Chaffin, Bard, Bigfoot, & Maher, 2012 ), increase program completion ( Damashek, Doughty, Ware, & Silovsky, 2011 ), and improve service satisfaction ( Damashek, Bard, & Hecht, 2012; Silovsky et al., 2011 ).

In this particular state, SafeCare had been implemented since 2009 by trained workers from private community-based agencies to deliver the model. Over 200 private providers have been trained throughout the state, but service utilization for SafeCare was gene-rally low ( Whitaker et al., 2012 ). Several attempts have been made to increase service utilization, including intensive training workshops focusing on the program, referral procedures, appropriate families, and reimbursement infrastructure across the majority of service regions in the state. In order to further examine the low service utilization, a survey was conducted in 2012 to assess public staff know-ledge and attitudes about SafeCare, barriers to SafeCare utilization, and referral practices for SafeCare.

In understanding this particular implementation, it is important to note that SafeCare was implemented alongside a standard paren-ting program which had no specific curriculum or protocol for deli-very (referred to in this paper as “standard parenting”). That is, staff could refer to either SafeCare or the standard parenting. Thus, in the rollout of SafeCare, the existing non-evidence-based parenting program was not removed, something has been referred to as de-adoption or exnovation ( Stirman et al., 2012 ). Second, because of differential levels of effort required for SafeCare versus standard parenting, reimbursement structures for each program were different, and SafeCare was the longer and more costly of the two programs. There were also different procedures and expectations regarding referring clients to SafeCare versus standard parenting. These issues – costs and administrative procedures – were suspected barriers to implementation and were assessed directly in the survey.

The survey was designed to assess factors thought to be related to service utilization. We assessed familiarity with and knowledge about SafeCare, attitudes toward and perceived effectiveness of SafeCare and standard parenting, perceived barriers to referrals, and actual SafeCare referrals. We also measured some aspects of the res-pondent, including the job position with an interest in examining differences between case managers and other supervisory and administrative staff. Case managers are the front line staff and as such make the bulk of the service referrals in child welfare. However, as the front line staff, they may be the last to be informed of new initiatives. Because of frequent turnover in child welfare positions, we also pay attention to job tenure. Newer staff may have had less of an opportunity to learn about a new EBP such as SafeCare, but may also may be more open to new service models ( Whitaker et al., 2012).

Thus, the specific goals of this article are:

• 1.

  To describe the knowledge, familiarity, and attitudes about SafeCare among public child welfare staff three years into a statewide implementation.

• 2.

  To examine referral practices, barriers to referrals, and predictors of referrals.

• 3.

  To examine differences by position and length of tenure at the child welfare system.

Respondent work characteristics . Job position was measured with a single item assessing the type of position (case manager, field program specialist, administrator, county or regional director, or other). We also assessed the employment unit of the respondent (investigations, family preservation, permanency, or other), the region in which they worked, and how long they had worked for the public child welfare system (1 year or less, 2-3 years, 3-5 years, 5-10 years, 10+ years).

Familiarity with SafeCare was measured with the single question, reading, “Overall, how familiar are you with the SafeCare program?” Respondents indicated their familiarity using a four-point Likert scale, 1 = not at all, 2 = slightly, 3 = somewhat, 4 = very.

Knowledge of SafeCare was measured in a few different ways. First, respondents were asked to list the three skills SafeCare targets (parent-child interactions, health, and safety), which were reviewed in the marketing and training workshops given in service regions, and open-ended responses were coded. A score of 0-3 representing the number of modules successfully identified was created for each participant. A dichotomous variable was also created indicating whether participants correctly identified any of the three modules. Finally, respondents were asked to identify the target age group for SafeCare via a multiple-choice question (0-5, 6-12, 12-18, 0-18), and responses were coded as correct or not.

Attitudes. A series of attitude items were presented to measure attitudes toward SafeCare and standard parenting. Based on a factor analysis, three scales were developed for analyses in this paper. The first two scales assessed satisfaction with SafeCare (5items) and satisfaction with standard parenting services (5 items), and included parallel items for each scale. Sample items are: “SafeCare (standard pa-renting) addresses family's needs very well” and “How much does SafeCare (standard parenting) seem to help families you refer?” Scale scores were created by averaging across items. A third scale, titled concerns about SafeCare , was formed by averaging three items (“SafeCare is too expensive”, “The referral process and paperwork for SafeCare is too cumbersome”, “SafeCare is no better than our usual fami-ly preservation programs”).

Perceived barriers to making SafeCare referrals were measured with 15 items. Respondents rated the extent to which each item posed a barrier to SafeCare utilization using a 4-point scale (1 = not a barrier, 2 = a small barrier, 3 = a moderate barrier, 4 = a large barrier ). Factor analyses indicated that barriers formed two factors: (1) lack of knowledge about SafeCare (7 items, e.g., “I have never heard of SafeCare”), (2) lack of fit between SafeCare and clients/context (8 items, e.g., “I prefer families to get parenting services other than SafeCare”, “The family needs other services like substance use counseling, domestic violence counseling, instead of SafeCare”). Scores for each barrier factor were created by averaging the items within the scale.

SafeCare referrals were measured by asking respondents if they had made any SafeCare referrals (“Have you made any referrals to agencies for FVS/SafeCare services?” yes/no). Because many higher level administrators may have taken the survey, respondents had the option to indicate that it was not their job to make referrals to providers. Respondents were also asked about how many referrals they had made. Because relatively few staff made referrals ( n = 43), ana-lyses focus only on whether or not each made a referral and not the number of referrals.

No information was available on the sample frame or the total number of potential respondents. A total of 294 respondents completed the survey. Respondents were case managers ( n = 167, 59%), supervisors ( n = 44, 15.5%), administrators or field program specia-lists ( n = 24, 8.5%), regional or county directors ( n = 17, 6%), and other ( n = 32, 11%). Almost 44% of participants worked for the child welfare system for 10 or more years, another 28% worked for the child welfare system for between 5 and 10 years, 17% percent between two and five years’ experience, and 12 percent had one year or less experience. Thus, overall, respondents represented an experienced workforce. Staff from all regions of the state responded to the survey. No additional demographic data were collected about respondents.

To describe the knowledge, attitudes, and referral practices of child welfare staff, simple descriptive statistics were used (frequencies, means). To examine differences in knowledge, attitudes, and practices by position type and job tenure, chi-square, t -tests, and ANOVAs were used, depending on the distribution of the dependent variable. Logistic regression was used to model predictors of SafeCare referrals.

Of the 294 respondents, 72 completed only the initial workforce questions (position, job tenure, region of work) and no additional parts of the survey, and thus, those 72 respondents do not appear in the analyses. Frequency analyses indicated that the 72 respondents who did not complete any survey items did not differ from the 222 respondents in terms of length of service ( p = .69), but were more likely to indicate a position of “other” than respondents who completed the survey (27.1% vs. 8.5%), x 2(5, N = 222) = 20.07, p < .01. Because of missing data on other variables, sample for analyses ranges from 173 to 222.

Of the 222 remaining respondents, 130 reported their position as case managers, 13 were field program specialists, 38 were supervisors, 16 were directors, 6 were administrators, and 19 were “other.” Because we were particularly interested in the actions of case mana-gers and because of the distribution of the variable, we re-classified position variable as case managers ( n = 130) vs. all others staff ( n = 92). Regarding job tenure, based on the distribution of responses, the tenure variable was grouped into three cate-gories: 0-5 years ( n = 64), 5-10 years (n = 65), and 10+ years ( n= 92).

Table 1 displays measures of familiarity and knowledge of SafeCare, overall and by position type and job tenure. On average, staff reported being slightly familiar with SafeCare ( M = 2.08 on a 4-point scale). Only about a third could correctly name any of the three focal areas of SafeCare, but most (83%) could identify the age group targe-ted by SafeCare (0-5). The mean number of modules correctly named was less than one out of three ( M = 0.74). When compared by position type, differences emerged between case managers and other staff: case managers reported lower familiarity with SafeCare, t(220) = 5.03, p < .01, named fewer modules, t(220) = 2.16, p = .03, and were less likely to correctly identify the age group SafeCare targets, x 2(1) = 6.45, p = .01.

Table 1 also depicts means and percentages for familiarity and knowledge by job tenure (overall means are repeated for sake of comparison across rows). There were no differences in these va-riables by job tenure.

Table 2 displays the three attitude measures – satisfaction with SafeCare, satisfaction with standard parenting, and concerns about SafeCare – overall, and by position and job tenure. Overall, attitudes were mildly favorable toward both SafeCare and standard parenting. There was no difference in SafeCare attitudes or standard parenting attitudes by position. Regarding concerns about SafeCare, non-case managers expressed greater concerns than case managers ( Ms = 2.42 vs. 2.16, p = .01). No differences emerged by job tenure.

Because parallel measures were used to create the satisfaction measures for SafeCare and standard parenting, we conducted ana-lyses to compare satisfaction with the two measures. A 2 3 2 repeated measured analysis of variance was conducted with Position (case manager vs. others) and Attitude Type (SafeCare vs. standard parenting) as the within subject factor. There was a significant effect of Attitude Type F (1, 137) = 3.90, p = .05, indicating that satisfaction with standard parenting was higher than satisfaction with SafeCare ( M s = 2.72 vs. 2.56). Because familiarity with SafeCare was overall low, we repeated this analysis using only the respondents who indicated a high degree of familiarity with SafeCare (those who responded were ‘somewhat’ or ‘very’ familiar with SafeCare, n = 69). Results of this analysis indicated no effects for Attitude Type or Position. Interestingly, the direction of the repor-ted satisfaction levels was reversed, such that satisfaction with SafeCare was nominally higher than satisfaction with standard parenting (2.96 vs. 2.88), but the Attitude Type effect was not significant ( p = .18).

Of the 222 respondents, 47 (21%) indicated it was not their job to make referrals (12 case managers, and 35 non-case managers) and 2 responses were missing. Of the remaining 173 respondents, 43 (24.8%) reported having made a referral for SafeCare. Case managers were significantly less likely to have made a referral (15.1%) than administrators/supervisors (46.3%), x 2(1) = 19.3, p < .01. No differences in referral rates were found by job tenure ( p = .66).

Barriers to SafeCare referrals are displayed in Table 3 , overall and by position and tenure. Based on factor analysis, two categories of barriers were created, (1) Lack of knowledge regarding SafeCare (7 items; e.g. “I have never heard of SafeCare,” “I don’t know enough about SafeCare,”) and (2) Lack of fit of SafeCare (8 items; e.g., “I prefer families to get parenting services other than SafeCare,” “The family needs other services like substance abuse counseling, domestic violence counseling, instead of SafeCare”). Regarding lack of know-ledge as a barrier, overall means indicated that lack of knowledge was a small to moderate barrier ( M = 2.31 on 4-point scale). However, there was a difference by position with case managers rating lack of knowledge as a greater barrier than those in other positions, t(185) = 4.27, p < .001. The second barrier – the fit of SafeCare – was rated as a small barrier overall ( M = 1.80), with no difference by position ( p =.18). No differences in barriers to referrals were found by job tenure (bottom on Table 3).

Logistic regression analyses were conducted to predict having made a referral. Predictors included position, job tenure, knowledge of SafeCare (# of modules correctly named), satisfaction with SafeCare, and the knowledge and fit barrier measures. In bivariate models, se-veral of the predictor variables were statistically related to having made a SafeCare referral ( Table 4 ). Being a Case Manager was related to fewer referrals, while familiarity with and knowledge of SafeCare and having more positive attitudes toward SafeCare were related to increased likelihood of referral. Endorsing lack of knowledge as a barrier to SafeCare referrals was related to fewer referrals. In the multivariate model, only two variables remained statistically significant: position and familiarity with SafeCare ( OR = 4.3). Case managers were 80% less likely to report making a referral than others ( OR = 0.20), and increased familiarity was associated with greater referrals ( OR = 4.3)

This paper examined the knowledge, awareness, attitudes, and referral practices and barriers of public child welfare workers in a state that had implemented the evidence-based parenting program, SafeCare. A large body of literature has demonstrated that implementation success is dependent upon a multiplicity of factors involving multiple individuals and systems ( Aarons, Hurlburt, & Horwitz, 2011 ). In systems such as child welfare that utilize public/private partnerships for assessing and serving clients, it is critical that both public and private systems be engaged in implementation. Public service workers typically assess and refer clients to private community-based organizations who provide the EBP services. If public service workers are unaware or lack knowledge of evidence-based programs, lack of referrals may result in poor uptake of the EBP. Very few studies have examined public child welfare workers knowledge, attitudes, and behaviors toward an EBP during an implementation.

There were several notable findings. First, overall there was a ge-neral lack of familiarity with and knowledge of the evidence-based program, SafeCare, and this was especially pronounced among the case managers, the front line staff, who reported the lowest levels of familiarity and knowledge about SafeCare. Second, attitudes about SafeCare were moderately favorable, but slightly less favorable than the standard parenting program. This latter finding appeared to be driven by those staff who were not familiar with SafeCare. Finally, a relatively low percentage of staff had made referrals for SafeCare, and this was especially true for the front line staff, case managers, who would be most likely to make specific referrals. The strongest predictors of referrals in multivariate analyses were position and familiarity with SafeCare.

It is unclear as to why case managers had less awareness and knowledge of SafeCare, but there are several possibilities. Case ma-nagers are entry level positions with high degrees of turnover ( Barak, Nissly, & Levin, 2001 ), and newer staff may have yet to learn about new program models such as SafeCare. In the current data, case ma-nagers had shorter tenure than other positions (e.g., 42% of case ma-nagers had fewer than five years’ experience vs. 10% of other staff). However, job tenure was not related to familiarity with and know-ledge of SafeCare. Another possibility pertains to how systems typically roll out an EBP. When large systems implement EBP, the impetus to do so typically begins with leadership ( Aarons et al., 2011 ), who consider what EBP to adopt, and how and when to roll those EBPs out. Front line staff may not be informed until the roll out is underway. If there are not concerted efforts to inform front line staff of new programs, they may remain unaware. In the current implementation, there was a concerted ongoing effort to inform the public child welfare staff of SafeCare, with regular email blasts, webinars, and even in-person visits from the implementation team and the state level program coordinator. However, the efforts for this implementation began at the top of the state hierarchy and it is unclear how much it trickled down to regional/county leadership and ultimately front line staff.

Related, it is noteworthy that the staff who were primarily in higher level positions (i.e., the non-case managers) including supervisors and directors, reported greater concerns about SafeCare than case managers. Leadership plays a central role in implementation models ( Aarons et al., 2011 ), and leadership variables have been demonstrated to affect staff attitudes and practices ( Aarons & Palinkas, 2007 ). Here, those in leadership positions expressed concerns about SafeCare (cost, process/paperwork), and this may ultimately affect referral patterns. It is noteworthy that non-case-managers, des-pite greater concerns about SafeCare, were still more likely to have made a referral. The data do not speak to why this might be, but managerial/administrative staff may experience more pressure to use the newer, evidence-based program than lower level staff.

The study has important implications for implementation. In systems with public/private partnerships, it is critical to educate the public workforce who may be assessing and referring clients, as well as training the private workforce who may be implementing the EBP. Public workers tend to have less positive attitudes toward EBP than private providers ( Aarons, Sommerfeld, & Walrath-Greene, 2009 ). Informational or educational efforts must be sustained over time. Staff turnover is high in child welfare systems ( Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009; Barak et al., 2001; Glisson, Dukes, & Green, 2006 ), and new staff must be educated on the various EBP available within their system. Implementation models that take a systems approach, such as those that use community development teams, or CDTs ( Saldana & Chamberlain, 2012 ) are one method of engaging both public and private systems when implementations of EBP are conducted. CDTs involve peer networks of system leaders, agency directors, and practitioners to implement a model and pro-blem-solve when barriers are encountered. As applied to the current data, CDTs could certainly assist with the dissemination of information on the EBP, and hopefully promote referrals. When CDTs are not an option, careful planning regarding training of referral sources is needed. Such training may need to be ongoing because of staff turnover. As systems add more EBP to their repertoire, staff training should include information on how to determine which services are appropriate for specific families.

In this particular implementation, the EBP was introduced without de-adoption/exnovation of the extant parenting program. Although this provides flexibility to referral agents, it may not be an ideal arrangement for promoting the use of the new EBP; when new practices are adopted without explicit de-adoption of the old practices, it may be especially difficult for new practices to succeed. This may be because of lack of familiarity with the newer model, fear of change, inertia, or commitment to the status quo ( Fixsen et al., 2005 ). Successful implementation likely involves replacing a non-evidence based program with an EBP rather than utilizing programs side-by-side and allowing them to compete for referrals. The de-adoption of non-EBP can send a clear message from an organization's leadership of the importance of adopting the new EBP, a variable which affects staff attitudes about EBP ( Aarons, Sommerfeld, & Walrath-Greene, 2009 ). The role of de-adoption or exnovation efforts in the success of an implementation has not been well-studied.

The study has several weaknesses that should be noted. First, the survey method involved a brief, one-time, cross sectional survey. There was no specific information on the sample frame (e.g., numbers of staff for certain positions), and thus the response rate and representativeness of the sample is unknown. The nature of the pro-ject dictated that a limited number of questions would be included, and as a result, limited questions on demographics, work history, and other important constructs were included, and no standardized questionnaires were included. Because data were collected in a cross sectional manner, the temporal and causal relationship between variables cannot be known. For example, it is not known if more favorable attitudes toward SafeCare would lead to greater SafeCare referrals, or that referrals (and thus experience with SafeCare) would lead to more favorable attitudes.

Implementation of EBP in large service systems is a challenging but critically important endeavor. It has become clear that implementation involves a complex set of processes that involve many individuals at various points within a system to ensure utilization and uptake of an EBP. Simply training practitioners to conduct the EBP is insufficient for uptake, as has been demonstrated in a number of studies ( Kolko et al., 2012; Whitaker et al., 2012 ). Efforts must be made to engage system leaders, referral agents, and other members of practice team to ensure effective utilization of an EBP.

The authors of this article declare no conflict of interest.

Parts of this research were supported by a contract with the state in which the research was carried out.