Quality of implementation in an evidence-based family prevention program: "The Family Competence Program"

The 154 families were over a period of 24 months, using the longitudinal research study conducted since the initial application of the Spanish SFP sessions. The sample used to gather information on family skill outcomes comprised 154 participant families, all in a risk situation. A global analysis was conducted, together with a separate analysis for each agency running the program: Proyecto Hombre (N = 50) and the Primary Care Social Services ( N = 104).

A Deplhi technique was undertaken, with the participation of 16 experts, 8 academics and 8 facilitators, the latter experienced in FCP applications ( Orte et al., 2014 ). The Delphi document included both a text framing the debate about the role of facilitators and a set of questions. There were two groups of questions (about the trainer profiles and on the methodology for trainer evaluation). The first group of questions included a list of items that the experts had to prioritize about facilitator profile (experience and training), facilitator skills, program adherence (fidelity to the contents of the program), group dynamic and family profiles (engagement of the families and their attitudes and comprehension). The second group of questions dealt with the issue of what should be prioritized when evaluating a facilitator and what kind of evaluation techniques to use. Each group of questions ended with an open question in which the expert could raise important issues that had not yet been mentioned, make observations or include relevant commentary.

Six of the program's 14 key sessions were analyzed. These were observed by external assessors, using a detailed checklist of the family engagement techniques that were used, their application, and the participants’ response to them. Fidelity tests of the six sessions under consideration were deemed to be very important since these sessions were thought to play a key role in the process undertaken during the program.

All these sessions entailed talks, discussions and activities, plus strict checks by the external assessors of the quality of their performance and the obtained responses.

The techniques used in the SFP can be grouped into three main types:

• 1.

  Talk-related techniques, involving the presentation of key aspects of the program. They should not be confused with lectures, but regarded as brief presentations, tailored to the participants’ level of understanding. They are presentations aimed at providing basic information needed for the different processes put into practice with the SFP.

• 2.

  Discussions. Exchanges of ideas, based on the contents of the program and the participants’ experiences. The aim is to foster an emotional experience and to engage the participants. Unless the program is personally embraced by them and opinions exchanged during discussions, the content matter will not be taken on board by the participants.

• 3.

  Group dynamics. Joint activities aimed at direct experimentation with the contents of the sessions.

Discussions and activities were held throughout all the sessions, since they are considered to be a basic factor in engagement. As for the talks, these played a more predominant role during the early sessions (to clarify concepts and set goals), although some kind of talk was included in all the sessions that were assessed in order to explain tasks to be completed at home, to sum up, explain or discuss information or as an invitation to continue applying certain aspects at home. In other words, during the sessions, the number of talks gradually tapered off while the presence of discussions and activities increased, since the families had started to acquire skills and know how during the program.

In total, information was gathered for 29 full applications of the program conducted at Proyecto Hombre and 11 full ones held by the Primary Care Social Services. Given the size of the applications, we were able to work with a very broad sample of sessions (720), 198 sessions from Proyecto Hombre and 522 from Social Services. Data for each and every one of the sessions were not available, since in some cases assessments could not be made. Despite this, accurate data was gathered for 700 sessions. That is, data was only missing for 20 sessions (2.8%).

External observers were used to assess the fidelity of each of the 6 key sessions with the planned intervention. To do this, systematic observational record sheets were completed, making detailed checks about the performance, participations and understanding of the talks, debates and activities.

To assess family skills, in addition to the checklist, evaluations of family outcomes were used, based on instruments validated for the Spanish population: BASC ( Reynolds & Kamphaus, 2004) and Kumpfer (Kumpfer, 1998 ) questionnaires validated for the Spanish population, with one version for the parents and another for the children. An Aggregate Family Skills Index was created based on ten indicators with information about all the families. Factor 3 of the factors relating to parents (“Family Cohesion”) was not taken into consideration, given that it is also included in the factors relating to children. Considered factors related to parents are: family resilience, relations between parents and children, family organization, positive parenting, parental skills (source: Kumpfer questionnaire for parents; Kumpfer, 1998 ). Considered factors related to children are: family engagement, family cohesion, control for problems at school, social skills, capacity to set limits (source: Kumpfer questionnaire for children; Kumpfer, 1998).

The Aggregate Family Skills Index ranges from 0 to 500 points, calculated according to the marks awarded for the ten indicators under consideration, processed using the following relative weightings: factors relating to parents, by 50%; factors relating to children by 50%. Positive skills were added up and so the higher the mark, the higher the family skills. The index is positively interpreted.

According to the descriptive data for the index for 2012–2013, better skills can be observed among Proyecto Hombre families (with a mean value of 362.48) than Social Services families (with a mean value of 334.00). The differences are not statistically significant.

Tables 1–4 show the prioritization mean score from the experts’ panel, as well as that of the facilitators and that of the academics; the order of the indicators evaluated according to the overall mean is also shown. Statistics are included ( N, chi-square, Kendall's W , degrees of freedom and statistical significance degrees) from the overall expert group as well as from the two separate groups (academics and facilitators).

The first indicator refers to facilitator experience and preparation. Program training (1.44) as well as family intervention experience (1.88) were two of the items to a total of 5 that facilitators and academics highlighted in the two rounds rated as the most important aspects in this dimension, as shows in Table 1. In this case, Kendall's W was higher in the facilitators’ group (.83) than for the academics (.63), being at a significant trust level ( p = .00) in the two groups.

With regards to the facilitators’ skills (Table 2 ), there was agreement on which three skills were the most important from a total of 5: communication (1.56), empathy (2.19) and group management (3.13) were the most valued skills, both by academics and experts. With significance for the three groups ( p = .00), academic and facilitator groups obtained similar scores for Kendall's W (.69 for academics and .64 for facilitators).

According to participants, the two most important issues from a total of 10 issues about aspects that impact negatively on the adherence to the program were (Table 3 ): lack of session preparation on the facilitator's part (1.13), and poor time management (2.31). With significance for the three groups ( p = .00), Kendall's coefficient of concordance was for the academics of .92 while there was more dispersion in the facilitators’ answers (.63 for Kendall's W coefficient).

Finally, recognizing the possibility of family change after program completion was valued as the prime indicator for the group dynamic and the family profile by both groups (1.69), followed by family attitude and willingness during the course of the session (2.19) ( Table 4 ). Significance trust level amongst the three groups was significant ( p = .00). It should be noted that the first item (‘recognizing the possibility of family change via the program’) received the highest scoring from the facilitators (1.38), especially if compared with the academics (2.0).

Each of the hypotheses was tested using the aggregate data for the sessions for each of the organizations taking part.

The first hypothesis was confirmed as there was no significant difference in the assessed quality of the family engagement techniques used in the applications of the Spanish version of the SFP conducted at Proyecto Hombre (11 applications) and at the Primary Care Social Services (29). As ordinal measures on a three-point scale had been chosen for the assessment process, non-parametric statistics were used. For comparisons between the organizations, calculations were made using the Mann–Whitney U test. Statistics are in Tables 5 and 6 . The obtained data confirmed that there were no significant differences in the quality of the applications conducted at Proyecto Hombre and at the Social Services confirming the Hypothesis 1 for the evaluation of family engagement techniques.

As for the second hypothesis in the assessment of family techniques, it was confirmed that more detailed, comprehensive talks, discussions and group activities lead to better family engagement outcomes (understanding and participation), although the results did not always coincide with the expected model. Thus, on some occasions, better outcomes were observed when the talks or discussions were not exhaustive but they had a level 2 of detail. That was, when “quite a few aspects of the subject were discussed”.

Chi-square tests were conducted to test for association patterns between high marks for techniques performed and high marks for observed outcomes showing significant results ( Table 7).

Lastly, the third hypothesis of a positive relationship between better family engagement outcomes (understanding and participation) and better long-term family skill outcomes was tested. For this purpose, correlations were calculated, based on Spearman's Rho, given the ordinal measures used in the assessments. As explained previously, the results were assessed using the Aggregate Family Skills Index, taking the mean value for the 154 families evaluated in the longitudinal study and participating in the 40 applications under analysis (11 conducted at Proyecto Hombre and 29 at the Social Services). Table 8 shows the correlations between positive family engagement outcomes and long-term family skill outcomes. The statistics for the correlations are shown separately for each organization running the program.

As Table 8 shows, all the techniques were correlated with better long-term family skill outcomes. That is, better family engagement dynamics during the sessions implied better family skill outcomes. Having said that, the most effective techniques in contributing to the long-term maintenance of family skills were the activities held at both organizations and the discussions held at the Social Services.