The authors of this article declare no conflict of interest.
A description is made of the quality of Spanish family support programmes, based on their impact, dissemination, scaling up in communities, and sustainability; 57 implemented programmes with informed evidence were selected by EurofamNet. Most of the programmes were shown to make a positive impact, using quantitative methodologies, and they were manualized, while about half of them defined the core contents and included professional training. From a cluster analysis of programmes with scaling up, those with a high and moderate level of systematization were identified, based on the existence of defined core contents, implementation conditions, institutional support, professional training, and reports of findings. The highly systematized programmes were characterized by a greater use of mixed methodologies, their scientific dissemination through different means, and their inclusion in services. A programme quality analysis is proposed, taking an integrated approach that relates the programme’s impact with its design, implementation, and evaluation of sustainability.
En este trabajo se presenta una descripción de la calidad de los programas españoles de apoyo a las familias, basándose en su impacto, difusión, diseminación institucional y sostenibilidad. En el marco de EurofamNet se seleccionaron 57 programas implementados con evidencia fundamentada. La mayoría de los programas mostraron un impacto positivo utilizando metodologías cuantitativas y estaban manualizados, mientras que cerca de la mitad de ellos definían los contenidos clave e incluían la formación de los profesionales. A partir de un análisis de conglomerados se identificaron los que tenían un nivel de sistematización alto y moderado, definidos los contenidos clave y las condiciones de implementación, apoyo institucional, formación profesional e informes de resultados. Los programas con alto nivel de sistematización se caracterizaron por un mayor uso de metodologías mixtas, su difusión científica a través de diferentes medios y su inclusión en las instituciones. Se propone un análisis de la calidad de los programas, con un enfoque integrado que relacione el impacto del programa con su diseño, implementación y la evaluación de la sostenibilidad.
The United Nations Convention on the Rights of the Child acknowledges the right of all children and adolescents to have their developmental and educational needs met in order to ensure their proper development, and suggests the preservation and improvement of family environments as a fundamental factor in achieving this goal (
Lots of structured family interventions can currently be found that share the same goal of promoting the well-being of children, adolescents, and their families, but there are differences in the degree to which they meet the quality standards of evidence-based practice (
Programme quality is reported in terms of the impact they make, among other aspects. An impact evaluation measures changes directly attributable to an intervention. It is considered to be one of the gold standards of interventions and not only does it serve to ascertain the extent to which goals have been achieved, but it also makes it possible to verify and improve the quality, efficiency, and effectiveness of interventions. Hence, impact evidence is a strong basis for continuing, diversifying and extending assessed interventions, and a good pretext for reconsidering the budget at the disposal of the associated service and for facilitating the said intervention’s dissemination and sustainability (
Impact outcomes should be assessed in an evaluation process that is rigorous, useful, feasible, adequate, and responsibly conducted (
There is also evidence that the impact of an intervention might tend to be moderated or mediated by the characteristics of the target population (e.g.,
Despite the high number of published results on the effectiveness of interventions, several review studies question the quality of the evidence. In the majority of cases, following a rigorous analysis of the risk of a methodological bias, the results are referred to as being inconclusive (e.g.,
Once the effectiveness of an intervention has been demonstrated through strong evidence, from which firm conclusions can be drawn on what works and under what conditions it can best be given, it is important to make sure that it can be easily implemented, disseminated, and evaluated with fidelity in different environments (
One of the objectives of evidence-based practice is to extend the use of these interventions in an appropriate way so that a high number of families can benefit from them. To be able to scale up an intervention in communities and services, an awareness of a wide range of factors that might influence the quality and sustainability of implementations is needed, in addition to the impact it makes. It depends on a combination of the characteristics of the implementation, the organization in charge of running the intervention, and any external support (social, economic and political) involved in the scaling up process (
With the exception of the general characteristics of implementations, which are widely described in review studies (e.g.,
As for the characteristics of implementations, before an intervention is scaled up it must comply with standards relating to materials, training, and technical support (
To guarantee the effectiveness of a new application, it is important to define the core programme components (i.e., components deemed to be essential in the achievement of envisaged outcomes) not subject to modification or elimination (
In this respect,
To scale up an intervention in communities and services, it is important to have an organized training programme for the professionals who run it (
According to
In short, the sustainability of an intervention is not only dependent on impact-related outcomes, but on other additional factors that play an important role in the scaling up process in communities and services. Only in situations in which these factors are taken into account is it recommendable to contemplate the final quality criterion of evidence-based programmes: their incorporation by services (
Recent findings on the quality of family support programs show implications related to (1) programme impacts, in terms of the achievement of envisaged goals, using RCT designs as a quality criterion; (2) the relationship between the impact and the characteristics of the population and the implementation conditions; and (3), exceptionally, information on some instances of scaling up in communities through a manual or core contents, relating this with a characteristic of the programme. However, no studies were found that made a comprehensive holistic analysis of how the characteristics of programmes might be jointly interrelated. This “integrated analysis of interrelations in the quality criteria of programmes” is considered to be one of the current challenges in the evaluation of programmes, and would provide a better insight into the impacts of family support programmes and related evidence. This study aims to contribute to the achievement of this goal, thanks to efforts by EurofamNet in compiling information on the design, implementation, evaluation, impact, and dissemination of family support programmes currently being implemented in Spain and on their scaling up in communities and services (
Theory-based documentary research was conducted, compiling data on positive parenting and/or family support programmes in Spain. Purposive non-probabilistic sampling methods were used so that a sample could be chosen to meet the objective of the study. To assess the results, a quantitative data analysis was conducted.
The sample was made up of 57 evidence-informed family support programmes conducted in Spain. Family support programmes aimed at tackling numerous different issues were selected, such as substance use or behaviour disorders, among others. More specifically, 45.60% were universal interventions, 61.40% were selective interventions, and 29.80% were indicated ones. They were targeted at families with children in different developmental stages, depending on the objective of the programme: 22 were aimed at early childhood, 32 were directed at mid childhood, 46 were focused on pre-adolescents, and 33 were for adolescents. As for the operating domain, most of the interventions could be applied to families (87.7% of the programmes) and/or be directed at parents (71.1% of them). Interventions that could be applied in healthcare scenarios (45.6% of the programmes) or in schools (38.5%) also predominated. All the programmes were run on a face-to-face basis (see
The inclusion criteria for the sample selection process were as follows: a) family support programmes of over 3 sessions, b) a defined theoretical basis, and c) programmes with available results. The exclusion criteria were: a) programmes with target population being adults unrelated to parenthood and family issues, b) programmes with an undefined methodology, contents, and/or structure, and c) a failure to identify the body or institution running the programme. Whether the programme was an original one by the authors or an adaptation of an existing one was also taken into account.
To collect the data, a Formative Evaluation Template (FET) was drawn up, based on the quality standards of evidence-based programmes and a consensus reached by a group of experts on the evaluation of EurofamNet programmes. The consensus on the items included in the FET was based on the following criteria: validity, precision, reliability, confidence, and coherence.
The FET was made up of a total of 43 items, which incorporated four informative categories: a) a description of the programme, b) the implementation conditions, c) its evaluation, and d) the impact, dissemination, and sustainability of the programme. Specifically, this study focused on the last category (quality of the programs in terms of their impact, and dissemination and sustainability of the programmes), which includes these items: 1) type of analysis used to explore changes brought about by the programme, 2) obtained results, 3) type of additional explored effects, 4) scientific dissemination, 5) the programmes’ integration in available services and in the community, and 6) sustainability.
The answers to the items on the impact, dissemination, and sustainability of the programme took a multiple-choice format. Each of the items was accompanied by an explanation to clarify the purpose of each question (
The informed-evidence programmes were identified and their data was recorded by members of the Spanish Family Support Network. This procedure was chosen since it contributed to two goals: first of all, the members were key informants (that is, they were experts with in-depth knowledge of one or more of the programmes and could provide exhaustive information on their characteristics, functioning, and implementation) and, secondly, since the network is made up of national, regional, and local bodies, access was also ensured to programmes conducted at a local level. Programmes were identified and compiled from May 2020 to April 2021.
Prior to gathering the data, the members of the network were given 5 hours of online training. This was aimed at providing instruction on the quality standards of the programmes and on the data compilation and recording process so as to boost their understanding and the efficiency and accuracy of operations. To guarantee the quality of the data-gathering process, they were supervised and guided by coordinators. Likewise, a computer system was developed to assist in the compilation of the data so that the information could be unified in a general database. The data was also stored on the intranet of EurofamNet’s website.
To tackle the analysis of univariate and bivariate data, frequencies and percentages were used, together with Yates’ chi-squared test to observe the significance of the frequency distribution of the 2 x 2 tables, corrected standardized residuals to facilitate the interpretation of the data, and the phi coefficient to obtain the effect size. Hence, a two-step cluster analysis was conducted. The goodness of the cluster structure is considered to be weak, satisfactory, or strong, depending on the quality coefficient or silhouette coefficient (
All the experts who participated in the study took part voluntarily after signing an informed consent form in accordance with the Declaration of Helsinki. This study was carried out in accordance with the European Cooperation in Science and Technology Association policy on inclusiveness and excellence, as written in the CA18123 project Memorandum of Understanding (
To achieve this first goal, a descriptive analysis was made of the type of analysis used to examine changes brought about by the programme, the results of evaluations of the programmes, the type of additional effects that were explored, the programmes’ scientific dissemination, scaling up in communities and services, and sustainability.
Firstly, from an examination of the type of analysis that was chosen it was observed that, in the case of quantitative parametric analyses, on 89.50% (
In terms of the achieved results of the programmes, 80.70% (
As for the additional effects that were explored, there was some variability when the effects of the intensity of the interventions were examined, since 21 of the programmes (36.80%) that reported this kind of data had conducted a relevant analysis, whereas another 21 had not (36.80%). When it came to analyses of the drop-out rate, this information was reported in 47.70% (
In relation to dissemination, conference proceedings were the most common way of reporting the results of their evaluation (66.70%,
Finally, in analyses of scaling-up in communities and services, 82.50% (
To meet the second goal, a two-step cluster analysis was conducted. For this purpose, the following variables were chosen: manualization, definition of core components, specification of implementation conditions, organizational support, professional training, and an explanation of the evaluation process. In accordance with the analysed data, two clusters were formed, with a “satisfactory” quality level, close to “strong”, as shown in the silhouette analysis.
The results point to the existence of a relationship between the clusters and the programmes’ impact, dissemination and sustainability. Firstly, as
The results also point to a significant relationship between the clusters and the use of books, chapters of books or conference proceedings for disclosure purposes, with a medium effect size, along with a trend effect in the use of journals and grey literature, with a low effect size. In all cases, greater use was observed when the programmes were characterized by a “high level of systematization”. Lastly, a trend effect between the clusters and the sustainability of the programmes was identified, more specifically, in the case of the programmes’ integration in available services, with a medium effect size. This was more common among highly systematized programmes (see
The purpose of this paper was to review the quality of family support programmes in Spain according to impact and scaling-up. Not only does this encompass the results of evaluations, but also aspects relating to the design of the implementation (manualization, core contents, and implementation conditions), the background context (professional training and institutional support), other evaluation-related aspects (in addition to outcomes), the dissemination of the programme, and its incorporation in services (scaling up).
In the case of our first objective (to describe the quality of Spanish parenting programmes), quantitative parametric analyses are used by most of the programmes, while a large number of them report data through descriptive analyses and some use multivariate analyses. Over half the programmes also analyse the data from a qualitative perspective. Systematic reviews of programmes focus on RCT quantitative analyses (e.g.,
One weakness worth highlighting is the fact that training professionals is provided in under half the cases. This training is important and it should be aimed at equipping professionals with the necessary skills to guarantee a programme’s effective application with fidelity to the core contents (
As for the strengths, mention must be made of the high number of programmes with a manual and specified core contents. This high level of manualization can be tied in with positive impacts, since the presence of a manual is an important factor in successful programmes (
To achieve the second objective, the programmes’ level of systematization is described, based on different quality standards associated with the programmes’ core contents, implementation conditions, professional training, and institutional support. Relatively low information had been reported on these standards, as stated in, e.g.,
Firstly, the “high systematization” cluster contains a higher percentage of programmes with defined core contents. This is a key element of a programme, determining aspects that are fundamental in the achievement of positive outcomes (
No differences were found in the two clusters’ level of manualization, although a quality analysis of the manuals is needed, following the approach suggested by Carroll and Rousanville (2008). Manualization is a basic ingredient: the presence of a manual ensures that important aspects of an intervention are described (such as its underlying theory, theory of change, methodology, and thematic contents) (
With the exception of the high level of manualization found in both clusters, among the analysed Spanish family support programmes, special note must be made of the two different levels of systematization and the two rates of compliance with quality standards. This made it possible to identify the key aspects that must be improved upon in programmes in the “moderate systematization” cluster, aspects which are all essential in improving the quality of impacts.
To overcome the limitations of recent research, this study strove to contextualize the impacts by taking an integrated approach to the quality of programmes. As a result, the third objective was to find interrelations between the family support programmes’ level of systematization and salient characteristics of their impact, dissemination, and sustainability. Exploring links between the level of systematization and the above characteristics is important, since evidence of an impact is the basis for continuing and disseminating an application and for promoting its sustainability (
As for the programmes’ dissemination, the results point to wider use of publications by the highly systematized cluster. This cluster coincides with the recent international trend toward greater scientific literature on parenting programmes, according to
As for sustainability, the highly systematized cluster featured more programmes that were incorporated in services, with a trend effect being identified between the clusters and the scaling-up. According to
Because very few systematic reviews take a comprehensive integrated approach to the impacts of parenting programmes, one limitation of this paper is the fact that it is harder to tie in the impact-related characteristics of Spanish programmes with the state of the art in international research on the subject. As mentioned earlier, the international systematic reviews that were found are mainly focused on RCTs. In the case of Spanish programmes, there are few RCTs. To overcome this situation, as mentioned earlier, a plural methodological approach of the assessment of these interventions could contribute with rigorous evaluations. Another limitation of the study is that analyzed programs could not be representative of the whole of existing programmes in Spain, due to the fact that they were identified through academic experts.
With regard to the implications of this study for public policies, the programmes’ low level of integration in community services must be highlighted even though they demonstrate that they had impact. It is important to make policy-makers and technical experts more aware of the importance of incorporating evidence-based programmes, particularly more highly systematized ones that make a positive impact. This is also a way of guaranteeing their sustainability. As for the practical implications for professionals from the social services, healthcare, and educational sectors, the study confirms the level of systematization and impact of the parenting programmes that these professionals run in Spain. It is vital for these professionals to give priority to training in programmes with positive impacts.
In terms of future lines of research, a systematic review of international parenting programmes must be made that goes further than just reporting on outcome-based impacts. Secondly, a more specific review must also be made of scaling-up or integration in community services. Thirdly, a guide to best practices should be drawn up so as to foster the adoption and integration of Spanish programmes with positive impacts in community services, together with recommendations for improving the dissemination of the impact-related outcomes of these interventions.
In conjunction, this paper proposes an integrated means of analysing the quality and level of systematization of programmes. To study the impacts of programmes, it takes into account possible interrelations among key aspects of the design, implementation, and evaluation process, aimed at gaining a better insight into parenting programmes. If we break down the different parts of the model, this integrated analysis highlights the need for: i) appropriate manualization and special attention to core components as basic ingredients in the design of a programme; ii) the careful alignment of the objectives, programme design, and evaluation design; iii) an evaluation design and outcome evaluation that not only incorporate RCTS, but also rigorous quantitative, qualitative, and in particular, mixed methodologies; iv) implementations that make mothers and fathers the focal point of the intervenion; and v) implementations given by trained professionals, with appropriate institutional support. All these factors are interrelated and they determine the impact that a programme makes. In turn, this impact should also be measured in terms of its relations with scaling-up in communities and services and with the dissemination of outcomes.
Cite this article as: Orte, C., Pérez-Padilla, J., Maya, J., Sánchez-Prieto, L., Amer, J., Baena, S., & Lorence, B. (2022). An integrated analysis of the impact of Spanish family support programmes with informed evidence.
Funding: This article is based upon work from COST Action CA18123 European Family Support Network, supported by COST (European Cooperation in Science and Technology)