An illustration of how program implementers can use population-specific analyses to facilitate the selection of evidence-based home visiting programs

Providing effective home visiting (HV) interventions to at-risk pregnant women and at-risk families with young children is a public health priority. The Patient Protection and Affordable Care Act established the Maternal, Infant, and Early Childhood Home Visiting Program (MIECHV) to provide $1.5 billion dollars to evidence-based HV programs (US Department of Health & Human Services, 2013). In order to determine what programs are effective, in 2009, the U.S. Department of Health and Human Services (DHHS) contracted with Mathematica Policy Research to conduct the Home Visiting Evidence of Effectiveness (HomVEE) review. Mathematical Policy Research considered the following eight maternal and child outcomes: (1) child health, (2) child development and school readiness, (3) reductions in juvenile delinquency, family violence and crime, (4) maternal health, (5) positive parenting practices, (6) reductions in child maltreatment, (7) family economic self-sufficiency, and (8) linkage and referrals to community resources and supports. In order for a program to meet criteria for evidence of effectiveness, either (a) at least one high- or moderate-quality study had to have found that the program had favorable, statistically significant impacts in two or more of the eight outcome domains or (b) at least two high- or moderate-quality studies with non-overlapping samples had to have found that the program had one or more favorable, statistically significant impact in the same outcome domain. Thus, depending on how the original evaluation studies were conducted, a program was considered effective if it had demonstrated favorable, statistically significant impacts on either one or two of eight specified maternal and child outcomes. As such, different HV programs were deemed effective based on their demonstrated impact on a variety of maternal and child health outcomes.

State administered MIECHV programs typically target high-need communities based on a profile of socio-demographic risk factors. Likewise, individual HV programs tend to families for services based on socio-demographic risk factors, such as young maternal age, single parent families, and low family income. A public health model focusing on preventing the development of adverse outcomes would argue for an understanding of the statistical relationships between prevalent adverse outcomes and the risk factors predictive of those outcomes in the population of interest. It cannot be assumed, however, that every individual risk factor (e.g., young maternal age, single mothers, or low family income) used to target families for HV services is necessarily predictive of all of the outcomes specified by DHHS as criteria for designating a HV program as evidence-based. Indeed, some of the commonly used socio-demographic risk factors used for targeting families for such programs, such as teenage childbearing, have been found to poorly predict future child outcomes, compared to other individual risk factors or combinations of risk factors (Chittleborough, Searle, Smithers, Brinkman, & Lynch, 2016). Also, it may be useful to consider that the total number of risk factors present in a family's life may be an important predictor of some outcomes. Finally, it cannot be assumed that risk factor-outcome associations reported in the literature from observational studies of a given population at a given time will generalize to other populations and times.

In this paper we propose that prior to implementing a HV intervention in a target community, as part of a needs assessment or feasibility study, program planners and implementers should use population-specific data to identify maternal and child health outcomes that are associated with risk factors in that community. Although to illustrate how this can be done, we used data from the control group of a randomized intervention, data can come from other less resource intense sources, such as retrospective surveys or administrative data (e.g., hospital and school records). Then, based on the results of the analyses, specific evidence-based HV programs should be ed for families in that community. This decision-making should take into ac the specific outcomes different HV programs on the approved MIECHV list have been shown to impact in evaluation studies. To illustrate how results from population-specific analyses could be used to HV programs for specific communities, we conducted population-specific analyses using data collected in a rural community in upstate New York (i.e., Elmira, New York).

Our illustrations assumes that program implementers have access to either (a) a well-designed observational study (e.g. representative community survey with good response rates) using well-measured constructs or (b) administrative data with good coverage of the population to be served by the HV program. In our illustration, we use statistical approaches likely to be familiar to local program planners and evaluators (rather than “cutting edge” approaches typically favored by university-based researchers).

Bivariate associations (e.g., correlations or t-tests) are a simple and common way to assess associations between individual risk factors and individual outcomes. However, there are major limitations to bivariate analyses. First&##44; they cannot definitively assess whether a risk factor other than the one of interest may be responsible for an observed association. For example, young maternal age may correlate with a child health outcome. But it may be that the reason why this association is observed is because women who have children at younger age are more likely to be poor, with poverty as the real “driver” of this association. So, in this case, targeting poor women may be a more effective strategy than ing communities or families based on maternal age. Second, bivariate associations cannot evaluate the potential impact of the number of risk factors present. Given the high rate of co-occurrence of risk factors in families, it may be that the number of maternal or family risk factors is more predictive of an adverse outcome than any specific risk factor.

Multivariate regression models allow for the examination of the predictive power of each risk factor independent of other measured risk factors. For example, the association between maternal age and child health could be assessed while acing for poverty status. There are also limitations of multivariate regression. For example, if risk factors are highly correlated, multivariate regression models may yield biased parameter estimates and reduced statistical significance for individual risk factors (Cohen, Cohen, West, & Aiken, 2003; Myers & Wells, 2003).

These models predict outcomes from a of the number of risk factors present and yield an estimate of the overall effects of multiple interrelated risk factors in a simple and comprehensive way without requiring large sample sizes to have adequate statistical power (Evans, Li, & Whipple, 2013; Lee & Harris, 2009). However, there are limitations to these models, including the assumption that all d risk factors are of equal importance (Lee & Harris, 2009). Additionally, for simple s, they dichotomizing outcomes to a present/absent status, which results in a loss of information regarding the severity of each risk factor and reduced statistical power (Cohen, 1983; Royston, Altman, & Sauerbrei, 2006).

Given that there are strengths and weaknesses to bivariate, multivariate, and cumulative risk analyses, we recommend that program implementers use all three methods and compare results from the methods to evaluate population-specific associations between risk factors and targeted outcomes.

Given that DHHS stipulated that HV programs must have shown impacts on only one or two (of eight) outcomes, it is important to conduct analyses to assess associations between risk factors and individual outcomes. However, some HV programs impact many of the outcome domains targeted by DHHS (US Department of Health & Human Services, 2013), and many states and communities are interested in improving several maternal and child health outcomes with their HV programs, not just one or two. As such, it would also be helpful to assess associations of a risk factor with several outcomes simultaneously since community-level and family-level risk factors might be chosen that have the most broad-based impact on desired outcomes, and programs might be ed based on their impact on multiple outcomes. In this case, the focus can shift from individual outcome measures to composite outcome scores.

There are also some statistical advantages to composite outcome scores. When assessing associations between several risk factors and several outcomes, it is possible to find statistically significant associations simply by chance. Statistical adjustments can correct for these potential errors, such as the Bonferroni adjustment; however, these adjustments tend to yield overly conservative estimates, especially when outcomes are highly correlated (Tyler, Normand, & Horton, 2011; Yoon et al., 2011). Combining outcomes into a composite measure addresses problems associated with multiple testing without requiring statistical adjustments (Freemantle, Wood, Eastaugh, & Calvert, 2003). However, it is important to note that there are also limitations to composite measures. They are extremely sensitive to missing data (Yoon et al., 2011), especially if a complete case analysis is adopted (Teixeira-Pinto, Siddique, Gibbons, & Normand, 2009). Also, it may be incorrect to assume that all outcomes making up the composite score are of equal importance (Drake et al., 2013; Prochaska, Velicer, Nigg, & Prochaska, 2008), or that a significant association between a risk factor and a composite score means that the risk factor equally affects all the outcomes that make up the composite (Freemantle et al., 2003).

Given that there are strengths and weaknesses to using individual outcomes and strengths and weaknesses to using composite outcome scores, we recommend that program implementers assess associations between risk factors and individual outcomes, as well as associations between risk factors and composite outcome scores.

The purpose of the present study was to conduct a set of analyses (i.e., bivariate, multivariate, and cumulative risk analyses) assessing risk-outcome associations on a community sample of at-risk families to illustrate how results from population-specific analyses can facilitate the ion of evidence-based HV interventions that are particularly likely to benefit families in a target community.

Participants were from the control group of the Nurse Family Partnership (NFP) program randomized clinical trial conducted in and around Elmira, a small semi-rural town in upstate New York. Between 1978 and 1980, pregnant women who had no previous live births were recruited from health clinics and doctors’ offices. Five hundred eligible women were invited to participate and 400 enrolled. The 400 women who enrolled were randomly assigned to the intervention group or the control group. Women completed informed consent at baseline interviews. The institutional review board at Cornell University in Ithaca, New York ed approval. All procedures were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. See previous publications for additional details about the original Elmira NFP study (e.g., Olds et al., 1997, 1998). Given that the intervention had several effects on maternal and child outcomes (Olds et al., 1997, 1998) and that the purpose of the present investigation was to examine associations of risk factors with maternal and child outcomes, rather than assess intervention effects, analyses were limited to data from 141 control group families.

Prior to randomization women completed informed consent and baseline intake assessments. At a 15-year follow-up, both women and children completed assessments. See earlier publications for details about intake assessments (Olds, Henderson, Chamberlin, & Tatelbaum, 1986; Olds, Henderson, & Kitzman, 1994; Olds, Henderson, Tatelbaum&##44; & Chamberlin, 1988) and follow-up assessments (Olds et al., 1997, 1998).

During the intake interviews women reported socio-demographic information, including age, highest grade attended, and marital status. Women also completed the Hollingshead Four-Factor index to estimate household SES (Hollingshead, 2011).

We computed a cumulative risk score by dichotomizing each risk factor into a 0/1 variable (maternal age under 18 years=1, less than a high school education=1, unmarried=1, below the mean Hollingshead score=1) and summing all dichotomized risk variables.

At the follow-up assessment women completed the Child Behavior Checklist, which is a measure that assesses child behavior problems occurring in the last 12 months and s an anxiety/depression subscale (Achenbach & Edelbrock, 1981; Ivanova et al., 2007).

We ed seventh through ninth grade English, Science, Math, and Social Studies letter grades to a five-point numeric scale and averaged them (range=0–4, with higher values indicating higher grades).

We obtained information on the total number of times a child received a school suspension before the 15-year assessment.

To a total early onset behavior problem score, we summed dichotomized early onset behavior problems variables. Early onset behavior problems d smoking cigarettes before age 12; having sexual intercourse before age 13; smoking marijuana before age 14 and using inhalants, hallucinogens, cocaine, or heroin; having a person in need of supervision report filed; being arrested; being convicted of a crime, and binge drinking before age 15. Cutoff ages were based on national data or frequency distributions of the behaviors in the NFP control group sample. See Eckenrode et al. (2001) for additional details on the early onset behavior problems measure.

Composite poor child outcome scores were computed by reverse coding academic achievement, standardizing all child outcome variables, and then summing the standardized child outcome variables.

At the 15-year follow-up, women completed the Mental Health Inventory, which yields an overall mental health score reflecting anxiety, depression, behavior and emotional control, general positive effect, and emotional ties (Veit & Ware, 1983). A low score is indicative of poor mental health.

At the 15-year follow-up, women completed an adapted questionnaire from the National Comorbidity Survey (Kessler, 1995) to assess the impact of substance use on the following life domains: missing work, experiencing trouble at work, having a motor vehicle crash or traffic violation, compromising child care, and receiving treatment. We used an overall score which summarized impact across all life domains assessed.

After receiving consent from the women, research staff obtained information from state Child Protective Service records on the total number of maltreatments reports involving the women and children up to the children's 15th birthdays.

A family instability measure was based on maternal reports of number of intimate partners, work hour changes and residence changes, and number of school transfers documented in the child's school records. See Marcynyszyn, Evans, and Eckenrode (2008) for additional information about this measure.

Variables used to this measure were collected at the 15-year follow-up and d total number of months a woman was employed since the birth of her first child, number of months a family was on Aid to Families with Dependent Children since the birth of the first child, highest grade attended by the mother, and family economic hardship. Family economic hardship was assessed with the Economic Hardship Questionnaire, a 12-item measure of changes in a family's style of living during the past six months (Lempers, Clarklempers, & Simons, 1989). Poor family economic self-sufficiency scores were computed by reverse coding months of maternal employment and highest maternal education, standardizing all family economic self-sufficiency variables, and summing all the variables.

At the 15-year follow-up, women reported the number of subsequent children they had.

We computed composite poor maternal outcome scores by reverse coding maternal mental health, standardizing all maternal outcome variables, and then summing the standardized maternal outcome variables.

We used three statistical analyses to assess associations between individual outcomes and composite poor outcome scores, and we standardized all predictors and outcomes so we could compare the magnitude of the associations. First, we examined bivariate associations by assessing correlations between each risk factor and outcome. Second, we conducted multivariate regression analyses, which d all risk factors as predictors in each model. Third, we conducted regression analyses using cumulative risk scores to predict each outcome.

See Table 1 for deive statistics and information about missing data. All risk factors were significantly, but moderately, correlated with the other risk factors and the cumulative risk score (results not shown).

See Table 2 for correlations between risk factors and outcomes. All risk factors were associated with multiple individual outcomes, such that greater levels of a risk factor predicted poorer outcomes. Younger maternal age and less maternal education were associated with poorer composite child outcome scores, and all four risk factors predicted poorer composite maternal outcome scores. Overall, marital status and household SES were less robust predictors of child outcomes, compared to maternal age and education.

See Table 3 for multivariate results. Compared to the bivariate models, there were fewer statistically significant results in the multivariate models, which assessed the impact of each risk factor while controlling for other risk factors. This suggests that results from multivariate models gave a more accurate estimate of the independent association between the risk factor and outcome. In all but one case, greater levels of a risk factor predicted poorer outcomes in the multivariate models. The results from the multivariate models showed that younger maternal age was associated with more family instability and subsequent children; lower levels of maternal education was associated with more child maltreatment reports and poorer family economic self-sufficiency; mothers being unmarried was associated with poorer maternal health, higher levels of maternal drug/alcohol impairment, higher levels of family instability, and poorer family economic self-sufficiency; and lower household SES was associated with lower levels of child academic achievement, more child maltreatment reports, and poorer family economic self-sufficiency. The only instance a risk factor predicted lower levels of a poor outcome was younger maternal age at intake predicted better family economic self-sufficiency scores.

None of the risk factors predicted composite poor child outcome scores. However, lower levels of maternal education, being unmarried at intake, and lower levels of household SES predicted less favorable composite maternal outcome scores.

The number of families at each level of cumulative risk was similar (see Fig. 1). See Table 4 for cumulative risk regression results. Cumulative risk significantly predicted the majority of the outcomes and the composite child and maternal poor outcome scores. In general, there was an increase in negative outcomes as the number of risk factors present increased (see Fig. 2 for examples).

Fig. 1.

(0.05MB).

Cumulative risk score distribution.

 

Fig. 2.

(0.29MB).

Examples of cumulative risk results.

 

We used data from the control group of the Elmira randomized trial of the NFP program to illustrate how analyses of population-specific data can help program implementers better target HV programs to families most likely to benefit from them. We used three types of analyses, specifically bivariate, multivariate, and cumulative risk analyses, to look at the association of risk factors with individual outcomes and child and maternal composite outcome scores.

A comparison of results from the bivariate models and multivariate models showed that there were several risk factors that statistically significantly predicted outcomes in the bivariate models, but no longer predicted outcomes once other risk factors were aced for in the multivariate models. This suggests that, if possible, program implementers should rely on results from multivariate analyses, rather than bivariate analyses, to obtain a more accurate depiction of what maternal or child health outcome should be targeted for at-risk families.

Once these outcomes have been identified, program implementers can a home-visiting program that has been shown to effectively address those outcomes (see the HomVEE Review for a summary of the outcomes affected by particular HV programs). For example, our multivariate results suggested that lower SES families in Elmira, New York were particularly at-risk for child maltreatment, poor family economic self-sufficiency, and poor child academic achievement. Based on the HomVEE review there were two programs (i.e., NFP and Parents as Teachers) that target all three of these outcomes. Similarly, because our results indicated that maternal age (and not education, marital status, or SES) predicted number of subsequent pregnancies, if the policy concern is reducing unplanned pregnancies, our analyses suggest that the best strategy may be targeting mothers in Elmira based on age with a program shown to impact the number of subsequent pregnancies (e.g. NFP).

Targeting families with a risk factor that has been shown to be associated with an adverse outcome increases the probability of delivering an intervention to a family with a mother or child that otherwise may have developed an adverse outcome. It is important to note that in any given community there will be women and children from families without specified risk factors, or have lower levels of a risk factor, that do develop adverse outcomes (e.g., children of older mothers may adverse outcomes). However, from a statistical perspective, targeting an at-risk family, rather than a family that does not have risk factors that have been shown to be associated with adverse outcomes, has a greater chance of having an impact. Thus, given that resources for HV programs are quite limited, it may be prudent to focus efforts on targeting families with risk factors that have been shown to be associated with adverse outcomes. The “prevention paradox” as popularized by the epidemiologist Geoffrey Rose (1985) is that “a large number of people at a small risk may give rise to more cases of disease than the small number who are at high risk.” However, offering expensive programs like home visitation to low risk families is beyond the capacity of most states and communities, so targeting high-risk families is still the preferred approach by many policy makers.

In addition to the bivariate and multivariate analyses, which assessed associations between individual risk factors and maternal and child outcomes, we also conducted cumulative risk analyses to assess the association between the number of risk factors present in families’ lives and maternal and child outcomes. We found that, in contrast to the estimated impact of individual risk factors in the multivariate models, cumulative risk scores predicted the majority of the individual outcomes, as well as the maternal and child composite outcome scores. This suggests that, if the policy goal is to impact multiple outcomes, several risk factors should be utilized in ing families and interventions should be chosen that have been shown to effectively impact a number of outcomes, such as NFP or Healthy Families America. The challenge with this approach, however, is to evaluate the costs and benefits of using cumulative risk scores in practice, since many community or state agencies may be uncomfortable limiting services to the small number of families at the very highest level of risk (e.g. only 13.5% of families in our Elmira sample had all four risk factors).

There were several limitations to the present study. First, the sample size was relatively small, which may have limited the statistical power to detect statistically significant results. Second, we also only examined a limited number of risk factors, and there are other important socio-demographic risk factors, such as parental employment status, that should also be considered when targeting families for HV programs. Third, the baseline risk factors were assessed in the late 1970s and early 1980s, which may limit the generalizability of our findings to the current social and political climate (e.g. in a post welfare-reform era) given that variables collected then may not have the same meaning as they do now. However, the purpose of our analyses was not to evaluate how HV programs should be disseminated in the population assessed in our analyses, but rather to demonstrate how population-specific analyses can be conducted to facilitate effective targeting of HV programs. Fourth, we provide an illustration of a method to target existing home-visiting programs, prior to implementation, to families most likely to benefit from them. Future research should also evaluate how home-visiting programs can more precisely target specific services to specific families within their own existing models once programs are adopted in a community. For example, a recent clinical trial, within the Nurse Family Partnership program, demonstrated that allowing nurses to tailor the frequency of visits and content of the program, depending on the needs and preferences of the parent participants, led to greater retention of parents in the program (Olds et al., 2015). Of course, modifications of program protocols should be done in the context of a rigorously researched program of quality improvement to assure that core program elements are not sacrificed in the name of personalizing program content.

In sum, our analyses illustrate the utility of population-specific examinations of associations between socio-demographic risk factors used for recruiting families into HV programs and the outcomes that are the intended targets of these interventions. The DHHS criteria specified that in order for a home visiting program to be considered effective, it had to evidence for effectiveness for only one or two outcomes. However, in practice, program implementers often operate under the assumption that evidence-based home visiting programs should be made available to families representing a variety of risk profiles and that they will be effective for improving multiple maternal and child outcomes. Our results demonstrated how conducting analyses using a sample from a particular population can help program implementers more systematically target HV programs to the at-risk families most likely to benefit from the interventions.

Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health (Grant No. T32MH103213). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This work was also supported by a National Science Foundation Graduate Research Fellowship (Grand No. 1342962).

The authors declare that they have no conflicts of interest.