Identifying Adjustment Difficulties in Chilean Schools: Psychometric Properties of the Teacher Observation of Classroom

Introduction

One of the main tasks in child-adolescent development is to adapt to the school environment. When a successful school adaptation is achieved, young people display better psychosocial adjustment in other contexts and their subjective well-being increases (Gutiérrez & Gonçalves, 2013; Viñas Poch et al., 2015).

School adjustment involves aspects such as motivation, participation in activities in the classroom, attitudes towards teachers, and being able to follow the rules and norms of behavior (Buyse et al., 2009; Ladd & Troop-Gordon, 2003). It is based on competencies such as social skills and problem-solving skills, decision-making skills, self-regulation, self-efficacy, and skills related to academic performance, among others (Fabian, 2006; Margetts, 2009).

When problems of school adjustment appear, it persists over time. For this reason, the early identification of students at risk can help prevent these problems (Margetts, 2009).

In this context, the school becomes one of the ideal places to conduct preventive interventions (Leiva, George, Squicciarini, et al., 2015; Weist et al., 2014). In Latin America, as in other parts of the world, these are useful for tackling several psychosocial and mental health issues (Garfin et al., 2014; Leiva, George, Antivilo, et al., 2015; Leiva, George, Squicciarini, et al., 2015).

Given this scenario, the school context is essential for psychosocial risk screening because it provides a short-term vision of the adaptation of children and adolescents to expected behaviors. Also, the schools offer a strategic opportunity to identify the early behaviors and symptoms of mental disorders through screening.

Schools offer an ideal medium for engaging in large-scale screening of children given that they provide standards of what is expected for their ages, offering a developmental vision of the children’s functioning (Gall et al., 2000). Teachers can provide information regarding a child’s behavior in the classroom and interaction with the other children in the grade. In fact, the classroom is a space of interaction that reveals adaptive and/or dysfunctional behaviors as well as strategies for managing them (Pagano et al., 2000).

This helps identify children that present mental health risk-type behaviors (George et al., 2012). The importance of detecting the risk in these spaces lies in their implications. For example, students with a mental health risk based on broad band screening – screening to measure general psychosocial functioning or mental health – (NRC & IOM, 2009) score low on standardized tests compared to students who have no mental health risk, controlling for covariates such as poverty, parents’ education, and pre-existing academic ability (Guzman et al., 2011).

However, school is not only one of the optimal spaces for early detection of risk factors and disorders. It is also ideal for conducting protective and restorative interventions (George et al., 2012). This evidence, combined with the crucial socializing influence of school and the connection between learning and mental disorders (Domitrovich et al., 2009; Macklem, 2014; Petras et al., 2005), makes it necessary to implement preventive strategies to reduce the future risk of psychological and behavioral disorders.

One of the keys to successful prevention in school is to detect adjustment difficulties early on (Ialongo et al., 2000; Kellam et al., 2011). To do this it is fundamental to generate universal assessment methods that provide timely information about the students who are at the highest risk of suffering from mental health problems (Kamphaus et al., 2014; Kellam et al., 2011).

This assessment requires brief, precise, valid, and reliable psychometric instruments that match the characteristics of each school (Daniels et al., 2017; Dowdy et al., 2010). One of the instruments that meet these requirements is the Teacher Observation of Classroom Adaptation-Revised (TOCA-R).

Teacher Observation of Classroom Adaptation-Revised (TOCA-R)

The TOCA-R measures school adaptation difficulties through a structured interview aimed at teachers (Kellam et al., 2011; Werthamer-Larsson et al., 1991). The teacher provides the information in the context of an interview conducted by a trained assessor. The teachers rate the adaptation of a student using a six-point Likert scale. The score reflects the frequency of occurrence of the behaviors, where 1 represents never and 6 almost always.

The TOCA-R is a revision of the TOCA (Kellam et al., 1975). The original TOCA was developed by the Woodlawn Research Center (Chicago, IL) to assess – in first grade – social adaptive classroom behaviors, as viewed by their teachers (Werthamer-Larsson et al., 1991).

Initially, the TOCA was a set of 110 items based on the existing literature about behavioral antecedents of mental health problems. Subsequently, this set of questions was reviewed and reduced to 58 items, which corresponded to behavioral aspects of the DSM-III childhood disorder criteria. These 58 items were tested on a sample of 200 children to reduce the number of items. All those items with low correlation, ambiguous factor loading, or low factor loading were eliminated to produce the 31 items used in the final instrument (Werthamer-Larsson et al., 1991).

This version underwent an important review by the Johns Hopkins Center for Early Prevention and Intervention for use in a large intervention study, giving rise to the TOCA-R (Koth et al., 2009). Similar to the original TOCA, the TOCA-R was administered by a trained evaluator in a face-to-face interview.

Although the original study by Werthamer-Larsson et al. (1991) considers a total of 31 items and three factors, subsequent studies report a different number of items and factors. Schaeffer et al. (2003) mention that teachers respond to 36 items about a child’s adaptation to classroom task demands, which account for three factors. Rains (2003), in his Fast Track Project Technical Report study, uses 16 items and 4 factors. On the other hand, Koth et al. (2009) report that the instrument includes 43 items of classroom behaviors and three factors. However, all of the scales showed an adequate level of internal reliability (see Table 1).

It should be mentioned that although the items account for conceptually similar factors, there is no clarity or agreement regarding the number of final items in the instrument.

Regarding the TOCA-R, it has gained exposure due to its robust psychometric properties (Koth et al., 2009; Kourkounasiou & Skordilis, 2014; Vahedi & Hajipour, 2012). This instrument has shown high levels of reliability (Lochman, 1995; Rains, 2003) - the alpha coefficient for each subscale was higher than .80 (Johns Hopkins Prevention Intervention Research Center, 2006; Werthamer-Larsson et al., 1991). Further, the test-retest correlation, with different interviewers, was approximately .75, and it had good internal consistency (King et al., 2011).

The instrument is validated by its predictive value and ability to describe children’s behavior reliably. It allows children with risk factors and unhealthy behaviors associated with mental health problems and a greater likelihood of presenting psychiatric disorders and risky behaviors as adults to be identified (Kellam et al., 2008; King et al., 2011; Koth et al., 2009; Lochman, 1995; Lynne-Landsman et al., 2010; Schaeffer et al., 2003). The TOCA-R predicts several externalizing disorder symptoms (Racz et al., 2013). It has been shown that disruptive/aggressive behavior in school is a predictor of violence among adolescents (Petras et al., 2004) and criminality in young adults (Schaeffer et al., 2006, 2003). Together their solid theoretical-empirical foundations have allowed testing the causal mechanisms that support the processes of change through experimental and longitudinal studies (Lee et al., 2011).

Also, this instrument was designed for administration in schools, adapting satisfactorily to their schedules and routines (Kellam et al., 2011; Koth et al., 2009), and its administration shows how effective the cost is by comparing it to a formal interview format (Koth et al., 2009).

The instrument is based on the life course/social field theory (Ialongo et al., 2000; Kellam et al., 2011), which regards mental health as the product of the tensions between the demands posed by the social contexts of each developmental stage and a subject’s ability to adapt to them (Kellam et al., 2011).

In this regard, the classroom constitutes one of the most relevant social fields in students’ life, given that they spend most of their time in it, meeting academic-social demands that vary depending on school culture and norms (Kellam et al., 2011). For this reason, the assessment made by teachers is key, because it expresses the adaptive tensions that predict their students’ future outcomes (Kellam et al., 2011; Petras et al., 2005).

In Latin America, the instrument was adapted in Chile using the 31 items proposed by Werthamer-Larsson et al. (1991), and that gave rise to the TOCA-R.

Despite using the same items proposed in the original version, six factors were established in the Chilean version (George et al., 1994), which sought to identify difficulties presented by children in the school context. These factors responded to domains identified theoretically to conduct mental health screenings in school contexts. However, there was no statistical certainty of the existence of the proposed factors.

This Chilean version also identified as TOCA-R (George et al., 1994) comprised Likert-type items and measured school adjustment difficulties in six domains: authority acceptance, cognitive achievements, social contact, emotional maturity, attention/concentration, and activity level. This tool was used in longitudinal studies of children at psychosocial risk at the beginning of their schooling (de la Barra et al., 2002, 2003, 2004; George et al., 1994). It is showing similar research findings of the prevalence of mental health disorders in the youth population (Toledo et al., 1997).

Later, George et al. (2004) reviewed and adjusted the instrument calling it TOCA-RR (Teacher Observation of Classroom Adaptation-R-Revised). This work was made with larger cohorts with the specific purpose of seeking a reduction in the number of items to facilitate its application in a program with national coverage. The results maintained the number of items and factors present in the first Chilean version of TOCA-R (George et al., 1994).

Also, this instrument was used as a screening tool (Murphy et al., 2015), a measure for assessing school programs (Guzmán et al., 2015; Leiva, George, Antivilo, et al., 2015; Leiva, George, Squicciarini, et al., 2015), and a predictive indicator of school performance (Guzman et al., 2011).

However, despite its extensive use, its psychometric specificities have not been assessed in adolescents, a population that is at a higher risk of suffering from mental health problems (Patel et al., 2007).

The assessment of school adaptation difficulties in adolescents is a critical aspect of mental health interventions in schools because this stage of development constitutes a window of opportunity to tackle problems effectively and efficiently. After this time passes, it has been observed that at-risk students who did not receive treatment develop persistent and refractory psychiatric problems entailing high social and personal costs (Kellam et al., 2011; Murphy et al., 2015).

Considering the above, the purpose of this study is to assess the psychometric properties of TOCA-RR in Chilean adolescents to contribute to the generation of valid and reliable screening instruments that can be used in research and school-based preventive interventions.

Method

Participants

The sample was not probabilistic and comprised 10,287 6th and 8th grade students from 136 public and subsidized schools from 12 regions of the country. The subsidized schools have shared funding, but they are establishments that are financed fundamentally with the contribution of the state. Public and subsidized schools have the same selection and similar characteristics in terms of academic structure and results.

At the time of assessment, all of them were in schools identified as presenting a high level of risk based on the National Association of School Assistance and Scholarships criteria (known as JUNAEB for its Spanish acronym) (JUNAEB, 2005). This criterion is constructed with variables associated with the socioeconomic status of those who enter the school system financed by the state. This criterion is an indicator of poverty. In the case of the young in this study, they belonged to vulnerable and low socioeconomic sectors.

Of the total sample, 5,815 (56.5%) students were in 6th grade and were 12.91 years old on average (SD = 1.391), 8,959 (87.1%) attended public schools, 1,179 (11.5%) lived in rural areas, 4,705 (45.7%) are beneficiaries of Solidary Chile a program destined to family in extreme poverty, and 4,768 (46.3%) were female.

Seven hundred and thirty-two teachers were interviewers. The average of children evaluated by each teacher was 14.053 (SD = 5.149). It is noteworthy that those teachers interviewed are teachers in charge of the class. In addition, the evaluation is done in the middle of the school year, when teachers already recognize their students.

The sample was obtained through the screening process included in Skills for Life (SFL), a national mental health program aimed at schools. The SFL program is implemented in public and subsidized private schools in Chile. The program currently has national coverage and has been a structured public response implemented in collaboration with local governments in order to promote awareness and prevention of mental health (George et al., 2004). The program is conducted in schools and involves the entire educational community, including principals, teachers, students, and parents.

The purpose of the program in the short-term is to increase success in school performance, elevate learning levels, and decrease the number of students who repeat a grade or drop out. In the long-term, it seeks to reduce health problems such as depression, suicide, and alcohol and drug use, prevent violent behavior, increase wellbeing, cultivate personal skills, and increase life expectancy.

Currently, the SFL program has a national reach and a high level of coverage in schools for children and adolescents aged 4 to 14. It is, therefore, necessary to assess and update the psychometric properties of the tools used in a program that operates on a large scale and needs studies that systematically validate the quality of the instruments used.

The details of this intervention and its results have been reported in other publications (Guzmán et al., 2015; Leiva, George, Antivilo, et al., 2015; Leiva, George, Squicciarini, et al., 2015).

Instrument

Teacher Observation of Classroom Adaptation R-Revised (TOCA-RR). The instrument measures classroom behavior. The teacher provides the information in the context of an interview conducted by a trained assessor. The teachers rate the adaptation of the student using a six-point Likert scale. The score reflects the frequency of occurrence of the behaviors, where 1 represents never and 6 almost always.

In this research, the Chilean version (TOCA-RR) was used. This instrument comprises 31 Likert-type items and measures school adjustment difficulties in six domains: authority acceptance (e.g., harms others, starts fights), cognitive achievements (e.g., completes assignments, works well alone), social contact (e.g., is sociable), emotional maturity (e.g., clings to the teacher), attention/concentration (e.g., is on task), and activity level (e.g., cannot sit still). These dimensions are calculated by adding up the items after inverting the values of positive statements.

Procedures and Data Analysis

The data were obtained from the screening administered to students who participated in the SFL program, within the context of an academic-institutional collaboration agreement. After obtaining parents’ consent and students’ assent, a team mostly composed of psychologists or social workers administered TOCA-RR to the homeroom teacher.

To examine the instrument’s psychometric properties, an exploratory factor analysis (EFA) was conducted with half the 6th and 8th grade students (n = 5,144). Afterward, a confirmatory factor analysis (CFA) was conducted with the remaining half (n = 5,145) in order to corroborate the structure identified. The samples for the analyses were determined by randomly assigning participants to two groups. The analyses were conducted using IBM SPSS and AMOS 22.

Factors were extracted with the maximum verisimilitude method and Varimax rotation. At the same time, the global fit was assessed through the Kaiser-Meyer-Olkin index of sampling adequacy and Bartlett’s test of sphericity. The number of factors retained was determined through a combination of statistical and visual criteria (Hair, et al., 2001). It was established that a high factor loading would be one with values over .40 and that a factor would require at least three items with high factor loadings to be preserved. In addition, the commonality of each item was analyzed to assess the scope of the explanation of the factors (Hair et al., 2001). In order to estimate the similarity of the factor structure between sexes and grades, the congruity coefficient was used (Lorenzo-Seva & ten Berge, 2006). It was decided to conduct an AFE because we had information about the structural composition of this instrument in 1st grade student population, but not on other levels. On the other hand, we wanted to know if the structural configuration was the same for males and females or if there were specifics for each gender.

Regarding CFA, CFI (comparative fit index), TLI (Tucker-Lewis index), RMSEA (root mean square error of approximation), and SRMR (standardized root mean square residual) were calculated. The values of these indicators were compared considering the model generated in this study, and that proposed for students from 1st to 4th grade (George et al., 1994). Finally, the reliability of each dimension was calculated using Cronbach’s alpha, average variance extracted (AVE), and the composite reliability coefficients (CR). Differences by sex, grade, rurality, and extreme poverty were examined through the Student’s-t test.

Results

Exploratory Factor Analysis

The KMO index (Kaiser, 1974) was estimated at .952, while Bartlett’s test of sphericity displayed adequate values, χ²₍₄₆₅₎ = 104721.874, p < .001). For factor extraction, latent roots, the variance explained by the factors (58.16%), and the visual aspect of the scree plot were considered, whose analysis suggested preserving four.

The rotated component matrix was reviewed in order to interpret the factors. The analysis of factor loadings revealed that items adequately saturated a single factor, except for items 15 and 19. The four factors were composed of more than three variables with high factor loadings. The first factor was found to be composed of 11 items, while the second comprised 9, the third 6, and the fourth 5. The commonality analysis revealed that all items were included in at least one factor and that their variance was satisfactorily explained by the factors extracted, except for items 10, 27, and 31, which displayed moderate values (see Table 2). Two criteria were used to identify the factors: a statistical one, which involved considering the highest factor loadings as indicators of the latent construct, and a theoretical-practical one, which entailed naming the factors following theoretical conventions that facilitated interpretation.

In the first factor, items 24 (starts fights with classmates) and 8 (harms or hurts others) display the highest factor loadings, while the rest of the items refer to aggressive, oppositional, and hyperactive behaviors; therefore, the factor was labeled Behavioral Difficulties (BD). In the second factor, inverted items 1 (completes assignments) and 4 (focuses) have the highest factor loadings, while the rest refer to behaviors involving poor performance in school tasks and a lack of attention in class, the factor was labeled Learning Difficulties (LD). In the third factor, inverted items 6 (is sociable/interacts with classmates) and 2 (friendly) display the highest factor loadings, while the rest of the items describe students’ level of sociability in school; therefore, the factor was labeled Social Difficulties (SD). In the fourth factor, the items with the highest loadings are 20 (clings to the teacher) and 3 (demands too much attention from the teacher), while the rest refer to the need for social attention and the desire to bond with others. Considering the above, the factor was entitled Autonomy Difficulties (AD).

Similarities between the Factors by Sex and Grade

To calculate the index of similarity between the factors by sex and grade, EFAs were carried out for each subgroups. In all cases, four factors were found after completing the same procedures described above. Afterward, the factor loadings were used to calculate the congruence coefficient (CC) according to the recommendations laid out by Lorenzo-Seva and ten Berge (2006). In the case of the sex variable, the CC was found to range from .989 to .993, which shows that the dimensionality structure of the TOCA-R is similar for men and women. For the grade variable, the CC ranged from .992 to .996, which shows that the factors are identical for the two grades.

Confirmatory Factor Analysis

Goodness of fit indicators were calculated for the four-factor model extracted from the EFA of the 31 items included in TOCA-RR. The model considered the grid of orthogonal latent constructs connected by covariance relations. The indicators were found to be only marginally adequate considering the recommendations reported in the international literature (Bandalos & Finney, 2010) (see Table 3), which led to the elimination of some items in order to improve these values.

A review of the standardized regression loadings revealed that three items (16, 31, 27) were below the general range. With respect to item 31, the results of the EFA revealed its dissimilar behavior; therefore, its elimination was coherent and well supported by evidence. With respect to item 16 (is confident), the CFA appears to confirm the theoretical intuition that being sure of oneself belongs to another construct (e.g., self-esteem), despite being linked to the learning difficulties factor. Therefore, it was also eliminated. Similarly, item 27 (needs affection to feel motivated to work) was eliminated due to its low theoretical concordance with the emotional dependence factor. The assessment of the new 28-item model showed that the indicators improved in all the pre-established criteria. Afterward, the fit indicators of the model for grade 1-4 students proposed by George et al. (1994) were calculated and then compared with those obtained in the present study. The four-factor model with 28 items was found to be the best in all criteria (see Table 3) (see Figure 1).

Internal Consistency

All dimensions displayed high reliability. For the total scale (28 items), Cronbach’s alpha reached .914 for BD, .948 for LD, .892 for SD, and .813 for AD. The AVE was .50 for BD, .68 for LD, .62 for SD, and .52 for AD. All values were greater than or equal to .50, which indicates that the construct explains more than half of the variance of all the indicators that compose it (Hair et al., 2009). Regarding the CR, .71 was obtained for BD, .81 for LD, .78 for SD, and .73 for AD. The values obtained indicate good levels of composite reliability, since all of them are greater than .70 (Hair et al., 2009).

Descriptive Data and Sociodemographic Differences in the TOCA-RR for Adolescents

Descriptive data are shown in Table 4. Differences in the scores of the factors according to sociodemographic characteristics were established using the Student t-test, given the normality of the variables tested through the Kolmogorov-Smirnov test. It was found that the score of behavior, learning, and autonomy difficulties has a significant difference between gender and grade. At the same time, poverty only accounts for differences regarding behavioral and learning difficulties (see Table 5)

Discussion

The aim of the present study was to assess the psychometric properties of TOCA-RR in adolescent students from Chile. Results show that the factor structure of the instrument must be reconsidered for this population because it was found that the empirical structure of the instrument is composed of four factors, not six, as in the case of students from 1st to 4th grade.

The existence of four factors with 28 items and not six factors with 31 items can be explained by the fact that the instrument proposed by George et al. (2004) was never analyzed psychometrically and that the six factors were established only in theoretical terms.

The names of the four factors were chosen to facilitate their use and interpretation (Behavioral Difficulties, Learning Difficulties, Social Difficulties, and Autonomy Difficulties). Since TOCA-R was constructed upon the basis of teachers’ identification of school maladjustment, it has always been dependent on the precision of the behavioral taxonomy behind teacher observations (Koth et al., 2009; Werthamer-Larsson et al., 1991); therefore, the changes in the factors’ names and specifications are part of the maturation of the instrument. In this context, the decision was made to complete this adjustment due to the requirements of the adolescent population and Chile’s cultural specificities, taking special care to consider the clinical significance of dimensions.

The new names have significant theoretical-practical consequences for the use of the instrument in school mental health programs: in this field, it is common to confuse risk with the presence of mental health problems, which affects the type of interventions carried out. It is essential to regard school adaptation difficulties as risk factors for developing mental health issues in the future (Lee et al., 2011; Petras et al., 2005) and not as mental health problems in themselves. It will make it possible to generate preventive strategies that improve student adjustment through the development of socioemotional competencies aligned with schools’ goals (Forman et al., 2009; Macklem, 2014).

Reliability was high in all factors, which makes it advisable to use this instrument in screening processes. This result is fundamental: due to the importance of detection in school mental health programs and the subsequent decisions made based on this information, it is necessary to use instruments that determine what is being measured and that are consistent when applied to multiple students. Also, it is required to research the sensitivity and specificity of this instrument, considering psychosocial risks, academic performance, and school dropout to improve the precision of the classifications made and the risk estimated.

Concerning comparative analyses, even though significant differences were found between the subgroups assessed, the small effect sizes calculated suggest that said differences must be studied in more depth and interpreted cautiously (Koth et al., 2009; Vahedi et al., 2012). Nevertheless, it is important to consider the differences found between adolescents living in extreme poverty (and who receive social aid through several public programs) and those not experiencing such conditions. The same result has been reported in several international studies: socioeconomic inequality increases risk, which is an especially relevant finding in Latin America, an unequal region in the world (Ortiz-Hernández et al., 2007). In this regard, Reiss (2013) found that economically disadvantaged adolescents are two to three times more likely to develop a mental health problem, which makes it necessary to consider that prevention strategies in the field of mental health must be aimed not only at developing competencies but also at dealing with inequality.

The main limitation of the study is the lack of representation of students from higher socioeconomic levels. Even though the sample is large and comprises schools from several regions of the country, it only represents the reality of public and subsidized schools, and it does not allow the results to be generalized to other possible populations of interest. It may constitute a bias, especially when many parents prefer to make an economic effort and enroll their children in private schools, which are not always beneficiaries of the program used to construct the sample.

Another limitation that should be considered is the lack of knowledge that the teacher might have of his/her students. Although a teacher’s self-report is carried out in the middle of the school year, there is always the chance of not recognizing who is being evaluated.

Finally, the lack of concurrent validity analysis should also be considered as a limitation, which could be tested in future studies.

This study suggests that TOCA-RR has robust psychometric properties for the assessment of school adaptation in adolescents from 6th to 8th grade and that it constitutes a valid and reliable screening tool for identifying risks in this stage of development. Future Latin American studies should focus on analyzing its predictive power and risk trajectories in the development of mental health problems. In addition, given its growing popularity in many regions of the world, it is necessary to make an effort to explore common transcultural factors that can improve our understanding of adaptation in multiple school contexts and cultures.