Conflict of Interest
The authors of this article declare no conflict of interest.
For preventive interventions in school mental health to be effective, it is essential to have valid and reliable screening instruments that can detect students in need of more support. In order to do this, the present study is aimed at assessing the psychometric properties of the Teacher Observation of Classroom Adaptation, an interview for teachers that measures school adjustment difficulties. The study evaluated 10,287 adolescents participating in a Chilean public program for improving mental health in schools. Of the total sample, 5,815 (56.5%) students were in 6th grade and were 12.91 years old on average (
Para que las intervenciones preventivas en salud mental escolar sean efectivas es esencial contar con instrumentos de detección válidos y confiables, que puedan identificar a los estudiantes que necesitan apoyo. Por ello, este estudio tuvo como objetivo evaluar las propiedades psicométricas del
One of the main tasks in child-adolescent development is to adapt to the school environment. When a successful school adaptation is achieved, young people display better psychosocial adjustment in other contexts and their subjective well-being increases (
School adjustment involves aspects such as motivation, participation in activities in the classroom, attitudes towards teachers, and being able to follow the rules and norms of behavior (
When problems of school adjustment appear, it persists over time. For this reason, the early identification of students at risk can help prevent these problems (
In this context, the school becomes one of the ideal places to conduct preventive interventions (Leiva, George, Squicciarini, et al., 2015;
Given this scenario, the school context is essential for psychosocial risk screening because it provides a short-term vision of the adaptation of children and adolescents to expected behaviors. Also, the schools offer a strategic opportunity to identify the early behaviors and symptoms of mental disorders through screening.
Schools offer an ideal medium for engaging in large-scale screening of children given that they provide standards of what is expected for their ages, offering a developmental vision of the children’s functioning (
This helps identify children that present mental health risk-type behaviors (
However, school is not only one of the optimal spaces for early detection of risk factors and disorders. It is also ideal for conducting protective and restorative interventions (
One of the keys to successful prevention in school is to detect adjustment difficulties early on (
This assessment requires brief, precise, valid, and reliable psychometric instruments that match the characteristics of each school (
The TOCA-R measures school adaptation difficulties through a structured interview aimed at teachers (
The TOCA-R is a revision of the TOCA (
Initially, the TOCA was a set of 110 items based on the existing literature about behavioral antecedents of mental health problems. Subsequently, this set of questions was reviewed and reduced to 58 items, which corresponded to behavioral aspects of the DSM-III childhood disorder criteria. These 58 items were tested on a sample of 200 children to reduce the number of items. All those items with low correlation, ambiguous factor loading, or low factor loading were eliminated to produce the 31 items used in the final instrument (
This version underwent an important review by the Johns Hopkins Center for Early Prevention and Intervention for use in a large intervention study, giving rise to the TOCA-R (
Although the original study by
It should be mentioned that although the items account for conceptually similar factors, there is no clarity or agreement regarding the number of final items in the instrument.
Regarding the TOCA-R, it has gained exposure due to its robust psychometric properties (
The instrument is validated by its predictive value and ability to describe children’s behavior reliably. It allows children with risk factors and unhealthy behaviors associated with mental health problems and a greater likelihood of presenting psychiatric disorders and risky behaviors as adults to be identified (
Also, this instrument was designed for administration in schools, adapting satisfactorily to their schedules and routines (
The instrument is based on the life course/social field theory (
In this regard, the classroom constitutes one of the most relevant social fields in students’ life, given that they spend most of their time in it, meeting academic-social demands that vary depending on school culture and norms (
In Latin America, the instrument was adapted in Chile using the 31 items proposed by
Despite using the same items proposed in the original version, six factors were established in the Chilean version (
This Chilean version also identified as TOCA-R (
Later,
Also, this instrument was used as a screening tool (
However, despite its extensive use, its psychometric specificities have not been assessed in adolescents, a population that is at a higher risk of suffering from mental health problems (
The assessment of school adaptation difficulties in adolescents is a critical aspect of mental health interventions in schools because this stage of development constitutes a window of opportunity to tackle problems effectively and efficiently. After this time passes, it has been observed that at-risk students who did not receive treatment develop persistent and refractory psychiatric problems entailing high social and personal costs (
Considering the above, the purpose of this study is to assess the psychometric properties of TOCA-RR in Chilean adolescents to contribute to the generation of valid and reliable screening instruments that can be used in research and school-based preventive interventions.
The sample was not probabilistic and comprised 10,287 6th and 8th grade students from 136 public and subsidized schools from 12 regions of the country. The subsidized schools have shared funding, but they are establishments that are financed fundamentally with the contribution of the state. Public and subsidized schools have the same selection and similar characteristics in terms of academic structure and results.
At the time of assessment, all of them were in schools identified as presenting a high level of risk based on the National Association of School Assistance and Scholarships criteria (known as JUNAEB for its Spanish acronym) (
Of the total sample, 5,815 (56.5%) students were in 6th grade and were 12.91 years old on average (
Seven hundred and thirty-two teachers were interviewers. The average of children evaluated by each teacher was 14.053 (
The sample was obtained through the screening process included in Skills for Life (SFL), a national mental health program aimed at schools. The SFL program is implemented in public and subsidized private schools in Chile. The program currently has national coverage and has been a structured public response implemented in collaboration with local governments in order to promote awareness and prevention of mental health (
The purpose of the program in the short-term is to increase success in school performance, elevate learning levels, and decrease the number of students who repeat a grade or drop out. In the long-term, it seeks to reduce health problems such as depression, suicide, and alcohol and drug use, prevent violent behavior, increase wellbeing, cultivate personal skills, and increase life expectancy.
Currently, the SFL program has a national reach and a high level of coverage in schools for children and adolescents aged 4 to 14. It is, therefore, necessary to assess and update the psychometric properties of the tools used in a program that operates on a large scale and needs studies that systematically validate the quality of the instruments used.
The details of this intervention and its results have been reported in other publications (
In this research, the Chilean version (TOCA-RR) was used. This instrument comprises 31 Likert-type items and measures school adjustment difficulties in six domains: authority acceptance (e.g., harms others, starts fights), cognitive achievements (e.g., completes assignments, works well alone), social contact (e.g., is sociable), emotional maturity (e.g., clings to the teacher), attention/concentration (e.g., is on task), and activity level (e.g., cannot sit still). These dimensions are calculated by adding up the items after inverting the values of positive statements.
The data were obtained from the screening administered to students who participated in the SFL program, within the context of an academic-institutional collaboration agreement. After obtaining parents’ consent and students’ assent, a team mostly composed of psychologists or social workers administered TOCA-RR to the homeroom teacher.
To examine the instrument’s psychometric properties, an exploratory factor analysis (EFA) was conducted with half the 6th and 8th grade students (
Factors were extracted with the maximum verisimilitude method and Varimax rotation. At the same time, the global fit was assessed through the Kaiser-Meyer-Olkin index of sampling adequacy and Bartlett’s test of sphericity. The number of factors retained was determined through a combination of statistical and visual criteria (Hair, et al., 2001). It was established that a high factor loading would be one with values over .40 and that a factor would require at least three items with high factor loadings to be preserved. In addition, the commonality of each item was analyzed to assess the scope of the explanation of the factors (
Regarding CFA, CFI (comparative fit index), TLI (Tucker-Lewis index), RMSEA (root mean square error of approximation), and SRMR (standardized root mean square residual) were calculated. The values of these indicators were compared considering the model generated in this study, and that proposed for students from 1st to 4th grade (
The KMO index (
The rotated component matrix was reviewed in order to interpret the factors. The analysis of factor loadings revealed that items adequately saturated a single factor, except for items 15 and 19. The four factors were composed of more than three variables with high factor loadings. The first factor was found to be composed of 11 items, while the second comprised 9, the third 6, and the fourth 5. The commonality analysis revealed that all items were included in at least one factor and that their variance was satisfactorily explained by the factors extracted, except for items 10, 27, and 31, which displayed moderate values (see
In the first factor, items 24 (starts fights with classmates) and 8 (harms or hurts others) display the highest factor loadings, while the rest of the items refer to aggressive, oppositional, and hyperactive behaviors; therefore, the factor was labeled Behavioral Difficulties (BD). In the second factor, inverted items 1 (completes assignments) and 4 (focuses) have the highest factor loadings, while the rest refer to behaviors involving poor performance in school tasks and a lack of attention in class, the factor was labeled Learning Difficulties (LD). In the third factor, inverted items 6 (is sociable/interacts with classmates) and 2 (friendly) display the highest factor loadings, while the rest of the items describe students’ level of sociability in school; therefore, the factor was labeled Social Difficulties (SD). In the fourth factor, the items with the highest loadings are 20 (clings to the teacher) and 3 (demands too much attention from the teacher), while the rest refer to the need for social attention and the desire to bond with others. Considering the above, the factor was entitled Autonomy Difficulties (AD).
To calculate the index of similarity between the factors by sex and grade, EFAs were carried out for each subgroups. In all cases, four factors were found after completing the same procedures described above. Afterward, the factor loadings were used to calculate the congruence coefficient (CC) according to the recommendations laid out by
Goodness of fit indicators were calculated for the four-factor model extracted from the EFA of the 31 items included in TOCA-RR. The model considered the grid of orthogonal latent constructs connected by covariance relations. The indicators were found to be only marginally adequate considering the recommendations reported in the international literature (
A review of the standardized regression loadings revealed that three items (16, 31, 27) were below the general range. With respect to item 31, the results of the EFA revealed its dissimilar behavior; therefore, its elimination was coherent and well supported by evidence. With respect to item 16 (is confident), the CFA appears to confirm the theoretical intuition that being sure of oneself belongs to another construct (e.g., self-esteem), despite being linked to the learning difficulties factor. Therefore, it was also eliminated. Similarly, item 27 (needs affection to feel motivated to work) was eliminated due to its low theoretical concordance with the emotional dependence factor. The assessment of the new 28-item model showed that the indicators improved in all the pre-established criteria. Afterward, the fit indicators of the model for grade 1-4 students proposed by
All dimensions displayed high reliability. For the total scale (28 items), Cronbach’s alpha reached .914 for BD, .948 for LD, .892 for SD, and .813 for AD. The AVE was .50 for BD, .68 for LD, .62 for SD, and .52 for AD. All values were greater than or equal to .50, which indicates that the construct explains more than half of the variance of all the indicators that compose it (
Descriptive data are shown in
BD = behavioral difficulties; LD = learning difficulties; SD = social difficulties; AD = autonomy difficulties.
*p < .05
The aim of the present study was to assess the psychometric properties of TOCA-RR in adolescent students from Chile. Results show that the factor structure of the instrument must be reconsidered for this population because it was found that the empirical structure of the instrument is composed of four factors, not six, as in the case of students from 1st to 4th grade.
The existence of four factors with 28 items and not six factors with 31 items can be explained by the fact that the instrument proposed by
The names of the four factors were chosen to facilitate their use and interpretation (Behavioral Difficulties, Learning Difficulties, Social Difficulties, and Autonomy Difficulties). Since TOCA-R was constructed upon the basis of teachers’ identification of school maladjustment, it has always been dependent on the precision of the behavioral taxonomy behind teacher observations (
The new names have significant theoretical-practical consequences for the use of the instrument in school mental health programs: in this field, it is common to confuse risk with the presence of mental health problems, which affects the type of interventions carried out. It is essential to regard school adaptation difficulties as risk factors for developing mental health issues in the future (
Reliability was high in all factors, which makes it advisable to use this instrument in screening processes. This result is fundamental: due to the importance of detection in school mental health programs and the subsequent decisions made based on this information, it is necessary to use instruments that determine what is being measured and that are consistent when applied to multiple students. Also, it is required to research the sensitivity and specificity of this instrument, considering psychosocial risks, academic performance, and school dropout to improve the precision of the classifications made and the risk estimated.
Concerning comparative analyses, even though significant differences were found between the subgroups assessed, the small effect sizes calculated suggest that said differences must be studied in more depth and interpreted cautiously (
The main limitation of the study is the lack of representation of students from higher socioeconomic levels. Even though the sample is large and comprises schools from several regions of the country, it only represents the reality of public and subsidized schools, and it does not allow the results to be generalized to other possible populations of interest. It may constitute a bias, especially when many parents prefer to make an economic effort and enroll their children in private schools, which are not always beneficiaries of the program used to construct the sample.
Another limitation that should be considered is the lack of knowledge that the teacher might have of his/her students. Although a teacher’s self-report is carried out in the middle of the school year, there is always the chance of not recognizing who is being evaluated.
Finally, the lack of concurrent validity analysis should also be considered as a limitation, which could be tested in future studies.
This study suggests that TOCA-RR has robust psychometric properties for the assessment of school adaptation in adolescents from 6th to 8th grade and that it constitutes a valid and reliable screening tool for identifying risks in this stage of development. Future Latin American studies should focus on analyzing its predictive power and risk trajectories in the development of mental health problems. In addition, given its growing popularity in many regions of the world, it is necessary to make an effort to explore common transcultural factors that can improve our understanding of adaptation in multiple school contexts and cultures.
We thank the Skills for Life Program and National School and Scholarship Assistance Council, for supporting this research.
Cite this article as: Leiva, L., Rojas-Andrade, R., Vargas, B., & Squicciarini, A. M. (2020). Identifying adjustment difficulties in chilean schools: Psychometric properties of the teacher observation of classroom.
Funding. This work was supported by the National Scientific and Technological Development Fund under the Grant Fondecyt Regular Project n° 1171634.