Assessing neighborhood disorder: Validation of a three-factor observational scale

Assessment of neighborhood disorder tipycally considers two types of disorder, physical and social ( Robinson, Lawton, Taylor, & Perkins, 2003; Skogan & Maxfield, 1981; Taylor & Shumaker, 1990 ). Physical disorder refer to urban landscapes with high levels of decay and deterioration. For example, abandoned houses, graffiti, trash on the streets, abandoned cars, used needles, and vacant lots would exemplify physical disorder ( Brunton-Smith, 2011; Garvin, Cannuscio & Branas, 2013; Robinson et al., 2003; Sampson & Raudenbush, 1999; Skogan, 1990; Taylor, 2001 , Toet & van Schaik, 2012 ). Some scholars, however, make a further distinction between physical disorder and physical decay: physical disorder would refer to features like dirt in the streets (litter, bottles, condoms), graffiti, abandoned cars, etc. (i.e., behavioral manifestations), whereas physical decay would refer to structural characteristics that can arise from lack of institutional investments and have long term effects, such as abandoned buildings, burn-out houses, badly deteriorated recreational facilities, etc. ( Sampson, 2009; Sampson & Raudenbush, 2004 ). As Sampson and Raudenbush (2004) argue, it is important to make this distinction because physical disorder is “limited to behavioral manifestations (e.g., graffiti, garbage in the streets) that can be conceptually decoupled from structural resources” (p. 326). Social disorder refer, on the other hand, to events in public places seen as potentially threatening, and can be exemplified by the presence of people taking drugs or alcohol in the street, drug dealing, fights and arguments, presence of homeless people, public drunkenness, street prostitution, high levels of police activity, and other criminal or not criminal activities that create a sense of danger ( Gracia, 2014; Gracia & Herrero, 2007; Robinson et al., 2003; Ross & Mirowsky, 2001; Sampson, 2009; Sampson & Raudenbush, 2004 ). Despite some studies suggesting that physical and social disorder may overlap, being order and disorder two ends of a single continuum ( Ross & Mirowsky, 1999; Xu, Fielder, & Flaming, 2005 ), most studies support the distinction between physical and social disorder ( Brunton-Smith, Sindall, & Tarling, 2010; LaGrange, Ferraro, & Supancic, 1992; Sampson & Raudenbush, 2004; Taylor & Shumaker, 1990 ).

In order to assess neighborhood disorder, researchers generally use three different approaches ( McDonell & Waters, 2011; Mooney et al., 2014 ). One approach, based on a more objective perspective, draws from neighborhood information from governmental or commercial data sources ( Cerdá et al., 2009; McDonell, 2007; Mooney et al., 2014 ). Although these data is freer from the variability and subjectivity of subjective perceptions of disorder ( Kubrin, 2008 ), however, this information is “often collected for administrative purposes, may not fully capture the construct of research interest, and may be collected at a spatial resolution that is not optimal for research purposes” ( Mooney et al., 2014 , p. 626-627). A second, and widely used, approach is based on resident's perceptions of their neighborhood physical or social characteristics. A number of limitations have been noted, however, regarding this approach, including “same source bias” (e.g., same source reporting perceived neighborhood disorder and related outcomes), confusion with other psychological constructs (e.g., fear of crime), or the influence of stereotypes and neighborhood prejudices (e.g., racial, ethnic, or socioeconomic composition) in perceptions of disorder ( Caughy, O’Campo, & Patterson, 2001; Duncan & Raudenbush, 1999; Gómez, Johnson, Selva, & Sallis, 2004; Mooney et al., 2014; Sampson, 2009; Sampson & Raudenbush, 1999, 2004; Schaefer-McDaniel, Caughy, O’ Campo, & Gearey, 2010 ). Finally, a third approach, that aims to overcome the above limitations, emphasizes the importance of using direct and systematic observations of neighborhood characteristics by trained researchers ( Franzini, Caughy, Nettles, & O’Campo, 2008; McDonell, 2007; McDonell & Waters, 2011; Neil, Parke, & McDowell, 2001; Raudenbush & Sampson, 1999; Reiss, 1971; Sampson & Raudenbush, 1999 ). This approach aims to obtain objective measures of neighborhood conditions, to capture a wide range of factors, which are not always available otherwise, and to allow its replication in other contexts ( Caughy et al., 2001; Cohen et al., 2000; Franzini et al., 2008; McDonell & Waters, 2011; Sampson & Raudenbush, 2004; Taylor, 2001 ).

This study aims to add to this body of research by validating an observational measure of neighborhood disorder in the context of a European city that may differ from the culture and structure of Anglo-Saxon cities where most of this type of measures have been developed and validated ( Le Galès & Zagrodzki, 2006; Summers, Cheshire, & Senn, 1999 ). The availability of a reliable and valid measure of neighborhood disorder in this context may be an important addition to the growing body of literature exploring neighborhood effects ( Sampson, 2012 ). To this end, independent observations by trained raters of neighborhood disorder will be conducted using small-areas of the city as the ecological units (i.e., census block groups, which are the smallest administrative sections of the city available). By using the smallest posible geographical units of the city we expect greater homogeneity of neighborhood characteristics ( Ocaña-Riola et al., 2008 ). Also, by using all census block groups of the city we will obtain greater variability and, as neighborhood characteristics tend to cluster in space, this will provide the possibility to explore the clustering of these characteristics for validation purposes ( Gracia et al., 2014).

This study presents data on the development and preliminary validation of an observational instrument to assess neighborhood disorder. The specific objectives of the study are: (1) to test the inter-rater reliability of the scale; (2) to test the factorial validity of the scale using confirmatory factor analysis: we expect a factorial structure reflecting three theoretically a priori factors, physical disorder, social disorder, and physical decay ( Sampson & Raudenbush, 2004 ); (3) to test the reliability of the scale by means of the composite reliability index; and (4) to test the criterion-related validity of the scale also employing structural equations. Drawing from social disorganization theory we expect associations between neighborhood disorder and three neighborhood structural characteristics, central in this theoretical perspective: neighborhood socioeconomic status, immigrant concentration, and residential instability ( Caughy et al., 2001; Jones, Pebley, & Sastry, 2011; Kubrin & Weitzer, 2003; McDonell & Waters, 2011; Mooney et al., 2014; Sampson et al., 1997; Sampson & Raudenbush, 1999 ). Also, for validation purposes, a spatial perspective will be applied. As we expect that disordered neighborhoods will cluster in space, rather than be randomly distributed in the city, spatial correlation analyses will be conducted to test whether neighborhood disorder shows a significant spatial pattern ( Bruinsma, Pauwels, Weerman, & Bernasco, 2013; Gracia et al., 2015; Quick, 2013; Veysey & Messner, 1999 ).

In this paper we describe the development and the psychometric properties of a preliminary validation of an observational scale assessing neighborhood disorder. This scale was implemented in the city of Valencia (Spain), using independent observations of all census block groups of the city conducted by trained raters. In general, results have shown that this scale is a psychometrically sound and valid instrument to assess three neighborhood disorder dimensions: physical disorder, social disorder, and physical decay.

Results regarding inter-rater reliability showed fair to substantial levels of agreement ( Landis & Koch, 1977 ), with stronger agreements for physical disorder and physical decay factors, and lower inter-rater reliability for social disorder. Although these results are slightly lower than in other studies (e.g., Caughy et al., 2001; Franzini et al., 2008; Jones et al., 2011 ), they are, however, comparable to others (e.g., Mooney et al., 2014; Wei et al., 2005 ). As to why social disorder showed lower levels of agreement, one possibility is that social disorder cues such as fights or public intoxication tend to be less stable over time (e.g., depending on the time of the day) than other physical features of neighborhoods, like those indicating physical decay and disorder, that are more temporally stable ( Jones et al., 2011).

With respect to the factor structure of the observational scale, our analyses aimed to validate a three theoretically-based neighborhood disorder dimensions ( Sampson & Raudenbush, 2004 ). To this end, several competing confirmatory factor analyses were first estimated and discarded, yielding a final and depurated version of the scale with good analytical fit. This final scale supported a three-factor model measuring physical disorder, social disorder, and physical decay, as theorized by Sampson and Raudenbush (2004).

This final three-factor model was obtained after removing some items for their low variability or poor consistency with their dimension. Although these items are usually present in other scales measuring neighborhood disorder, however, indicators such as abandoned cars, used condoms and syringes (physical disorder), prostitution (social disorder), or abandoned commercial buildings (physical decay) were discarded in the final model. The presence of these items was very low and, in the case of syringes, there was no presence at all in any of the census block groups observed. Although the time of the observation may have influenced the low presence of some of these indicators (e.g., prostitution), another posible explanation is that, given the characteristics of cities like Valencia, with high density in a relative small area ( Le Galès & Zagrodzki, 2006; Summers et al., 1999 ), some of these indicators could be more present in the outskirts of the city, and therefore outside of the boundaries where the observations were made. This also suggests that differences between the present scale and others developed elsewhere may reflect context-specific features of the cities. Given that these results may be context-dependent, we do not favour the uncritical use of the short-version of the originally proposed scale destilled for this particular study. On the contrary, careful theoretical considerations previous to the use of this scale in other cities should consider whether some (or all) of the items removed could have enough variability and importance as to be included in the instrument. Clearly, a subsequent depuration of the scale according to its psychometric properties is always possible. On the other hand, results for internal consistency of the final three-factor scale by confirmatory analyses also supported its reliability, with CRI values between .82 and .83 for the three scales.

Two different criterion-related validity tests of this observational measure of neighborhood disorder were conducted. For the first one, and drawing from social disorganization theory, we explored associations with three criteria tapping neighborhood characteristics central to this theoretical perspective: neighborhood socioeconomic status, immigrant concentration, and residential instability, masured at the census block group level. As expected, correlations between measures of neighborhood disorder and these structural characteristics of neighborhoods were mostly in the expected direction ( Kubrin & Weitzer, 2003; Sampson & Raudenbush, 1999; Sampson et al., 1997 ), although with stronger associations for physical decay. For example, high levels of physical decay were significantly associated with lower neighborhood socioeconomic status, higher rates of immigrant concentrarion, and higher residential instability. Social disorder was positively related to levels of immigrant concentration and residential instability, but its association with neighborhood socioeconomic status was not statistically significant. Physical disorder was also negatively related to neighborhood socioeconomic status; however, associations with immigrant concentration and residential instability did not reach significance. These results partly support previous research where significant associations between disorder and a number of neighborhood characteristics were also found, especially those regarding the relationship between physical disorder and decay and neighborhood socioeconomic indicators ( Caughy et al., 2001; Jones et al., 2011; McDonell & Waters, 2011; Mooney et al., 2014; Sampson & Raudenbush, 1999 ). Also, some of our results support studies that fail to find a significant relationship between physical disorder and residential instability ( Sampson & Raudenbush, 1999 ), or between physical disorder and immigrant concentration ( Jones et al., 2011 ). It is interesting to note that physical decay was associated with all neighborhood characteristics (socieconomic status, immigrant concentration, and residential instability), suggesting that differentiating between physical disorder and physical decay is an important theoretical distinction that may provide a more detailed analysis when assessing neighborhood disorder and exploring its relationships with different outcomes and processess ( Sampson, 2009; Sampson & Raudenbush, 2004 ).

As we expected that disordered neighborhoods would tend to cluster together in space ( Bruinsma et al., 2013; DiMaggio, 2015; Quick, 2013; Veysey & Messner, 1999 ), for the second criterion-related validity test we used a spatial analytical approach to assess the spatial distribution of the different types of neighborhood disorder across all census block groups observed. Results showed that the three types of disorder (physical disorder, social disorder, and physical decay) were spatially clustered, confirming that they were not randomly distributed in the city. This reflects the existence of different areas of the cities where neighborhood disorder tends to concentrate and shows that this neighborhood risk factor tend to cluster in space. These results support the idea that different manifestations of neighborhood disorder, as other characteristics of the cities, are not randomly distributed in space. As illustrated by a growing body of literature linking neighborhood disorder with a wide array of outcomes, including crime, violence, or health, the spatially patterned nature of this risk factor makes more likely that related outcomes will also be spatially patterned ( Cunradi, Mair, Ponicki, & Remer, 2011; Diez-Roux & Mair, 2010; Freisthler et al., 2007; Gracia et al., 2015; Kawachi & Berkman, 2003; Law, Quick, & Chan, 2014; O’Campo et al., 2015; Sampson, 2012 ).

Finally, this study has both strengths and limitations. Among the strenghts, the use of independent observations of neighborhood conditions, rather than residents’ subjective perceptions, allows to overcome some of the limitations noted in the literature regarding this later approach ( Caughy et al., 2001; Mooney et al., 2014; Sampson & Raudenbush, 1999; Schaefer-McDaniel et al., 2010 ). Relatedly, for the observation of neighborhood disorder at the level of ecological units (rather than the personal level), we also use a high-resolution approach. We used the smallest administrative units available (i.e., census block groups) that allow greater homogeneity and precision than lower resolutions such as census tracks or postal codes, commonly used in other neighborhood studies ( Beyer et al., 2015; Bursik, Grasmick, & Chamlin, 1990; Kaufman, Dole, Savitz, & Herring, 2003; Campo, Xue, Wang, & Caughy, 1997 ). By using small-area units, we also reduced potential ecological bias, as this resolution is closer to the individual level ( Gracia et al., 2015; Lawson, 2006 , Ocaña-Riola et al., 2008 ). We used census block groups in our study, which substantially reduced this potential bias. Finally, we used all census block groups of the city, rather than selecting only a sample of them, which provided greater variablity and the possibility to explore potential significant spatial patterns in the distribution of neighborhood disorder across the city ( Caughy et al., 2001; Mooney et al., 2014 ). In this regard, the use of spatial methods to complement the criterion-related validity of our observational scale is an important addition to the existing literature, as neighborhoods, from this perspective, are not treated as independent units ( Gorman, Gruenewald, & Waller, 2013; Mooney et al., 2014; Morenoff, Sampson, & Raudenbush, 2001 ). Although widely used in epidemiological studies ( Lawson, 2009 ), and despite its advantages, this methodological approach is still uncommon in neighborhood studies, and future research would clearly benefit from incorporating a spatial perspective ( Cunradi et al., 2011; Gracia et al., 2014, 2015; Law & Quick, 2013; Law et al., 2014; Sparks, 2011 ). As for limitations, as noted above, some measures of neighborhood disorder may have been affected by the time of the day they were observed. Our observations were limited to business hours, and the same places may have shown different characteristics at night ( Caughy et al., 2001 ). Future research would benefit from including different observations during the day, and revisiting the same areas during night hours. Also, other neighborhood indicators such as trash in the streets may be present only at specific moments, and repeated obervations of the same area would be advisable, although clearly more costly ( Wei et al., 2005 ). In this regard, recently new technologies, such as virtual environments, or Google Street View, provide powerful and easy accesible tools that may help to enhance neighborhood research ( Odgers et al., 2012; Toet & van Schaik, 2012 ).

In conclusion, this paper provides evidence of a reliable and valid observational measure to assess neighborhood disorder. Adequate measures to assess neighborhood characteristics are important research and intervention tools, as they are key to better understanding neighborhood proceses, as well as to evaluate related outcomes and monitor changes after grass-roots efforts or oficial initiatives to reduce neighborhood inequalities.

The authors of this article declare no conflict of interest.

This research was supported by the Spanish Ministerio de Economía y Competitividad (PSI2014-54561-P). Miriam Marco was supported by the FPU program of the Spanish Ministerio de Educación, Cultura y Deporte (FPU2013/00164).