Phonological Awareness Assessment Test (PACOF) for Pre-school Children: Evidence of Validity and Reliability

Introduction

Phonological awareness has been defined as the ability to discriminate, identify, and manipulate the units of oral language, such as phonemes, onset-rimes, syllables, or words (Bdeir et al., 2022; Herrera et al., 2021; Justi et al., 2021). Thus, it involves phonological sensitivity, but also metalinguistic abilities. Accordingly, research has frequently distinguished two levels of phonological processing: epiphonological and metaphonological (Gombert, 1992), the former being related to the implicit knowledge of the linguistic units, without intentional control, and the latter being related to a more explicit level of awareness of language units and involving their intentional control (Ecalle & Magnan, 2002). Moreover, there is a wide variety of phonological awareness tasks, including producing rhymes, segmenting larger units into smaller ones, or identifying, deleting, and blending the separate sounds of a word (Herrera et al., 2021). These last types of tasks can be performed for units positioned anywhere within the word (at the beginning, middle, or end of the word). Research has also indicated that syllable, intra-syllable, and phoneme awareness are distinguishable, yet intercorrelated, dimensions of phonological awareness (Defior & Serrano, 2011; Justi et al., 2021; Meira et al., 2019).

Phonological awareness has been identified as one the main components of emergent literacy. This is a designation for a set of skills, knowledge, and attitudes that have been identified as facilitators of reading and writing acquisition (Lonigan et al., 2000). Research in the last decades has consistently pointed out that phonological awareness is one of the emergent literacy components that is most reliable and robust in predicting reading and writing acquisition in alphabetic languages (Castro & Barrera, 2019; Critten et al., 2021; Defior & Serrano, 2011; Míguez-Álvarez et al., 2022). A good level of phonological awareness assists in learning to read, facilitating the understanding of the alphabetic principle and the acquisition of decoding skills (e.g., Burke et al., 2009; Carson et al., 2014; Hogan et al., 2005; Justi & Roazzi, 2012). Research has also shown that phonological awareness is frequently and consistently impaired in children with reading difficulties compared to normal readers (Deuschle & Cechella, 2009; Moura et al., 2015). Accordingly, the assessment of phonological awareness in pre-school years (or at the beginning of primary school) has been considered a cornerstone for both preventing reading problems (Hogan et al., 2005; Lane et al., 2002) and guiding tailored interventions aimed at overcoming difficulties with acquiring this important skill (Capovilla et al., 2004; Hogan et al., 2005; Lane et al., 2002). Assessment with a validated instrument makes it possible to confidently define feasible objectives for both intervention and research (Godoy et al., 2014).

As described previously, phonological awareness is a complex construct: it comprises multiple dimensions, different levels of processing, different tasks, and different positions of the units to be identified/manipulated. Its assessment must consider all of these variables. In Portugal, there is a growing interest in the contribution of phonological awareness to learning of reading, but there are difficulties with assessing this construct in a holistic way. In an effort to overcome these difficulties, the Phonological Awareness Assessment Test [Prova de Avaliação de Consciência Fonológica – PACOF] was developed for use with children between the ages of 5 and 6 years.

The PACOF contains three sub-scales: Syllable Awareness, Intra-syllable Awareness, and Phoneme Awareness. Syllable Awareness is comprised of ten tasks: Syllable Segmentation, Syllable Synthesis, Initial Syllable Identification, Initial Syllable Word Association, Initial Syllable Naming, Final Syllable Identification, Final Syllable Word Association, Final Syllable Naming, Syllable Recognition and Syllable Location. Intra-syllable Awareness is comprised of two tasks: Rhyming Word Recognition and Rhyming Word Association. Phoneme Awareness is comprised of six tasks: Initial Phoneme Recognition, Initial Phoneme Identification, Initial Phoneme Word Association, Initial Phoneme Naming, Phoneme Recognition and Phoneme Location. Eight key criteria were considered in the construction of the PACOF: i) the type of cognitive task; ii) the representation level of the phonological unit; iii) the nature of the stimulus (type, frequency, form of presentation, presence/absence of visual support and imageability of the stimulus); iv) the acoustic properties of the target segment; v) the syllable structure; vi) the location of the target segment within the word; vii) the length of the word; and viii) the articulatory complexity.

The PACOF is a fully computer-based test that uses software specifically developed for this purpose, which makes it possible to maintain constant conditions for the administration of the stimuli and instructions. It also allows responses, response times, and scores to be recorded; the results are automatically generated and presented in a final report.

This article presents two studies designed to validate this test in a Portuguese sample. In the first study, evidence of concurrent validity and reliability was collected, while evidence of predictive validity was collected in the second study .

Study 1 – Evidence of Concurrent Validity and Reliability

This study had two objectives: (1) to collect evidence of concurrent validity by analysing the relationship between the scores obtained on the PACOF and on a different test assessing the same construct and (2) to assess the accuracy of the scores obtained on the PACOF using the test-retest method.

Method

Participants

A convenience sample comprising 30 children between the ages of 5 and 6 years enrolled in pre-school education was used. The criteria for inclusion in the sample were as follows: i) absence of hearing and developmental disorders and ii) having European Portuguese as a native language. The average age of the study sample ranged from 5 years (60 months) to 6 years and 11 months (71 months). The children in the sample were equally distributed between genders (15 girls and 15 boys). Of the 30 children participating in this study, 17 attended pre-school institutions in a rural setting, and 13 attended pre-schools in an urban setting of the Minho region, located in the north of Portugal. At the time of the study, approximately 67% of the children had already completed three years of pre-school education.

Regarding education level, approximately half of all the mothers and fathers had completed the first 6 years of basic education or less. Regarding parents’ professions, most of the parents (63% of the fathers and 53% of the mothers) were classified as “labourers, artisans, and similar workers”. The socio-economic level of the participants was indicated by pre-school teachers, who classified each child as being from low, medium, or high socio-economic level, based on their knowledge of the families’ socioeconomic situation, considering, for example, whether families received social support from the government or children received free or reduced-price meals at school. In this sample, 77% of the children were classified as belonging to a family context with a medium socio-economic level.

Measures

Phonological Awareness Assessment Test (PACOF). The PACOF comprises 141 items that are distributed among 18 tasks, which are grouped into three sub-scales: Syllable Awareness, Intra-syllable Awareness, and Phoneme Awareness. It is a computer-based test that is administered individually with no time limit. All the materials, including the instructions, are presented to the examinee through the software developed for the assessment. Similarly, all the responses of the assessed children are recorded and stored by the application. Each of the test’s tasks includes two sample items to familiarise the child with the proposed task and ensure that the instructions were understood. Figure 1 depicts some examples of the application.

The Syllable Awareness sub-scale comprises 10 tasks:

• Syllable Segmentation (9 items): the child hears a word and then must segment it into syllables. To do this, the child colours in a small square for each syllable of the word or can choose to select the corresponding numeral.
• Syllable Synthesis (4 items): the child hears each of the syllables that make up a word and then must produce the word obtained after the syllables are combined.
• Initial Syllable Identification (9 items): the child hears a stimulus syllable and the names of four images/words and then must select the image that begins with the stimulus syllable.
• Initial Syllable Association (9 items): the child hears the names of a stimulus image and four more images and then must select the image that begins with the same syllable as the stimulus image. Initial Syllable Naming (9 items): the child hears the name of an image and must identify and name the initial syllable of the word. Final Syllable Identification (9 items): the child hears the stimulus syllable and the names of four images and then must click on the image whose name ends with the stimulus syllable.
• Final Syllable Association (10 items): the child hears the name of a stimulus image and four more images and then must click on the image whose name ends with the same syllable as the stimulus image.
• Final Syllable Naming (10 items): the child hears a word and then must identify and name the final syllable of the word/image.
• Syllable Recognition (7 items): the child hears the stimulus syllable and the names of two images and then must select the image whose name contains the stimulus syllable. The syllable stimulus may be at the beginning or end of one of the two words.
• Syllable Location (7 items): the child hears a stimulus syllable and the name of an image. Beneath the image are three squares, which correspond to the three syllables of the word heard. The child must mark the first, second or third square according to the position of the target syllable of the word (beginning, middle or end).

The Intra-Syllable Awareness sub-scale comprises two tasks:

• Rhyming Word Recognition (7 items): the child hears two words (corresponding to two images) and then must indicate whether the two words rhyme by clicking on the corresponding icon (correct if the two words rhyme and incorrect if the two words do not rhyme).
• - Rhyming Word Association (7 items): the child hears a stimulus word (associated with an image) and four more words (also associated with images) and then must select the image whose name rhymes with the stimulus word.

Finally, the Phoneme Awareness sub-scale comprises six tasks:

• Initial Phoneme Recognition (7 items): the child hears a stimulus sound and is then instructed to select which of two images contains the target heard.
• Initial Phoneme Identification (7 items): the child hears a stimulus sound and the names of four images and then must select the image that begins with the same phoneme.
• Initial Phoneme Association (9 items): the child hears the names of a word (associated with an image) and four more words (associated with images) and then must select the image that begins with the same sound as the stimulus word.
• Initial Phoneme Naming (10 items): the child hears the name of a word (associated with an image) and then must identify and name the initial phoneme of that word.
• Phoneme Recognition (6 items): the child hears a stimulus sound and the names of two images and then must select the image that contains the stimulus sound. This sound may be at the beginning, middle, or end of one of the two words.
• Phoneme Location (5 items): the child hears a phoneme and the name of an image. Beneath the image are three squares that correspond to the three syllables of the word heard. The child must mark the first, second, or third square according to the position of the phoneme in the stimulus word (beginning, middle, or end).

Each item is awarded 1 point when the answer is correct or 0 points when the answer is incorrect. The scores of the items in each sub-scale are added together to obtain a total score for each dimension. The scores of the three sub-scales can also be added together to obtain a total phonological awareness score.

The items were selected in a previous study (Meira, 2017) with a stratified national sample (including Continental Portugal and the Autonomous Regions of the Azores and Madeira) and comprised 257 children between the ages of 5 and 6 years who were assessed using a first version of the test with 225 items. Rasch model analyses were used to assess the difficulty parameters of the items, the estimates of the subjects and the fit levels of the test. The 141 items that were retained in the test demonstrated an adequate degree of difficulty for the target population and a good fit to the model, as well as high values (>.70) in the Rasch model’s reliability indicators.

Battery of Phonological Tests (BPF; Silva, 2008). The objective of the BPF is to evaluate phonological awareness in children between the ages of five and six years. Of the 6 tasks that make it up, only 3 were used – Initial Syllable Classification, Syllable Analysis, and Initial Phoneme Classification – as the other 3 tasks assess manipulation skills that are not included in the PACOF. In Initial Syllable and Initial Phoneme Classification, the child is asked to select two out of four words according to a syllable or phoneme criterion, respectively. In Syllable Analysis, the child is asked to segment words into syllables. For each item, 1 point is awarded for a correct answer, and 0 points are awarded for an incorrect response. Regarding the validation data for the Portuguese population, the item-total correlations of the Initial Phoneme Classification and Syllable Analysis tasks were greater than .10, and those of the Initial Syllable Classification task were greater than .12. Cronbach’s alpha values were high for all three tasks: Initial Syllable Classification, .77; Initial Phoneme Classification, .64; and Syllable Analysis, .70. The test manual does not make reference to evidence of validity.

Procedure

Ethical and regulatory procedures were followed, and authorisation was obtained from the Directorate-General for Education of the Portuguese Ministry of Education (No. 009300006). The consent of school directors, kindergarten teachers, and those responsible for the children’s education was also obtained. The PACOF was administered to each child at two different times, with an interval of one to two weeks between administrations. During the first administration of the PACOF, each child was also assessed with the BPF. The tests were applied individually according to the procedures outlined in the test manuals. Of the 30 subjects in the sample, 16 were tested first with the PACOF, followed by the BPF; for the remaining 14, the tests were administered in the reverse order.

Statistical Analysis

Although the BPF does not group its tasks into sub-scales, for the purposes of this study the results of the BPF Initial Syllable Classification and Syllable Analysis tasks were added in order to obtain a total score for Syllable Awareness, as measured by the BPF. The BPF Initial Phoneme Classification task was also considered as an indicator of Phoneme Awareness. First, descriptive statistics of the scores for each sub-scale were calculated, as were the total scores for each of the two instruments. To obtain evidence of concurrent validity, the results obtained on the BPF and the PACOF were correlated using the Pearson correlation coefficient. Regarding reliability, the temporal stability of the PACOF test-retest results was analysed by calculating the intraclass correlation coefficient (ICC). After verifying the normal distribution of the results, a set of t-tests for paired samples was performed to verify whether the scores improved between the first and second application of the PACOF.

Results and Discussion

Table 1 presents the descriptive statistics of the scores obtained on the PACOF and the BPF. The skewness and kurtosis values were low; thus, parametric tests were performed to test the correlation between the scores on the different scales and sub-scales.

Table 2 presents the Pearson correlations between the PACOF sub-scales and the BPF sub-scales for the analysis of concurrent validity.

The three PACOF sub-scales – Syllable, Intra-syllable, and Phoneme Awareness – were significantly correlated with one another and with at least one sub-scale of the BPF. A more detailed analysis showed that the PACOF Syllable and Intra-syllable Awareness sub-scales had significant correlations with the BPF Syllable Awareness sub-scale. However, the PACOF Phoneme Awareness sub-scale did not have significant correlations with the BPF Phoneme Awareness sub-scale. This result seems to be due to the fact that the latter presents low mean results with little dispersion and contains only one task (which is distinct from the tasks included on the PACOF). The correlation between the total scores on the BPF and the PACOF was .56 (p = .001), which reflects a high association between the two instruments. Table 3 presents the ICC coefficients of the test-retest results for the PACOF.

The ICC was significant for all the variables. The ICC values for the different sub-scales ranged from .75 to .89. For the PACOF total score, the ICC value was .92. The values indicate that the total PACOF scores and the sub-scale scores at both assessment points showed high stability over time (ICC = .75), supporting the reliability of the scores.

Study 2 – Evidence of Predictive Validity

Study 2 sought to assess whether the PACOF results obtained at the end of the pre-school year were predictors of subsequent school performance. Accordingly, a sub-sample of children was followed, and the test results were correlated with their grades in the subject of Portuguese over the three school terms of their first year of primary school. The grades obtained during the three school terms were considered, as was as the grade at the end of the year.

Method

Participants

This study used a convenience sample comprising 52 children (26 children from an urban area and 26 children from a rural environment). The criteria for inclusion in the sample were similar to those of the previous study. The average age was 70.44 months (SD = 3.99), corresponding to approximately 5 years and 10 months. The minimum age was 63 months, and the maximum age was 77 months. The sample comprised of 40.4% girls and 59.6% boys. Regarding the distribution of participants by pre-school setting, 38 of the children attended public pre-school institutions (73%) and 14 attended private institutions (27%). Regarding parental education level, the 2^nd cycle of basic education (i.e., six years of education) was the most frequent academic level achieved among the parents of the children in the sample (38% of the fathers and 35% of the mothers). Regarding parents’ professions, approximately half of the fathers and mothers (52% and 50%, respectively) in this study had jobs in the category of “labourers, artisans, and similar workers”. Regarding socio-economic level, the information collected from pre-school teachers indicated that 87% of the children belonged to a family context with a medium socio-economic level.

Measures

In the final year of pre-school education, the PACOF (described above) was administered. At the end of the first year of primary school, the grades obtained in Portuguese for the three school terms, as well as the final grade, were collected from the children’s teachers. These grades are described on a 4-point ordinal scale: insufficient (0), sufficient (1), good (2), and very good (3).

Procedure

The PACOF was applied at the pre-school institutions that the children attended according to the procedures indicated in Study 1. The school grades were collected from individual assessment reports, which are prepared by teachers at the end of each school term.

Statistical Analysis

First, descriptive statistics and correlation coefficients between sub-scales were calculated. Ordinal regression analyses were performed using the PLUM procedure with a logit link function. Children´s grades in Portuguese in the first year of primary school (first term, second term, third term, and final grades) were considered dependent variables and the PACOF results (total and Syllable, Intra-syllable, and Phoneme Awareness sub-scale scores) were considered predictors.

Results and Discussion

The descriptive statistics presented in Table 4 verify that the PACOF total and sub-scale scores presented dispersion, indicating that the scores captured inter-individual variability. There were also significant correlations between the PACOF total and sub-scale scores; the Syllable Awareness and Phoneme Awareness sub-scales; the syllable Awareness and Intra-syllable Awareness sub-scales; the total PACOF score, the sub-scale scores, the second- and third-term grades and the final grade in Portuguese in the first year of primary school; and the PACOF total score, the Syllable and Phoneme Awareness sub-scale scores and the final grade in Portuguese in the first year of primary school.

Table 5 presents the results of the ordinal logistic regression models for the total score and the scores for each PACOF sub-scale. The results indicate that both the total score and the Syllable and Phoneme Awareness sub-scale scores were statistically significant predictors of the grades in Portuguese at the end of the three school terms and at the end of the school year. Regarding the score on the Intra-syllable Awareness sub-scale, the model also predicted performance in Portuguese in the second and third terms and the grades at the end of the year. However, it did not predict performance in Portuguese at the end of the first term. The percentages of variance explained by the Intra-syllable Awareness sub-scale scores were also slightly lower than those explained by the other dimensions.

General Discussion and Conclusions

Previous studies have reported that different phonological awareness tasks produce different performances (e.g., Anthony & Francis, 2005; Capovilla & Capovilla, 2000; Denton et al., 2000; Lefebvre et al., 2008; Liberman et al., 1974). A phonological awareness assessment instrument should have multiple dimensions with different tasks and different levels of complexity, as well as a careful selection of items (Runge & Watkins, 2006; Trehearne et al., 2003). In addition to theoretical representativeness, the psychometric quality of the measurement instrument is extremely important as there are important decisions to be made based on the results obtained. Accordingly, the objective of this study was to gather evidence of the concurrent and predictive validity, as well as reliability indicators, of a computer-based phonological awareness assessment test, the PACOF, designed to assess children enrolled in pre-school. Two studies were conducted to achieve this objective.

In the first study, evidence of concurrent validity was collected: the correlations between the PACOF results and the results of a different test assessing the same construct were significant and of moderate magnitude. The ICC coefficient calculation provided evidence that the scores have good stability.

The results of the second study made it possible to conclude that phonological awareness – evaluated through the PACOF – is predictive of subsequent performance in Portuguese in the first year of basic education, an area that broadly encompasses the early learning of reading and writing. The power of phonological awareness to predict school performance in the early years of education was also verified in other investigations carried out for Portuguese (e.g., Cadime et al., 2009; Cruz et al., 2014; Figueira & Botelho, 2017; Justi & Roazzi, 2012) and other languages (e.g., Carson et al., 2014; Hogan et al., 2005; Hulme et al., 2002; McBride-Chang et al., 2004; Muter et al., 1998). The empirical evidence of this study therefore corroborates the evidence obtained in other studies regarding the facilitating role of phonological awareness in literacy (e.g., Ball, 1993b; Barrera & Maluf, 2003; Capovilla et al., 2004; Torgesen et al., 1994; Wagner et al., 1997).

The empirical results obtained thus indicate that the PACOF seems a valid and reliable instrument for assessing phonological awareness in the age group for which it was constructed. Its psychometric properties show that it is adequate for i) understanding children’s strengths and weaknesses and, consequently, providing clear indicators not only for the design of preventive interventions but also for the teaching of reading; ii) monitoring interventions; and iii) assisting in the allocation of educational resources.

Despite the qualities of the PACOF as an instrument for assessing phonological awareness, the studies that were performed have limitations. The main limitation in related to the characteristics of the samples: in both studies the sample size is low, and none of the samples is representative of the Portuguese population. Future studies should collect additional evidence of validity using larger and more representative samples. Another limitation is the fact that, in the second study, classroom grades were considered rather than the results of a reading skills test. Previous research has shown that classroom grades usually have only moderate correlations with students’ performance in reading, since other parameters such as effort or motivation are also taken into account in the grades (Capellini et al., 2004; Milanowski, 2004; Pinheiro, 2001). Thus, in future studies, more evidence of predictive validity should be collected using standardized reading tests, instead of classroom grades. Future studies with children with dyslexia should also be conducted. According to the International Dyslexia Association, this particular learning disorder is considered to have a neurobiological origin and is characterised by difficulties that typically result from a phonological deficit (Anthony & Francis, 2005; Gottardo et al., 1994; Lee, 2008; Lyon et al., 2003; Ramus, 2001). This phonological deficit encompasses three different but correlated constructs: phonological memory, rapid naming, and phonological awareness (Whitehurst & Lonigan, 2002). Of the three elements of phonological processing, phonological awareness is most consistently affected in dyslexic children (Snowling, 2014). Further studies using the PACOF with children identified as at risk of dyslexia could contribute to its validation as an instrument for the early identification of this disorder.

The theoretical framework underlying the construction of the PACOF and the results of assessments using this instrument may also guide future programmes to promote phonological awareness. The software developed for the PACOF can also serve as a basis for the construction of a computerized intervention programme, as research has shown that a computer-based intervention generally achieves better results than more traditional interventions (Snape & Nicol, 2003; Trinh, 2011).