Defining and measuring the quality of Early Learning and Child Care: A literature review

Staff-to-child interactions

Numerous studies have suggested that children’s interactions with educators (such as, staff-to-child interaction) are a critical factor leading to better academic and social outcomes (Buhs et al., 2006; Downer et al., 2010; O’Connor & McCartney, 2007; Ponitz et al., 2009; Von Suchodoletz et al., 2017). Causal evidence has been compiled from intervention studies, and has demonstrated that educators who receive training and coaching on staff-to-child interactions provided children in their classrooms improvements in academic, social, and self-regulatory functioning (for example, Mashburn et al., 2010; Raver et al., 2011). Mashburn et al., (2010) implemented a teacher professional development program designed to improve the quality of staff-to-child interactions in pre-kindergarten classrooms. Their randomized control study examined the impacts of video exemplars and individualized consultation on effective staff-to-child interactions, and on language and literacy development of pre-kindergarten children. Children whose educators were assigned to receive the video exemplars and who participated in consultations on effective interaction (treatment group) made small but significant gains in receptive language skills compared to children whose educators were only assigned to the video exemplars condition (control group).

Longitudinal research also highlights the importance of ELCC staff-to-child interactions. In the NICHD Study of Early Child Care and Youth Development, children who experienced more responsive caregiving demonstrated higher cognitive skills and academic achievement, and fewer externalizing problems at age 15 (Vandell et al., 2010). In this study, the authors measured three aspects of child care from birth to 4½ years: the type of care; the quantity (hours per week); and quality of care. Quality was assessed by two half-day visits scheduled within a 2-week interval at 6 and 36 months, and one additional half-day visit at 54 months. Analysis was performed to test the direct associations between early child care experiences and adolescent outcomes at age 15. After controlling for several confounding factors, results indicated that child care quality showed weak but significant associations with children’s cognitive skills and academic achievement at age 15. Children who experienced higher quality care had significantly higher levels of achievement at age 15, with stronger association between staff-to-child interactions and achievement at moderately high levels of quality than at low or very low levels. These findings suggest the existence of small but long-lasting benefits to children’s academic developmental trajectory resulting from preschool ELCC staff-to-child interactions (Vandell et al., 2010).

The Teaching Through Interactions framework (Hamre et al., 2013) describes three dimensions of staff-to-child interactions that have received theoretical and empirical backing: emotional support; instructional support; and classroom organization (see Table 1 in Appendix for clear description of these dimensions and of their subcomponents). The dimension of emotional support is characterized as educators developing positive relationships with children and being sensitive to their needs. It has been shown that children gain prosocial and self-regulatory skills when educators provide consistent emotional support (Curby et al., 2013; Johnson et al., 2013; Williford et al., 2013). Instructional supports sustain higher order thinking skills, provide feedback, and provide the basis for cognitive development (Hamre et al., 2014; Howes et al., 2008). Organizational and managerial aspects of child care also help young children to develop self-regulatory and executive-functioning skills. Children are better able to control their behaviour and cognition in groups/classrooms when educators manage behaviour and attention proactively (Rimm-Kaufman et al., 2009; Downer et al., 2010). In support of these three dimensions, a recent meta-analysis found a consistent, positive association between the quality of staff-to-child interactions and children’s literacy and numeracy (Von Suchodoletz et al., 2017) when using a combined score of staff emotional, instructional and organisational support to children.

Staff-to-child interaction is a complex indicator of process quality and may vary under specific conditions such as a child’s preference, an educator’s knowledge and skills, and program activities and organizational features (Pianta et al., 2015; Booren et al., 2012; Phillips & Lowenstein, 2011). For example, one study examined the extent to which children’s preschool classroom interactions with educators, peers, and tasks varied across activity settings (such as, large group, free choice, meals, etc.) (Booren et al., 2012). In this study, the authors examined classroom activity settings in relation to children’s observed behaviour during interactions using the Classroom Assessment Score System (CLASS) (Pianta et al., 2008). Over two days, children were observed on eight occasions for an average of 80 minutes in total. Findings indicated that, on average, children’s interactions with educators were more positive during teacher-directed activities (such as, structured settings such as large group), compared to unstructured time such as free choice and recess. Children also engaged in fewer conflictual interactions with peers during teacher-directed activities than during free choice, recess, and routine/transitional periods. Nevertheless, children’s interactions with tasks (engagement and self-reliance) were more positive during child-focused activities such as free choice, compared to teacher-directed activities.

The nature of staff-to-child interactions also vary in child care settings according to the child's age, and the household income (Dowsett et al., 2008). Indeed, one study revealed that two year olds from lower income families had, overall, fewer interactions with adults, but also more frequent negative interactions than those from higher income families. At age three, children from lower income households had more frequent negative interactions with peers, even though their frequency of peer interactions was similar to those of children in higher income families. At age four and a half, there were no differences in negative interactions with adults or peers between children of lower income and children of higher income households, but the amount of peer play varied across income groups and across types of care. Children in home-based family child care whose families were in the highest income category had significantly higher levels of peer play in comparison to children in informal/relative care (such as, being cared of by the father, mother's partner, grandparent, or other relative).

Altogether, studies found weak but significant child benefits to positive staff-to-child interaction. Determining the quality level of interactions that children experience directly with the child care staff is, however, not an easy task, and might vary under several conditions. Some initial work done in this area suggests that children benefit most from programs that are above specific thresholds, and that variations in the quality of interactions below these thresholds are not systematically associated with expected outcomes for children (Burchinal et al., 2010; Burchinal et al., 2014; Weiland & Yoshikawa, 2013). However, the approaches to identifying these thresholds and the scales used to measure staff-to-child interaction vary from one study to another, making it difficult to conclude on a quality threshold/turning point. As a result, it is not yet clear whether specific thresholds exist for reliably measuring the impact of staff-to-child interactions on children outcomes. Research cannot yet pinpoint how robust process quality (for example, staff-to-child interactions) should be in order to promote positive child outcomes.

Development-focused curricula

Curricula are paramount in ensuring that children experience high-quality care that facilitates cognitive development and school readiness (Yoshikawa et al., 2013). Children having more opportunities to engage in age-appropriate activities show larger gains during the preschool years, and those gains are maintained into the school years (Sylva et al., 2012). Indeed, a development-appropriate curriculum provided with intensive exposure to language-based content and a structured, diverse set of learning activities is beneficial (Yoshikawa et al., 2013; Clements, 2007); especially for children from lower income families (see Reynolds & Temple, 2008 for a review). However, curricula vary widely in their design focus (Clements, 2007), and research provides little evidence on specific pedagogical practices that can be used to support children’s development in ELCC settings.

Most successful curricula are closely linked to an educator’s training and implementation of pedagogical practices. They are characterized by intensive professional development that often involves coaching: expert teachers provide feedback and support for in-classroom practice, either in person or in some cases through classroom teaching videos. Some curricula also incorporate assessments carried out at multiple points to inform educators of the child progress and provide children with individualize instruction.

Curricula also vary according to the age of the child. There is evidence that children in the first three years of life need routine care and a balanced curriculum of play and learning-based activities such as storybook reading. For older pre-schoolers (age 4 to 5  years), research underlines the benefits of a more academic oriented-curriculum, focussed on school-readiness skills such as letter and number knowledge, in order to be prepared for the type of learning tasks encountered in elementary school (Bus et al., 2012). It should be acknowledged, however, that experts in the field do not all agree on the content of the curriculum. There remain debates about which environments should provide academically focused, structured activities as opposed to play and unstructured time.

Thus, a development-focused curriculum combined with integrated professional development and monitoring of children’s progress seems to best support child developmental outcomes during the preschool years (Dickinson, 2011; Yoshikawa et al., 2013). However, more evidence is needed about the effectiveness of such curricula. Most studies about curricula implementation involve the extensive support of the developer (lack of independent studies), or do not go beyond initial demonstrations of efficacy (Weiland & Yoshikawa, 2013). Further, they usually involve relatively small numbers of children, limiting generalization of the results.

Group composition

There is considerable evidence that a more favourable child-staff ratio (fewer children per practitioner in a group) and group size (number of children per group) provide conditions that promote optimal child development (Bradley & Vandell, 2007; Dalli et al., 2011; Huntsman, 2008; Phillips & Lowenstein, 2011; Karoly et al., 2005; Japel et al., 2005; Drouin et al., 2004). It has been hypothesized that the impact of group size and child-staff ratio leads to better child outcomes through more individualized attention and learning opportunities, such as,  through staff-to-child interaction  (Howes et al., 1992; Kontos et al., 1994). However, there is no clear evidence of a direct link between ELCC group composition and children development (Perlman et al., 2017).

Perlman et al. (2017) conducted a recent literature review and meta-analysis on child outcomes and child-staff ratios in ELCC settings. The authors selected studies targeting preschool-age children (30 to 72 months) attending child care centers, preschool programs, nursery schools, pre-kindergarten programs, and Head Start programs. The review excluded studies that only examined home-based family child care and those which did not distinguish home-based from center-based child care. Looking initially at 6,988 studies, only 29 met the inclusion criteria for systematic review: child care type, age of children served, child outcomes (cognitive, academic, social-emotional, health, or motor outcomes), study design (cross-sectional and longitudinal), and outcome reporting. Studies using identical child outcome measures and identical operationalization of ratios could be meta-analyzed from the pool of studies selected for the systematic review. Results from the meta-analysis, which included only three studies exploring association between ratios and children’s receptive language, were not statistically significant. Similarly, results from the systematic review revealed few statistically significant relationships between child-staff ratios and child outcomes. Indeed, only two studies reported statistically significant relationships between child-staff ratios and cognitive child outcomes. Two studies reported significant results for math outcomes but with a contradictory effect (Colwell et al., 2013; Anders et al., 2012). Most studies showed non-statistically significant associations between child-staff ratios and child language outcomes; again, with some showing contradictory effects. Few studies revealed positive behaviour outcomes or fewer child behaviour problems in classrooms with smaller number of children per staff, and none of the studies reported a significant association between child-staff ratios with physical outcomes (see Perlman et al., 2017 for a review). 

Results from Perlman et al. (2017) highlight the heterogeneity of previous studies, and the lack of in-depth understanding of which mechanisms lead to better child outcomes. The overall picture revealed by the available literature is that few, if any, relationships exist between child-staff ratios in ELCC programs and children’s developmental outcomes. Substantial heterogeneity in the assessment of ratios, outcomes measured, and statistics used to capture associations limited the quantitative synthesis. For instance, staff-child ratios can be defined in different ways (Mashburn et al., 2008; Mashburn et al., 2009; Sabol et al., 2013; Anders et al., 2012; Huntmans, 2008), and this has to be taken into account when interpreting research findings. Some studies rely on staff reporting, while some use the total number of child care places available divided by the number of full-time equivalent staff employed. Others divide child care places by the number of staff at work at any given time. The latter is the preferred method when staffs work a shift system. A third and more accurate way of measuring ratios is to use the number of staff and children actually observed in the same area over a given period of time. In light of this heterogeneity in measuring child-staff ratios, and lack of results consistency, research cannot provide a sound empirical basis for recommending universally appropriate group sizes or optimal child-staff ratios (Expert Advisory Panel on Quality Early Childhood Education and Care, 2009). At best, it can only specify different upper and lower limits appropriate under a range of different conditions (Munton et al., 2002).

Space and physical environment standards

To date, the quality of child care space and infrastructure has been considered a main factor in promoting child well-being and development (Evans, 2006). It is argued that children should have spaces big enough for their needs, and that resources should be accessible so as to stimulate engagement and play in the learning environment (Evans, 2006). There is, however, limited direct evidence of links between the quality of the physical environment and children’s learning and achievement.

One study found that children in settings with more stimulating, varied and well organised learning materials had higher scores on language comprehension and short-term memory tests at the age of 4½ years old (NICHD Early Child Care Research Network, 2003), but no association with other language and cognitive measures (letter-word identification, problem solving) were found. It is also not clear whether children`s language and cognitive improvement resulted directly from the quality of the materials or whether it was mediated through other quality aspects such as group composition, or again, the staff-to-child interactions.

The need to take children’s age into consideration when conducting studies might be one issue limiting the exploration of impact of ELCC physical characteristics on child development. Similarly to development-focused curricula, it is argued that appropriate environments for children should match each stage of development. For example, providing attention to the children might be more significant for the very young age group rather than providing them with educational tools (Trevarthen et al., 2003); and for pre-schoolers who begin to use objects in more complex situations, materials that offer opportunities for more advanced learning may be increasingly important.

Staff qualification

With regard to the question of what professional training is needed for quality child care and the observation of concrete improvement in child development, once again findings are mixed. There is some consensus that to ensure high-quality ELCC, early childhood caregivers/educators should be trained at the bachelor’s degree level and have credentials in courses that are specific to early childhood (Kelley & Camili, 2007; Melhuish et al., 2015; Dalli et al., 2011; Howes & Brown, 2000; Munton et al., 2002). For example, there is evidence that having specialised training in early childhood education is associated to higher process quality (Bigras et al., 2010). However, evidence supporting the effect of caregivers/educators qualifications on child outcomes is not as conclusive as this consensus would suggest (Philips et al., 2009; Early et al., 2007; Kontos et al., 1996). A meta-analysis conducted by Kelley and Camilli (2007) revealed that educators with a bachelor’s degree had small but statistically significant greater impacts on child outcomes in center-based ELCC than educators with less education. On the other hand, another recently published systematic review and meta-analysis revealed no significant association between staff’s educational pre-service qualifications and children’s math outcomes, and very weak significant associations with children’s language outcomes (vocabulary and letter/word identification; Falenchuk et al., 2017).

Research in this area is observational in nature and subject to the inherent biases of that research design. Results from Falenchuk et al. (2017) were hampered by heterogeneity in how staff education was defined, as well as the child outcomes that were assessed. There are multiple approaches to the measurement of staff education, including years of education, scales based on completed degrees, and definitions of thresholds or levels of education (for example, BA/No-BA or whether or not they adhere to local quality standards for staff education). This type of inconsistency in past findings makes it difficult to extract patterns of results from the literature on staff education.

There is also complexity in disentangling the educators training, as well as the effects of their workplace environment (for example, cooperation among the staff, educator’s satisfaction with work, and educator’s perception of work-related stress) on their teaching practice (Munton et al., 2002; Whitebook et al., 2009). Many child care centres also provide additional on-the-job training and supervision, especially for educators with lower non-specific training which may have contributed to the pattern of mixed results as to whether pre-service qualification has unique contribution to high-quality ELCC, and to children’s developmental trajectories. Moreover, it is likely that educators’ qualification and training interact with other variables such as wages, and child-staff ratios. For instance, studies have shown that years of work experience do not predict quality of staff-to-child interactions, as more experienced staff may be faced with a larger child-staff ratio (Pianta et al., 2005) or group of children (Connor et al., 2005). Research, therefore, has to simultaneously consider these important contextual issues to more accurately estimate the effect of educators training.

Setting aside these difficulties, child care research that has considered relationships between staff qualification and training, and observed programme quality, concludes that both qualifications and training affect the ability of staff to provide sensitive, responsive, and stimulating care and education (Dalli et al., 2011; Howes & Brown, 2000; Munton et al., 2002). Coaching or mentoring that provides support to educators on how to implement content-rich and engaging curricula shows some promise in helping to ensure that such instruction is being provided. Coaching or mentoring involves modeling positive instructional approaches and providing feedback on the educator’s implementation in a way that sets goals but is also supportive.

Working conditions

Research points out that the ability of staff to attend the needs of children is influenced not only by their level of education and training but also by external factors, such as their work environment, salary and work benefits (Shonkoff & Philips, 2000). In ELCC settings these are often referred to as wages and working hours, as well as characteristics such as non-financial benefits, team-work, manager’s leadership, and workload. These all can affect staff`s ability to positively interact with children.

Indeed, good working conditions have been associated with process quality. The You bet I care! dataset provides data on 325 classrooms in 224 center-based child care across Canada, including onsite observations of quality and considerable information about staff and leadership. The survey uses two measures of staff-to-child interaction quality, the ITERS (Infant-Toddler Environment Rating Scale) and ECERS-R (Early Childhood Environment Rating Scale – Revised) scores. Results have shown that more positive staff-to-child interactions were associated with higher staff wages, better benefits, and more favorable staff-child ratios (Goelman et al., 2000).

Good working conditions have also been associated with higher job satisfaction (Flanagan et al. 2013), and lower staff turnover (Beach & Flanagan, 2007). It is hypothesized that wage levels affect whether adequate numbers of qualified and experienced staff will be attracted (recruitment) or stay in the field (retention) (Child Care Human Resources Sector Council, 2013). This in turn has an impact on the stability and quality of the staff-to-child interactions (Goelman et al., 2000). In support of this hypothesis, the most recent Canadian survey of centre-based child care staff and directors, You bet we still care!, revealed that, despite increase in provincial average median wages (using adjusted dollars), respondents still identified the issues of compensation as the least satisfying aspect of the work and one of the main reasons for which staff leave the field (Flanagan et al., 2013). Additional data has also shown that highly trained staff are more likely to leave their jobs if they earned lower wages, worked in a climate of frequent turnover of highly trained co-workers, and worked with a greater percentage of staff with no BA degree (Whitebook & Sakai, 2003).

Wages and working conditions have thus been considered as effective strategies for the recruitment and retention of qualified ELCC staff. Consistent with this strategy, higher levels of training have been associated with higher wages and higher rates of full-time employment in center-based child care (Beach, 2013; Beach & Costigliola, 2005; Cleveland & Hyatt, 2000), although this varied according to the type of child care arrangement. For instance, family child care providers face specific challenges associated with working alone for long hours with no breaks, as well as having very limited access to health benefits and pension plans (Cox, 2005).

It is also important to point out that more research is needed in this area. Available studies mostly focus on the effects of the working conditions on staff satisfaction and practices, rather than on child development. You bet I care! (Goelman et al., 2000) and You bet we still care! (Flanagan et al., 2013) are the only Canada-wide surveys measuring aspects of ELCC quality, such as, staff wages, working conditions, and staff-to-child interaction. The surveys focused, however, only on employers and staff working in full-time licensed centre-based ELCC serving 0 to 6 year olds, and without data collected on children outcomes, it is difficult to connect current workforce issues with the quality of ELCC and its concrete impact on children.

In sum, studies indicate mixed pattern of associations between structural indicators of quality and child outcomes; most of them reporting no significant direct relationship with emerging academic skills or more broad developmental outcomes. One possible explanation is that structural characteristics may be more indirectly related to child outcomes in that they provide the foundation for process quality, the primary mechanism for children’s development and learning (Pianta et al. 2005). Another explanation is that structural features are interrelated, and that only by looking at combinations of structural indicators can we better understand how structural features promote development and learning. Contextual factors also need to be taken into consideration to order to increase understanding of the mechanisms at play between structure, process and child development. Lastly, there may be optimal combinations or levels at which some of the most commonly utilized structural levers such as qualifications and ratios have an impact on process quality and children’s development, nevertheless research is still largely focused on estimating the unique benefits of each indicator on a one-to-one relationship.

Overall, structural and process indicators provide two different lenses when judging the quality of ELCC. Children benefit most when educators engage in stimulating interactions that support learning and are emotionally supportive. Development-focused curricula with specific learning priorities, such as language and literacy, math, or socio-emotional development, provide the most promising boost to children’s learning, especially when combined with professional development and the monitoring of children’s progress. Guidelines about group size, child-staff ratios, physical ELCC environment, and staff qualifications help to increase the likelihood of, but  do not assure, supportive and stimulating interactions, and direct benefits on child outcomes (See Appendix: Table 2 for a short summary of these results).

Environmental rating scales (ECERS-R – ITERS-R – FCCERS-R)

The environmental rating scales are designed to assess process quality in early childhood or school age care groups. The Infant/Toddler Environment Rating Scale – Revised Edition (ITERS-R; Harms et al., 2006) was developed for children from birth to 2½ years of age in centre-based settings, while the Early Childhood Environment Rating Scale – Revised Edition (ECERS-R) is meant for children between the ages of 2½ and 5 in center-based or preschool settings. These two scales both have seven subscales, with slightly different items (ITERS-R= 39 items and ECERS-R= 43 items). Each item that compose the seven subscales is scored on a Likert-point scale, ranging from (1) inadequate quality to (7) excellent quality. One subscale focuses on structural quality (space and furnishings) while the other six are process quality related targeting curriculum and staff-to-child interactions.

1. provision of space and furnishings, items including indoor space, furniture for routine care, room arrangement and space for privacy
2. personal care routines, which focus on teaching and practice of routines
3. language-reasoning, including formal and informal use of language, development of reasoning skills and communication
4. activities, such as, the provision and quality of activities including fine motor, art, music, dramatic play and math/number
5. interactions including items on discipline, supervision and facilitation of proper interactions with staff and among children
6. program structure focusing on the schedule, time for free play vs. group time
7. parents and staff which focus on interaction, cooperation and provisions for personal and professional needs

ITERS-R and ECERS-R have an overall good inter-rater reliability, with an agreement of 91.6% on all scores given by the raters and strong internal consistency (α = 0.93). However, previous factor analysis studies have not validated that factorial structure and reported fewer factors than the seven subscales of quality for both ITERS-R and ECERS-R. For example, one study found four factors on the ITERS-R relating to Materials/Activities, Safety/Organization, Language/Interactions, and Parents/Staff. Similarly, other studies have failed to confirm the expected seven-factor structure of the ECERS-R, reporting only two factors: Materials/Activities and Language/Interaction (Cassidy et al., 2005). Some authors also recommend caution when using the Space and Furnishing, and the Personal Care Routines subscales. They also suggest excluding item 32 from the Program Structure subscale, unless most center-based programs assessed include children with disabilities (Cryer et al., 2004a, 2004b).

The Family Child Care Environment Rating Scale – Revised Edition (FCCERS-R) is build to assess home-based family child care programs conducted in a provider’s home. Similar to those in ITERS-R and ECERS-R, FCCERS-R has seven subscales totalizing 38 items. However, little information on the FCCERS-R measurement properties are available. Again, some studies revealed fewer dimensions than the seven subscales developed for this instrument (Schaack et al., 2013), suggesting the possibility for an abridged versions and thus, time and cost savings for training raters on this scale.

Caregiver interaction scale

The Caregiver Interaction Scale (CIS) focus is on the nature of the interactions between the lead educator and the 36 to 60 month-old children in the room. The CIS measures four specific dimensions of educator behaviour (Arnett, 1986): sensitivity (10 items); harshness (8 items); detachment (4 items) and permissiveness (4 items). This scale illustrates educators strengths and the areas where they can improve. It can be used with children of various age groups, in both centre‐based and home-based child care settings.

This scale is known as a valid and reliable measure (Cronbach’s alpha of 0.90 and 0.91 for harshness and warmth/responsiveness dimensions, respectively; Layzer et al., 1993). However, more recent studies provide a different overview on the validity of this scale. Colwell et al. (2013) found a bifactor structure to the scale (instead of four dimensions), after applying a factor analysis and item response theory method. The authors also found the items to be skewed: most educators display positive interactions with children. This suggests that the CIS is not well suited to distinguish between educators with “highly” versus “moderately” positive interactions with children.

Classroom assessment scoring system

Classroom Assessment Scoring System - Toddler (CLASS-Toddler) is a measure of ELCC process quality designed for child care settings with children from ages 36 months to kindergarten (Pianta et al., 2008). CLASS-Toddler is a tool for analyzing the quality of staff-to-child interactions in the classroom across two domains: Emotional and Behavioral Support, and Engaged Support for Learning. The first domain  includes five dimensions: Positive Climate, Negative Climate, Teacher Sensitivity, Regard for Child Perspective, and Behaviour Guidance. These dimensions focus on the emotional connection between the educators and children. It includes educator’s responsiveness to children and awareness of children’s developmental and individual needs, the degree to which classroom activities and interactions reflect the interests of the children and encourage their autonomy, and the use of effective methods to prevent and redirect problem behaviour. The second domain, the Engaged Support for Learning, is assessed through two dimensions: Facilitation of Learning, and Development and Language Modeling. These dimensions consider teachers’ ability to facilitate classroom routines, materials, and activities. It also takes into account teachers’ ability to use language stimulation and facilitation techniques in supporting children’s language development.

All of these seven dimensions are rated on a scale ranging from 1 (low) to 7 (high). The dimensions are similar throughout child care and preschool settings (La Paro et al., 2011). However, the descriptions of each item are adapted to whether it is child care or preschool classrooms.

Observational record of the caregiving environment

Comparing the quality of different types of care is sometimes complicated by the fact that different instruments have been designed for each, such as the ECERS-R, ITERS-R and FCCERS-R. To enable comparisons between different types of care, the NICHD developed the Observational Record of the Caregiving Environment (ORCE; NICHD Early Child Care Research Network, 1996). The ORCE has been used in a wide variety of early childhood settings with children from birth to 60 months. This instrument has a strong focus on the quality of interactions between the child and the caregiver (or other adults present). It seeks to capture the interactions experienced individually by each one of the children in the classroom, instead of characterizing the overall quality of staff-to-child interaction in the classroom at large.

The ORCE has five versions (6, 15, 24, 36 and 54 months). Each version is divided into the same three domains: behaviour scales (21 items), the qualitative ratings (13 items), and the structural variables (7 items) (Burchinal et al., 2011; Vandell & Wolfe, 2000).  The behaviour scale provides a record of specific actions while the qualitative ratings assess the caregiver’s behaviour in relation to the child. Ratings are made on a 4-point scale for each item, and a total score is derived from averaging the scores of the three domains (Vandell & Wolfe, 2000). In terms of construct and predictive validity, the ORCE yields an expected positive association with structural variables such as the level of caregiver education and the child-staff ratio (NICHD Early Child Care Research Network, 1996, 2000). The ORCE score has also been associated with child outcomes such as social functioning in first grade (NICHD Early Child Care Research Network, 2003).

Observation vs. questionnaire

Several factors need to be taken into consideration while choosing an instrument for measuring the quality of ELCC. As observational assessments are more complex and time-intensive than using a checklist or questionnaire based on educator’s report, they require more time and training and are not always feasible for researchers and practitioners. Observation requires being physically present for data collection and coding while children experience child care. Instruments used for observational measurements also require a longer training period in order for observers to become familiar with the tools. The observer must be able to document, code or assign a score for each dimension, while maintaining objectivity. For this reason, observers trained in this type of assessment are required to have accurate knowledge of the instrument and its administration rules. The main advantages of observational scales is that they describe more clearly the experiences and interactions of children at the child care. This in turn predicts child outcomes with greater accuracy than interviews or checklists (Zaslow et al., 2006). However, observational assessment does not always produce reliable data (Raudenbush & Sadoff, 2008). One risk is that these measurements may not be representative of a typical day. Other risks include susceptibility to measurement error and observer bias (Zaslow et al., 2006; NICHD, 1996).

In the case of questionnaires, interviews, or self-reporting instruments, data is collected directly from questions answered by the child’s mother or the child care educator. This has several advantages in terms of ease of administration, speed, cost, and training requirements; but it does not prevent subject biases. For example, the mother may only have partial information about what is really happening while her child is in child care. Similarly, the educator might not be willing to share information that could reveal non-optimal care provided to the child.

In sum, several factors should be taken into account when selecting instruments to measure ELCC quality: the dimensions of quality to be measured (process vs. structural quality); the training, and administration time and cost; the age and characteristics of children; the context of assessment (center-based, or home-based; small or large groups); and the validity of the instrument. More importantly, whether it is observational assessment or another type of measurement, it should also be acknowledged that one instrument might not cover all indicators or dimensions of quality. For example, most observational instruments include both process and structural quality components, but do not usually include measurement of educator’s pre-qualification and training, or wages and working conditions. A thorough assessment of ELCC quality should rather use multiple-informant sources and combine several methods of evaluation to overcome these limitations.