Appendix: Healthy settings for young people in Canada – Methodology

Appendix: Methodology

The sample

The international HBSC data file requires samples of young people aged 11, 13, and 15 years. In order to achieve these age groups, the Canadian research team samples students in Grades 6 through 10. A systematic, single-stage cluster sample approach is used to select whole classes of students from across Canada to participate in the study. As a requirement of the international protocol, the sample is drawn to be self-weighting. Classes are identified using school directories and lists. The number of classes in specific schools is estimated based on the grades in the school, the numbers of teachers, the total enrollment, and enrollment by grade depending on the province. Classes are given an approximately equal chance of being selected and are ordered on the sample lists according to school jurisdiction, province, language of instruction, public/Roman Catholic designation, community size, and community location within a province. Classes are therefore proportionally distributed according to these characteristics. Private and special schools are not included in the sample. School boards are then approached for permission.

Participation in the survey is voluntary. Parental consents, approved by Queen’s University’s General Research Ethics Board, are required for students to participate in the survey. At the student level, 74% of the estimated students in the sample participated in 2005-06. Fewer than 10% of students declined to participate or spoiled their questionnaires intentionally. For the remainder, the most common reasons for student nonparticipation are attributed to: failure to return consent forms; failure to receive parental consent; or absence on the day of survey administration.

Data collection

The student questionnaires are administered to school classes by teachers and are designed to be filled out during one 40-minute class. While there are one or two openended questions, almost all of the questions can be answered by checking off a response option. Students are guaranteed anonymity by sealing their unsigned survey in an envelope and teachers are asked to closely follow a specific set of instructions regarding active consent and survey administration.

Data management

Completed questionnaires are returned to Queen’s University. Data are entered manually by data entry clerks into simple text files for incorporation into the statistics software package SPSS. Sample batches of questionnaires from each clerk are tested for accuracy, problem areas are addressed, and if errors are identified they are corrected. The data cleaning includes an analysis of out-of-range values and inconsistencies across related questions. Individual questionnaires with suspicious response patterns or that are clearly not filled out in a serious manner are considered for exclusion from the data set.

Sampling limitations and sources of bias

Despite the comprehensiveness of the HBSC survey and the use of a representative sample, there are some important limitations in relation to the sampling methodologies and to reliance on self-administered questionnaires.

Since the school class is the unit/cluster employed in the sampling procedure, it is possible that those who make up a cluster may have a lot in common in terms of behaviours and/or attitudes. For example, they may have access to the same cafeteria food or share similar views about a teacher or their school. On the other hand, relationships with parents or student patterns of headaches are less likely to be shared by classmates. This is called the sample design effect. The effect on the interpretation of results is that the design factor increases the size of confidence intervals for some items on the survey, relative to what it would be for a random sample of students. For most items on the survey, the design factor increases the confidence interval by less than 1% in either direction.

The most likely source of bias in the HBSC survey is due to the fact that participating students are required to provide parental consent, as well as giving consent themselves, in order to participate. It is reasonable to expect that those who do not return parental consent forms and those who decline may not share the characteristics that are prevalent among the participating students. For example, students who do not return parental consents may be less likely to have positive communication with their parents.

Survey administration limitations and sources of bias

Some students’ responses to survey questions may be influenced by the following factors:

• Communication between students during survey administration or lack of privacy with students sitting too close to each other.
• Student interpretation of the survey questions (i.e., how they understand the question when they read it; some of the questions are long and if read quickly might be misinterpreted).
• Student characteristics such as their cultural and religious beliefs that might influence how they respond to some sensitive questions.
• Limitations in students’ memory in recalling behaviours and events that happened in the past year.
• A desire to present oneself in a positive way, especially if students do not trust that their responses will be treated confidentially.
• A desire to promote a certain image that is acceptable to peers, since students may discuss the way they responded to some of the items.
• Teacher and student level of interest in completing the survey.
• Time of day when students in schools complete the survey (e.g., they could be tired and unable to concentrate at the end of day).

Reliability/validity

A survey is considered to have high reliability when students give the same responses to the survey questions on two separate occasions during a short time frame. It is not possible to assess reliability by having students complete the HBSC survey on two different occasions, because of the complicated logistics of repeating survey administration in schools across Canada. Reliability of the questions is verified by testing the relationships in a subset of questions after the surveys have been completed, to see how the responses to similar questions are answered and if they actually measure the same thing (see Composite scale measures later in this Appendix). Reliability is necessary, but not sufficient, to establish validity.

A survey is considered to have high validity when it accurately measures what it is supposed to measure. Validity is sometimes estimated by comparing survey results to another measure (e.g., comparing smoking self-reports to saliva nicotine levels; comparing selfreports of academic achievement to actual school grades). Validity can be also be estimated by determining whether certain questions assessed by the survey relate to other questions in predictable ways (e.g., testing whether a question about intelligence is related to grades in school, because intelligence is expected to relate to progress in school). Estimating the validity of some of the HBSC questions is possible by comparing our findings to other surveys in Canada. For example, in the 2001-02 survey, we found that there was a significant decline in the proportion of girls in Grade 10 who were smoking. Other studies in Canada found similar results, which gave more credibility to our findings.

Composite scale measures

A parent trust and communication scale was developed based on the following six survey items: How easy is it for you to talk to Father about things that really bother you?; How easy is it for you to talk to Mother about things that really bother you?; My parents understand me; I have a happy home life; My parents trust me; and What my parents think of me is important. For the first two items, the response options in the 2006 survey were: very easy=5, easy=4, difficult=3, very difficult=2, and don’t have or see this person= 1. For the last four items, the response scale was: strongly agree=5, agree=4, neither agree nor disagree=3, disagree=2, and strongly disagree=1. Only those students with complete data for the six items were included in this measure. The responses across the six items were summed and the resulting score was recoded into thirds (low, middle, and high) for presentation. The reliability of the scale as measured by Cronbach’s alpha is .80.

An attitude towards school scale was developed based on the following nine survey items: The rules in this school are fair; Our school is a nice place to be; I feel I belong at this school; I am encouraged to express my own views in my class(es); Our teachers treat us fairly; When I need extra help, I can get it; My teachers are interested in me as a person; Most of my teachers are friendly; and How do you feel about school at present? For the first eight items, the response scale was: strongly agree=1, agree=2, neither agree nor disagree=3, disagree=4, and strongly disagree=5. For the ninth item, the response scale was: I like it a lot=1, I like it a bit=2, I don’t like it very much=4, and I don’t like it at all=5 (note: although originally on a scale of 1 to 4, this item was converted to a 1 to 5 scale with no middle category to give it the approximate weight of the other variables). The responses were reversed for positively worded items. Only those students with complete data for the nine items were included for this measure. The responses across the nine items were then summed and the resulting score was recoded into thirds (low, middle, and high) for presentation. The reliability of the scale as measured by Cronbach’s alpha is .87.

A communication with friends scale was developed based on the following three items: How easy is it for you to talk to friends of the same sex about things that really bother you?; How easy is it for you to talk to friends of the opposite sex about things that really bother you?; and How easy is it for you to talk to your best friend about things that really bother you? The response options for the three items were: very easy=5, easy=4, difficult= 3, very difficult=2, and don’t have or see this person=1. Only those students with complete data for these three items were included in the measure. The responses across the three items were summed and the resulting score was recoded into thirds (low, middle, and high) for presentation. The reliability of the scale as measured by Cronbach’s alpha is .74.

A pro-social attitudes of friends scale was developed for Grade 9 and 10 students only, based on three survey questions: My friends like school; My friends think good marks are important; and My friends get along with their parents. The response options for the three items were: none=1, a few=2, some=3, most=4, and all=5. Only those students with complete data for the three items were included in the measure. The responses across the three items were summed and the resulting score was recoded into thirds (low, middle, and high) for presentation. The reliability of the scale as measured by Cronbach’s alpha is .68.

Four measures were used in the development of a family affluence scale: Does your family own a car, van or truck?; Do you have your own bedroom for yourself?; During the past 12 months, how many times did you travel away on holiday (vacation) with your family?; and How many computers does your family own? The items were recoded as follows: having one’s own bedroom – no=0 and yes=1; family ownership of a car – zero=0, one=1, and two or more=2; going on holiday/vacation (in the past 12 months) – not at all=0, once=1, twice or more=2; and family ownership of a computer – none=0, one=1, and two or more=2. Only those students with complete data for the four items were included in the measure. The responses across the four items were summed and the resulting score was recoded into low, moderate, and high. The three groups are calculated according to the HBSC international protocol and are not thirds.

A psychosomatic symptoms scale was developed based on the following eight survey items: In the last 6 months, how often have you had the following: headache, stomach ache, backache, feeling low (depressed), irritability or bad temper, feeling nervous, difficulties in getting to sleep, and feeling dizzy. The response scale for each item was: about every day=1, more than once a week=2, about every week=3, about every month=4, and rarely or never=5. Only those students with complete data for the eight items were included for this measure. The psychosomatic symptoms scale is a count of the number out of the eight items on which a student responded about every day, more than once a week, or about every week.

An emotional well-being scale was developed based on the following nine survey items: I have trouble making decisions; I am often sorry for the things I do; I have confidence in myself; I often wish I were someone else; I often feel helpless; I would change how I look if I could; I often feel left out of things; I often feel lonely; and I often have a hard time saying no. For each of the items, the response scale was: strongly agree=1, agree=2, neither agree nor disagree=3, disagree=4, and strongly disagree=5. The responses were reversed for the one positively worded item. Only those students with complete data for the nine items were included for this measure. The responses across the nine items were then summed and the resulting score was recoded into thirds (low, middle, and high) for presentation. The reliability of the scale as measured by Cronbach’s alpha is .82.

For additional information

The web site for the international HBSC study may be found at www.hbsc.org/.

Information on the Canadian team may be found at educ.queensu.ca/~speg/.

HBSC publications may be found through the national funder, the Public Health Agency of Canada, at www.phac-aspc.gc.ca/.