Evaluation of Learning and Labour Market Information as disseminated by Employment and Social Development Canada using a web-based consolidated approach

Official title: Sectoral Initiatives Program-funded Labour Market Information: Evaluation Directorate Strategic and Service Policy Branch

On this page

• List of acronyms
• Executive summary
• Management response
• 1.0 Introduction
• 2.0 Program background
  • 2.1 Employment and Social Development Canada (ESDC) legal requirements
  • 2.2 Job Bank and learning and labour market information over the years
  • 2.3 Learning and labour market information on Job Bank
  • 2.4 Government of Canada’s priorities: Access to more and better quality learning and labour market information
• 3.0 Methodology and limitations
• 4.0 Findings
  • 4.1 Relevance of learning and labour market information disseminated through Job Bank
  • 4.2 Effectiveness of the learning and labour market information disseminated through Job Bank
  • 4.3 Efficiency of the web-based consolidated approach to disseminating learning and labour market information
• 5.0 Conclusion and recommendations
• Annex 1: Evaluation questions
• Annex 2: Evaluation technical reports and data sources
• Annex 3: Major changes made to Job Bank and learning and labour market information activities since 2005
• Annex 4: References to learning and labour market information in Budgets 2012 to 2016

List of acronyms

• EI: Employment Insurance
• ESDC: Employment and Social Development Canada
• FLMM: Forum of Labour Market Ministers
• LFS: Labour Force Survey
• LLMI: Learning and Labour Market Information
• LMI: Labour Market Information
• WiC: Working in Canada

Alternate formats

Evaluation of Learning and Labour Market Information as Disseminated by ESDC Using a web-based consolidated approach - Final report [PDF - 3.64 MB]

• Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

Executive summary

This report presents the key findings and recommendations of the evaluation report entitled Learning and Labour Market Information (LLMI) as disseminated by ESDC using a web-based consolidated approach.

The evaluation broadly assessed the effectiveness of:

• (i) ESDC’s web-based consolidated approach to disseminating learning and labour market information via Job Bank; and
• (ii) the information posted on Job Bank to support Canadians in their decisions in acquiring education and skills, planning their career, and other labour market activities such as job search and job posting.

In 2009, ESDC was operating the Working in Canada website, the former Job Bank website, and multiple other LLMI websites and activities to provide learning and labour market information to Canadians. Since beginning consolidation activities of LLMI-related data in March 2014, ESDC merged these two main websites and subsequently added new tools and LLMI activities to create the current version of Job Bank (see Figure 1). This new website provides Canadians with a broad range of information and assistance for job searching, recruiting and hiring, career exploration, and labour market information. Some of the tools and pathways available to users are: Job Search, Job Match, Job Alerts, Job Bank for Employers, Explore Careers, Career Tool, and Labour Market Trends and News.

This evaluation considered a broad range of ESDC’s activities surrounding LLMI production and dissemination during the period from 2010 to 2015. In order to provide the most recent information available, data collection activities continued through to 2016. It is important to note that the dynamic nature of information available on Job Bank means that there are ongoing changes beyond 2016.

The evaluation report scope did not cover all types of ESDC’s learning and labour market information-related activities, nor all of Job Bank tools. Two complementary evaluations are currently being undertaken by the Evaluation Directorate:

1. The “Job Bank Evaluation” was launched in 2016 to assess the impact of the Job Match function on job search activities and is to be completed in 2018 to 2019.
2. The “Evaluation of Connecting Canadian with Available Jobs Initiative,” including the Job Alert function of Job Bank, is near completion. Its focus is on the enhancements to Job Alerts and labour market information and their effectiveness in supporting job-search activities and helping unemployed workers return to work more quickly.

The main outcomes of the program examined by this evaluation were the quality and quantity of learning and labour market information provided on Job Bank and the ability of users to find, understand and use that information. Net impacts resulting from LLMI activities were not assessed because many other factors that are not under the program’s control also influence users’ decisions in their job search and career and education planning.

Key findings

1. All lines of evidence suggested that the learning and labour market information provided by ESDC addresses the demonstrated needs of Canadians in general as well as the various target groups specifically. Canadians want comprehensive and local learning and labour market information to help them find jobs, make informed decisions about training, education and career pathways, and prepare for employment. Furthermore, target groups such as youth or Indigenous peoples have specific data needs and benefit from customized services.
2. The approach to disseminating learning and labour market information through a web-based consolidated Job Bank website was found to be an effective way to provide a wide range of information to Canadians. Because most Canadians today rely on the Internet as their main source of learning and labour market information, Job Bank successfully delivers this information in one website.
3. Overall, more than two-thirds (68%) of employers with Job Bank accounts were satisfied with the services they received with respect to posting a job, and the same proportion were satisfied with the site in general. However, among the employers who were not satisfied with the services they were receiving from Job Bank, 10% reported dissatisfaction with the processing time for posting a job, and 21% expressed dissatisfaction with the lack of flexibility when listing job requirements.
4. The production and dissemination of the learning and labour market information is considered a shared responsibility between federal, provincial and territorial governments. That said, there is a need for more clarity and coordination to ensure that all segments of the Canadian population have access to the same comprehensive and high quality learning and labour market information.
5. Job Bank provides learning and labour information that can be customized by the user to compare information by occupation and economic regions. However, more information is needed at the local or sub-economic regional levels to better reflect local labour market conditions in order to support job searches, education and career planning.
6. The Web analytics and consultations conducted with partners (close-end surveys, emails, phone calls, etc.) are some of the ways the Job Bank program gathers information to improve its product and processes. However, during the evaluation data collection process, it was recognized that the program needed to conduct client satisfaction surveys in order to gather such feedback and improve performance on a regular basis.
7. Areas where Job Bank can improve include:
   1. Provide learning and labour market information in a manner that is timely and accessible to all users
   2. Provide learning and labour market information that can be easily accessed and is not too aggregated or broad (for example local level versus economic region)
   3. Increase Canadian awareness of the Job Bank website and service, including the wide range of learning and labour market information available above and beyond the basic job postings
   4. Offer greater support to employers to meet their hiring needs in a timely manner and to facilitate the process of posting jobs

Recommendations

1. Further develop awareness, especially among students, youth and their influencers, of the Job Bank website and the learning and labour market information it provides for various users and target groups
2. Continue to consider the needs of various target groups (for example given differences in their literacy levels, preferred presentation styles, unique information needs) in the development and delivery of learning and labour market information in formats that are easy to access and use
3. Continue to find innovative ways to collect and provide users with learning and labour market information which reflects local realities (for example, data based on smaller geographic locations)
4. Continue to find ways to support employers posting jobs on Job Bank so as to more effectively meet their hiring needs in a timely manner
5. Monitor client satisfaction on a regular basis to continuously improve user experience and to ensure that feedback mechanisms are in place to inform further product development

Management response

Introduction

The findings from the evaluation indicate that the learning and labour market information on Job Bank as well as the dissemination approach used are effective in helping Canadians pursue their labour market objectives. The evaluation also highlights several areas for improvement in order for ESDC to better respond to the specific needs of various users including students, youth, employment equity groups, as well as employers.

The evaluation complements what was heard through recent federal-provincial/territorial (F-P/T) consultations under the Forum of Labour Market Ministers (FLMM) with respect to employment and skills training programming and services in Canada^{Footnote 1}. In particular, the consultations emphasized the need for services which are known, accessible and adaptable to a diverse population, including vulnerable groups. The need for labour market information (LMI) which is timely, relevant, accurate, reliable, granular and accessible, as well as the importance of F-P/T collaboration in producing this level of LMI, was also identified. This supports the establishment of the Labour Market Information (LMI) Council and its initial priorities to:

• address the need for more granular local LMI;
• prioritize consistency of LMI through alignment of definitions, standards, and methodologies; and
• develop and promote a collaborative platform for disseminating LMI for Canadians in consultation with stakeholders.

Management will consider the findings and recommendations to guide the ongoing work of the Department in providing learning and labour market information online. The outcomes of the evaluation will also be used in conjunction with the Department’s evaluation of the Connecting Canadians with Available Jobs initiative and upcoming Job Match evaluation to assess the effectiveness of Job Bank overall.

Recommendations

1. Further develop awareness (especially among students, youth and their influencers) of the Job Bank website and the learning and labour market information it provides for various users and target groups.

Management agrees that increasing awareness of the learning and labour market information on Job Bank is important for job seekers, especially among students, youth and their influencers.

ESDC has developed an engagement strategy to reach key users and will continue to implement this strategy to raise awareness of Job Bank and its learning and labour market information by participating in career-focused events (for example career fairs) that target specific client groups, including students, youth, and career counselors.

To develop awareness of the Job Bank website among influencers of youth and students, ESDC will identify potential partnership opportunities with school boards and existing career counseling services and look to influence the labour market and learning information content of other tools and resources.

In addition, Job Bank will continue to build its social media presence and strengthen its communication strategy to further promote its website and services to these client groups.

2. Continue to consider the needs of various target groups (for example given differences in their literacy levels, preferred presentation styles, unique information needs) in the development and delivery of learning and labour market information in formats which are easy to access and use.

Management agrees that various target groups have different needs in terms of accessing learning and labour market information.

Job Bank meets the World Wide Web Consortium’s Web Content Accessibility Guidelines 2.0 for persons with disabilities and is required to meet the level of conformance of AA.

In addition, in November 2015, Job Bank put in place a dedicated working group that reviews all existing and new content on the website to ensure quality of language, clarity, and consistency.

Job Bank updated the look of its homepage and job searching services in fall 2016 with the goal of enhancing user experience by making them more accessible and mobile-friendly. Other planned improvements for the Job Bank website include updating the presentation and the content of the Career Tool and the Explore Careers section to render them more interactive, user-friendly, as well as mobile-friendly. The career exploration will be seamlessly interlinked with labour market information and the tools will cover various client segments.

Job Bank is currently developing dedicated pages on its website for specific employment groups. A new veterans’ page on Job Bank was launched in May 2017 and similar pages for other groups will soon be available as well. These pages are being tailored to accommodate their respective target audiences in terms of their presentation and the content they provide.

ESDC is also in the process of expanding the current suite of client segment profiles, which provide provincial and local labour market information for select target groups including youth, seniors, Indigenous peoples, newcomers, and persons with disabilities. These reports will seek to be graphically dynamic, presenting trends and narratives in a manner that is more likely to solicit user engagement and foster greater connections to each of these groups. In particular, they will highlight potential risks that may hinder labour market integration, while suggesting regional, provincial and federal program opportunities that could help the targeted individuals overcome these challenges. It is anticipated that all of these reports will be available by the end of 2019, beginning with the youth report which is scheduled to be disseminated in early 2018.

Job Bank will also explore adding support tools to the website such as help text and instructional videos in order to assist users in navigating the website and the various information and services available.

Lastly, there are opportunities to encourage a wide range of stakeholders and service delivery networks to enhance assisted services for clients and target groups seeking learning and labour market information using the Job Bank website and its related services.

3. Continue to find innovative ways to collect and provide users with learning and labour market information which reflects local realities (for example, data based on smaller geographic locations).

Management recognizes the importance of providing users with learning and labour market information that more closely reflects local realities.

In 2014, the Job Vacancy and Wage Survey was developed in collaboration with Statistics Canada to address the need for more granular labour market information at the local level. The data collection began in February 2015 and the first release of job vacancy statistics was on August 13, 2015. The first dissemination of the 2016 wage component of the Job Vacancy and Wage Survey is expected in June 2017.

Recognizing the limitations of survey data to provide information at a detailed geographical area, ESDC will continue to seek ways to leverage administrative data sources, which has the potential to generate timely and detailed local labour market information.

ESDC has been working in collaboration with Statistics Canada exploring a more refined geography based on commuting patterns called Self-Contained Labour Market Areas. This geography is smaller than the economic regions and shows great potential in terms of its relevance for labour market information. Work will continue with Statistics Canada to determine the feasibility of producing survey and administrative data at the level of this new geography.

4. Continue to find ways to support employers in posting jobs on Job Bank so as to more effectively meet their hiring needs in a timely manner.

During the consultations under the Forum of Labour Market Ministers stakeholders highlighted the important role that governments could play in providing trusted and authoritative information. For the employers posting jobs this means ensuring the genuineness of the business and the available job.

Management agrees to continue exploring ways to provide responsive and effective services to employers while maintaining a high level of integrity. New standardization and monitoring tools have been put in place to allow Job Bank and provincial and territorial officials to manage operational standards. To date, these measures have proven to be effective in helping Job Bank validate new employers and job postings within the respective five day and two day service standards.

Although free text is no longer available to employers creating a job posting, Job Bank introduced a new feature that allows employers to suggest new job titles and skills if their desired options are not available in the database. Since this feature was implemented, hundreds of new job titles and skills have been added to better meet employers’ needs. In addition, employers have the ability to post a link to their own website to provide additional information about the business and the position.

Job Bank will explore adding support tools to the website such as help text and instructional videos in order to assist employers in navigating the website and the job posting service.

5. Monitor client satisfaction on a regular basis to continuously improve user experience and to ensure that feedback mechanisms are in place to inform further product development.

Management agrees that monitoring feedback on client satisfaction would be a useful complement to the existing feedback mechanisms.

Job Bank regularly monitors client experience from various sources including a Job Alerts exit questionnaire, a Job Match survey, an employer close-out report, and a Career Tool feedback survey. The feedback obtained from these sources informs Job Bank of the effectiveness and user experience of the respective services and supports Job Bank in making ongoing enhancements.

However, there is no overall feedback experience or satisfaction measure for the website. In 2017 to 2018, Job Bank will introduce a new client feedback mechanism targeted at users of the learning and labour market information on the website to monitor trends over time and to better meet their needs.

1.0 Introduction

This report presents the findings and recommendations of the evaluation entitled Learning and Labour Market Information^{Footnote 2} (LLMI) as disseminated by ESDC using a web-based consolidated approach. The evaluation broadly assessed the effectiveness of the dissemination of the information through the Job Bank website to support Canadians in making informed decisions regarding their education and career planning, job searching and job posting, and other labour market activities.

The evaluation was designed to assess the extent to which the information disseminated through the Job Bank website^{Footnote 3} is comprehensive, up-to-date, easy to access, easy to use, and available in a timely manner to all users who need it. The evaluation also assessed whether ESDC’s dissemination of its learning and labour market information through a single website, the Job Bank, is the most efficient means to the desired end – namely, to provide job and labour market information to Canadians. Finally, the evaluation examined user experience by type, including examining the specific needs of various target groups.

The evaluation covers activities related to the dissemination of LLMI from 2010 to 2015, initially via the Working in Canada website and the old Job Bank website, and then via the new Job Bank website.^{Footnote 4} In addition, more recent data and information were also examined, as available. It is recognized that Job Bank continued to evolve during the period of the evaluation and thus, data collected at various points reflect findings at that time.

2.0 Program background

This section provides a description of the program, a brief history of the learning and labour market information disseminated by ESDC using a web-based approach, and a listing of related activities undertaken during the period covered by the evaluation.

2.1 Employment and Social Development Canada (ESDC) legal requirements

ESDC provides a collection of products and services to Canadians (for example job seekers, employers and other providers of training and employment services, various levels of government and private sector) through its learning and labour market information activities. These products and services help Canadians better understand the labour market and supports them in making informed decisions regarding the acquisition of education and skills, career planning, finding employment and, in the case of employers, hiring workers.

2.2 Job Bank and learning and labour market information over the years^{Footnote 5}

Launched in 1980, Job Bank provided electronic job listing at Service Canada centres across Canada. In 1996, it was transformed into an Internet-based skills matching system with the aim of helping employers and workers connect online.

The “Summative Evaluation of Human Resources and Social Development Canada (HRSDC) Labour Market Information Products and Services” (2005) examined the impact of labour market information on the behaviour of unemployed individuals when searching for a job. The evaluation found that the labour market information products and services had a positive effect in reducing unemployment durations in the short term.

Subsequently, the Working in Canada (WiC) website was launched in 2007 to further improve the production and efficiency of learning and labour market information available to new and prospective immigrants.

In 2009, the Working in Canada website refocused its services in order to:

• reduce skill mismatches
• respond to labour shortages and unemployment by providing access to learning and labour information
• better inform job seekers by highlighting the skills and qualifications that employers were looking for
• facilitate access to information for all aspects of the labour market

An evaluation of the Working in Canada website activities, entitled “Summative Evaluation of the Going to Canada Immigration Portal Initiative” (2007 to 2010), was completed in 2011 in collaboration with Citizenship and Immigration Canada (now Immigration, Refugees and Citizenship Canada). The evaluation recommended establishing performance measures and consistent web data collection across all partners of the Portal Initiative as well as improving partners’ abilities to assess ongoing performance.

ESDC, along with other departments and levels of government, has been taking action to improve the quality, comprehensiveness and dissemination of learning and labour market information over the years (see Annex 3 for a brief list of actions). In 2010, as per the 2009 Drummond Report^{Footnote 6} recommendations, the Working in Canada website was designated as ESDC’s single integrated website for disseminating labour market information, drawing labour market information from over 14 sources (including ESDC, Statistics Canada, sectoral councils, provincial and territorial governments, private job boards). The Department has refocused its activities in the areas of labour market information, including streamlining and standardizing products and improving information on occupations, jobs, education, and employment outlooks. By 2012, the website was disseminating information from 30 different sources. On March 6, 2014, ESDC merged the WiC website and the former Job Bank website into a single platform under the name “Job Bank”. Figure 1 shows the evolution of the consolidation process.

Figure 1 – Text description

Figure 1 illustrates the evolution of how the ESDC disseminating its LLMI from many sources before 2010 to one consolidated web-based approach in March 2014.

From left to right the first section lists many learning and labour market information sources available either online websites or as publications prior to 2010. The listed sources include Labour Market Information System, Work Destination, National Occupation Classification, Essential Skills, Can Learn, Public Service Commission Jobs, Job Bank, Working in Canada Information Integration Tool (WiC-IIT), Citizenship and Immigration, Canada Mortgage and Housing Corporation (CMHC), Service Canada, Red Seal, Canadian Occupation Projection System, Working in Canada Administration Tool, Skill and Knowledge Checklist, and data from Statistics Canada such as Labour Force Survey, Census, and the Classification of Instructional Programs (CIP).

The second (middle) section has two square boxes, Job Bank is on top. Within the Job Bank box, it shows the website. It also states that the website was launched and functioning independently since 1996. Under Job Bank box, there is another box with written text: “Working in Canada Website in 2010”. In 2010, the Working in Canada website was designated as ESDC’s single integrated website for disseminating Labour market information, drawing labour market information from 14 sources. By 2012, the Working in Canada Website was disseminating information from 30 different sources.

The third section, on the right-hand side of the figure shows one single Job Bank icon. On March 6, 2014, ESDC merged the Working in Canada website and the former Job Bank website into a single platform under the name, Job Bank.

Source: Created for the purpose of this report.

2.3 Learning and labour market information on Job Bank

ESDC’s learning and labour market information activities consist of the development, collection, production and dissemination of high quality and reliable information in various formats. The information is gathered from multiple sources, including ESDC, Statistics Canada, sectoral councils, provincial and territorial governments, private job boards, and is disseminated through Job Bank website.

With a view to support Canadians in making well-informed labour market decisions, ESDC disseminates many types of information and provides services through Job Bank, including:

Find a Job

• Job Search allows job seekers to search job postings by type of occupation within chosen geographical regions in Canada. Several filters allow users to adjust their search by date posted, type of employment (full time, part time, seasonal, etc.), and more.
• Job Alerts is a free email service that notifies subscribers up to twice a day of new job postings in their chosen field and categories.
• Job Match matches current job openings with job seekers’ profiles based on provided criteria, including location, skills, education, credentials, and experience.

Recruit and Hire

• Employers or their third-party representatives can register and open an account to post a Job where they can list job postings and requirements.
• Employer Resources provides a range of information to help employers make human resources decisions.

Explore Careers

• Explore Careers offers valuable information about the labour market, such as reports on wage by sector, career outlooks for a variety of occupations according to region, essential skills profiles; and educational requirements by occupation.
• Specifically targeted to young job seekers, the Career Tool allows users to search programs and areas of post-secondary study to help them make more informed choices based on career outlooks (for example labour participation, median earnings and type of jobs found) and other labour market information.
• Job Market Trends and News provides labour market information such as unemployment rate, average salary, and labour market news. It also provides industry information, such as sectoral labour market information studies, labour market profiles, and so on.

2.4 Government of Canada’s priorities: Access to more and better quality learning and labour market information

Since 2009, the Government of Canada has stressed the importance of providing access to comprehensive, up-to-date and timely learning and labour market information for Canadians. To that end, between 2012 and 2016 the Government made several budget announcements relating to LLMI. For instance, Budget 2012 committed to providing $21 million over two years to enhance the content and timeliness of the job and labour market information for Canadians searching for employment. Budget 2013 committed to reallocating $19 million over two years to inform young people about fields of study in demand. This investment was split between the Youth Employment Strategy and the web-based Career Tool (available now on Job Bank website). The relevant budgets details in recent years are listed in Annex 4.

3.0 Methodology and limitations

This evaluation used multiple lines of evidence, including:

• key informant interviews (fall 2015);
• a document review (summer/fall 2015);
• data analysis (Canadians Out-of-Employment Panel Survey 2013 and 2014);
• a Job Bank user and non-user survey (winter 2016);^{Footnote 7}
• focus groups with three categories of Job Bank users (job seekers, employers and students, March 2016);
• Web analytics study (summer/fall 2015); and
• expert panels (economists, youth employment counsellors and web designers, summer 2016).

The detailed methodological discussions for different lines of evidence can be found in the respective technical reports.^{Footnote 8} In order to address the evaluation questions (listed in Annex 1), data was collected and analysed from the learning and labour market information needed by job seekers, employers and targets groups (youth/students, newcomers/recent immigrants, Indigenous peoples and persons with disabilities).

Evaluation findings were mainly based on users’ experience of Job Bank and their experiences of the quality and the comprehensiveness of the information disseminated through the website. For users, the interface through which they access the information is often indistinguishable from the information itself. The survey and the focus groups were conducted with job seekers and employers who were already aware of Job Bank. Views from these sources represent those of Job Bank users and cannot be extrapolated to the Canadian population in general. Therefore, users’ views were complemented with information from the document review, key informant interviews, and expert panels. As ESDC continues to evolve Job Bank and the production and dissemination of LLMI, the views expressed by key informant interviewees, focus groups and expert panel participants reflects the website and information at the time when these lines of evidence were developed.

The evaluation did not aim to assess all of ESDC’s activities or initiatives related to learning and labour market information. Two other evaluations (Job Bank Evaluation including Job Match, and Evaluation of Connecting Canadian with Available Jobs Initiative including Job Alert) are underway which assess some specific initiatives related to learning and labour market information.

4.0 Findings

4.1 Relevance of learning and labour market information disseminated through Job Bank

The evaluation assesses the relevance of ESDC disseminating consolidated learning and labour market information via the Job Bank website. Specifically, the evaluation examines whether:

• The information provided by ESDC addresses a demonstrated need among Canadians generally and among target groups specifically;
• The web-based consolidated delivery approach to disseminate this information aligns with ESDC’s mandate and the government’s priorities; and
• It is consistent with the federal government’s roles and responsibilities.

4.1.1 Meeting the learning and labour market information needs of Canadians

Evaluation findings show that Canadians want information to help them find jobs and make informed decisions about training, education and career pathways. According to the 2016 telephone survey findings, job seekers, employers, and career and education counsellors use Job Bank’s learning and labour market information to make their respective decisions. Specifically, 65% of job seekers and 52% of employers were either very or somewhat satisfied with the ability of the Job Bank to provide the information they needed.

Job Bank users also consulted other sources to complement the information found on Job Bank (see Table 1). For instance, Table 1 shows that education seekers^{Footnote 9} also used private job boards (32%), university and college (academic) websites (31%), and provincial and territorial governments’ websites/job boards (24%) as other sources for information related to Career.

Table 1: Most common sources for information on careers

	Pop-up	Telephone	Telephone
	Education seekers	Employers with a Job Bank account	Job seekers
Number of survey participants	618	413	1,702
	%	%	%
Job Bank	55	24	20
Private job boards	32	8	13
University and college (academic) websites	31	5	10
Provincial/territorial governments websites/job boards	24	12	9
Social media	18	11	12
Other federal departments	13	9	5
Career counselling organizations	12	1	5
Trade and professional associations	9	14	7
Municipal websites	7	2	4
Industry groups	4	8	3
Banking websites	2	1	2
Think tanks	1	0	<1
Chambers of commerce	1	2	1
Internet search (unspecified)	0	3	2
Other	7	4	6
I do not seek that information	10	42	39

Source: Telephone Survey and Pop-up Survey 2016.

In expert panel discussions conducted in summer 2016, there was consensus that most of the information needed by job seekers is available on the Job Bank website. Two career counsellors indicated that Job Bank is a good source for entry- to mid-level candidates. This view was echoed by focus group discussion participants, especially students, who also pointed out that jobs posted on the website are representative of middle level positions.

Some key informant interviewees contacted in fall 2015 highlighted the lack of relevant local learning and labour market information for smaller geographical areas of economic regions.

Job seekers and employers in focus group discussions indicated that some information provided on Job Bank, such as wage information, was not reflective of local market conditions. For these study participants, the wage posted on Job Bank is often higher than what, in their experience, is paid. In addition, focus group participants indicated that wage information provided at the local level and complemented with information on the cost of living (for example rent, transportation, etc.) in specific areas would be very useful.

The telephone survey indicated that some education seekers found the information provided through Job Bank was too broad for their more specific learning needs. For example, 20% of education seekers did not find the information provided on Job Bank useful and a quarter (26%) of them indicated that the information available on Job Bank was too broad or they were not able to find the information they need.

Employers

Employers with a Job Bank account used Job Bank mainly to post job vacancies. According to the survey undertaken in early 2016, employers were appreciative of Job Bank to post their job vacancies for free compared to other job boards who charge a fee. Employers had success in hiring candidates through Job Bank. They use it as their first place to post jobs and as common source of other LLMI.

Employers expressed the following concerns that affect their experience using Job Bank:

• The results of the 2016 telephone survey indicated that among employers who were not satisfied with Job Bank, 10% of them indicated that it takes too long to post a job on Job Bank. Employer participants from the focus group discussions in Quebec indicated that they would like the number of days it takes for a job posting to appear on Job Bank website be reduced.
• Employer participants in the focus group discussions from both Ontario and Atlantic regions indicated that wages specified on Job Bank website were higher than those that were generally paid for similar positions and were not reflective of their local working conditions. This may be due to the fact that
  • (i) this information is calculated based on the Labour Force Survey for national classification codes and different economic regions (not for smaller geographical regions); and
  • (ii) only the median wage information is displayed alongside the job, although a minimum and maximum wage are available elsewhere on the site.
• At the time of the evaluation, employers who were surveyed and those who participated in the focus group discussions were concerned about the lack of flexibility in customizing the job descriptions. Employers participating in the focus group in Ontario indicated that the requirement to post jobs by National Occupational Classification code was a hindrance.
• Finally, some employers expressed the view that the Job Match tool provided them with weak matching results. One employer from Ontario described his experience as the following: “My Company is currently posting two jobs. We’ve had over 50 applicants and none have been suitable. You need to make it so it is more customizable.”

Youth

Expert panelists and key informants all agreed that youth (including students) have a special need for forecasts of future labour market conditions that could help them plan educational and career paths. They need information on labour demand, wage forecasts for different job categories, expected trends for jobs in various industrial sectors, skills/education required for these jobs, and costs and returns of various educational paths. Youth and especially students have a particular need for local job postings to help them find jobs while they are studying (for example jobs on or very close to campus). The interviewees having experience with these groups^{Footnote 10} stated that LLMI for direct use by these groups needs to be prepared in age appropriate formats and presented to them at a lower reading level, as well as through the latest technology and communication tools that appeal to youth (for example social media). The career counselor experts suggested that youth engagement and LLMI awareness strategies should focus on key influencers in youth career decisions, including, for example, parents, guidance and career counsellors, youth service providers, band councils, and immigrant service agencies.

In summer 2016, expert panelists indicated that given young people’s lack of job and career experience, they could benefit from a website specific to their needs rather than Job Bank in its current form.

Recent immigrants^{Footnote 11}

Data from the early 2016 telephone survey revealed that 40% of jobs are obtained through job seekers’ personal networks. Respondents to the 2013 and 2014 Canadian Out-of-Employment Panel surveys^{Footnote 12} indicated that talking to friends or relatives as a method for searching jobs was reported second after searching on the Internet. Given the importance of contacts, recent immigrants who may not have well-established networks may have specific information needs. According to the key informants interviewed, recent immigrants share similar information needs with less experienced labour market participants such as youth and students. Additionally, recent immigrants may benefit from information about the resources available to help them integrate and understand the requirements of federal, provincial and territorial regulatory bodies. At the time of the evaluation, Job Bank did not provide specialized content for recent immigrants such as employment agencies, careers and education counselors, or training programs.

Expert panelists indicated that to facilitate recent immigrants finding jobs quickly, links to support programs and services similar to those provided to employers on Job Bank through the Employer Module could be helpful.

People with disabilities

According to key informant interviews conducted in fall 2015, persons with disabilities have a need for LLMI delivered to them in accessible formats. Job Bank has made progress in providing information accessible to people with certain disabilities and could work with employers who post jobs on the website to provide more information on accommodations (for example wheelchair accessibility). People with disabilities seek to identify employers who are willing to employ and accommodate them.

One expert panelist suggested that people with mobility challenges would benefit from the addition of a filter identifying the extent to which a workplace is accessible (for example by wheelchair, public transit, elevator, accessible washrooms, etc.).

Indigenous peoples

According to a 2015 study published by the Centre for the Study of Living Standards,^{Footnote 13} current sources of LLMI, such as Statistics Canada surveys (National Household Survey, Labour Force Survey, Aboriginal People’s Survey), provide extensive information. A continuing challenge is the delivery of information to Indigenous peoples living on reserve.

Older workers

In February 2016, a pop-up survey was launched on the Job Bank website for about three weeks, recording over 20,000 responses. Results revealed that 17% of job seeker respondents were 55 years of age and older. Older job seekers were less likely to use Job Bank, but for those who did use it they visited the site frequently. They were also more aware of the different types of learning and labour market information that Job Bank provides.

4.1.2 Disseminating information via a single website, Job Bank, is an appropriate mechanism for meeting Canadians’ needs

Canadians are increasingly relying on the Internet as a source of information and services. The decision to consolidate ESDC’s learning and labour market information and disseminate it through one website, Job Bank, was an effective approach as it makes the information produced from many sources easier to access and use.

The quantitative analysis of the Canadian Out-of-Employment Panel 2013 and 2014 survey data showed that close to 90% of job seekers surveyed used the Internet to look for a job, and of those who used the Internet to search for a job, 66.8% in 2013 and 57.1% in 2014 were WiC or Job Bank website users. The findings from the 2105 Web analytics study also show that Job Bank has an average of more than 5 million visits every month. This same study showed that although the Job Search section on the Job Bank site is the most used feature (in proportion as well as in absolute numbers), visits to other sections (Explore Career and Job Market Trends and News) also steadily increased between January 2010 and November 2015. In addition, results from the 2016 telephone survey showed that both job seekers (71%) and employers (70%) agreed that the information on career planning is easy to find. The percentages of job seekers and employers who agreed that information on labour market trends is easy to find was 67% and 63%, respectively.

4.1.3 The web-based single-window approach is aligned with departmental and government priorities

As per the Employment Insurance Act, Employment Insurance Regulations, and the Employment Service Convention (No. 88) of the International Labour Organization, ESDC has the legal obligation to deliver a National Employment Service to help workers find suitable employment, to help employers find suitable workers, and to make information about the labour market available to the public.

Since 2012, federal Budgets have supported investments in labour market information as a reflection of the government priority and commitment to provide Canadians with comprehensive, up-to-date, and timely learning and labour market information. For instance, Budget 2014 committed “…to invest $11.8 million over two years and $3.3 million per year ongoing to launch an enhanced Job Matching service to ensure that Canadians are given the first chance at available jobs that match their skills in their local area.”^{Footnote 14} According to the Budget, the purpose of enhancing the Job Matching Service was to “…build on the launch of a modernized and easy-to-use consolidated National Job Bank, providing job seekers and employers with timely access to job postings and consolidated labour market information.”^{Footnote 15} Budget 2015 confirmed new investments in enhanced learning and labour market information with the reallocation of $4 million over two years for “Enhancing labour market information, including the launch of a new one-stop national labour market information portal.”^{Footnote 16} It also announced the launch of a new Career Tool, ensuring “An improved Job Bank that will make it easier for workers to find information on job opportunities outside their local geographic area.”

Job Bank supports the above priorities by acting as a one-stop national website, providing the Career Tool, comprehensive labour market information, and offering access to job opportunities. In addition, the creation of a single point of service for Canadians looking for jobs and learning and labour market information is consistent with the federal government’s Web Renewal Initiative. The Government of Canada launched the initiative in 2013 with the objective of making its online information and services easier to find and use through a single website. The initiative reflects the Government’s commitment to simplify and improve its web content.

4.1.4 The dissemination of learning and Labour market information is aligned with government roles and responsibilities

The role of the federal government is to ensure that all segments of the Canadian population have access to comprehensive and high quality learning and labour market information to help them make informed decisions. Particularly, the federal government is responsible for providing employment services and labour market information to segments of the Canadian population (that is, Indigenous people) underserved by the private sector; as such it is considered a “public good”.

In interviews, program representatives indicated that the federal government has a coordination role in the production and dissemination of learning and labour market information. Key informant interviewees from provincial and territorial governments consider the collection and dissemination of LLMI as a shared responsibility, confident that they can produce LLMI that is better tailored to provincial and local needs than the federal government. They see the federal government’s role as one of facilitation.

The document review indicates that the federal government is increasingly partnering with the provinces and territories to lead and facilitate the production and dissemination of labour market information. This collaboration is proceeding via the Forum of Labour Market Ministers, which has begun to take over the planning of Canada’s learning and labour market information provision to reduce overlap and increase complementarity between the federal and provincial and territorial governments.

4.2 Effectiveness of the learning and labour market information disseminated through Job Bank

The evaluation assesses the effectiveness of the learning and labour market information disseminated by ESDC via Job Bank by examining the quality of the information and users’ experience and satisfaction with the Job Bank website and the information it provides. Factors considered include the extent to which the information posted on Job Bank is useful, up-to-date, standardized, comprehensive, and authoritative.

4.2.1 Information disseminated through Job Bank is up-to-date based on available data

In general, LLMI disseminated through Job Bank is perceived as up-to-date. Job seekers and employers responding to the 2016 telephone survey indicated that information on career planning is up-to-date. For example, almost three quarters (74%) of surveyed employers agreed that job postings are up-to-date (40% strongly agree and 34% somewhat agree) and 67% agreed that information on career planning is up-to-date (35% strongly agree and 32% somewhat agree). Most focus group participants reached in March 2016 also found the information on Job Bank is up-to-date.

Key informant interviewees contacted in fall 2015 pointed out that certain information on Job Bank was “out-of-date”. This is often due to the delay between the time data is collected and released by Statistics Canada (for example Census and National Graduate Survey) and the time it is subsequently posted on Job Bank. The interviewees were pleased that Job Bank informed its users of the reference period and/or publication date of LLMI products.

Some LLMI disseminated on Job Bank may be considered outdated due to the frequency at which the data is collected and made available. For instance, the data source that informs median and range earnings associated with post-secondary programs is the National Household Survey, which is only available every five years. Findings from the expert panels indicated that some sources of data are updated more frequently (for example the Labour Force Survey) than others (for example the National Household Survey) and this affects the extent to which information is up-to-date or not.

With regard to job postings, job seekers in the focus group discussions found some job vacancies to be out-of-date, as they remained on Job Bank even though the employers had already filled them.

4.2.2 The standardization of learning and labour market information posted on Job Bank has improved its quality

Since the methods used to produce the learning and labour market information, and the way it is disseminated on Job Bank have been standardized, many interviewees noted that the quality of information has improved. By ensuring all information is presented by the same denominator (occupation, province or economic region), the information can be compared at the national, provincial and territorial, and economic region levels.

At the time of the evaluation, it was reported that, in some cases, standardization has had a negative effect on the quality and the accessibility of LLMI. Findings from key informant interviews show that for some job seekers and employers, the information is not sufficiently disaggregated to reflect the labour market conditions in their local areas. As indicated by key informants, this may be partially explained by the fact that not all local data produced in Service Canada regional offices or by provinces and territories is disseminated. This could be because they are not standardized at a local level, the quality is an issue or they are simply not available.

4.2.3 Information is easy to find and use

Overall, most users indicated that accessing information on Job Bank is easy. This finding was confirmed by the focus group discussion participants, especially in Ontario and Quebec.

Both employers and job seekers who responded to the 2016 telephone survey agreed that the information on Job Bank was easy to find. For instance, 67% of job seekers agreed (including 39% strongly agreed and 28% somewhat agreed) that the labour market trends are easy to find.

However, some participants in focus group discussions also pointed out that when applying for a job through Job Bank, they have to “click through” a number of pages to get access to an employer's site or to find the exact information they are looking for. One of the focus group participants shared his experience in trying to get the information he needed through the Job Bank search box as the following:

“I wanted to know the outlook of truck drivers in Vancouver, but the first thing to show up was Alberta. Then I tried to search by city “Vancouver”, the first occupation outlook show up was related to law not truck driver.”

Overall, the findings suggest that the navigation instructions on Job Bank and the LLMI it provides are easy to find and use, nonetheless a few challenges were identified by some users.

4.2.4 Information available on Job Bank is integrated and of quality but its comprehensiveness could be improved

Many key informant interviewees across all groups stated that in recent years ESDC has improved the quality of its learning and labour market information through different initiatives such as:

• Launching a new survey such as the Job Vacancy and Wage Survey, which collect additional information on available jobs and average offered hourly wage
• Partnering with private job boards Workopolis and Monster to provide a higher volume and broader range of jobs through Job Bank

Job Bank provides learning and labour market information either by providing it directly on its site or via links to external sources (for example links to post-secondary programs). Findings from the document review and key informant interviews suggested that the information provided by Job Bank is integrated and comprehensive since users can access a wide range of different information via a single window. For example, Job Bank provides:

• Data at the national, provincial and territorial, and economic region levels
• Data on all sectors of the economy
• Data on all 4-digit National Occupational Classification occupations
• Data relevant to different types of users (for example job seekers, employers, education seekers, and providers of training and re-employment services)

According to some lines of evidence, the information posted on Job Bank is not sufficiently comprehensive. For instance:

1. Expert economists and career counsellors contacted in summer 2016 indicated that the learning and labour market information on Job Bank is not sufficiently relevant at the local level. Due to the sample stratification issue with the national surveys that feed the LLMI on Job Bank, information tends to be more accurate at the broader geographical level and not always accurate at local level or for remote areas and small cities due to lack of sufficient data
2. In other cases, local LLMI from Service Canada regional offices was available (that is information on a particular aspect of a specific community) but it was not disseminated due to a lack of standardization of the data at that level or was difficult to find it easily on Job Bank website. To improve the availability of local data, some expert panelists suggested that ESDC further develop collaborations and agreements with provinces and territories and other local labour market information producers to collect, tap into, and share more local data and make them available to users
3. The document review revealed that LLMI tailored to apprentices and prospective apprentices is lacking on Job Bank. Job Bank does not link to the National Occupational Analyses for Trades or National Occupational Standards developed by sectoral organizations to help the above clientele. In terms of ability to provide access to comprehensive LLMI from external sources, expert panelists, key informant interviewees and focus group participants indicated that Job Bank should continue to provide links to external LLMI sources to facilitate Canadians’ access to the fullest range of LLMI available

4.2.5 ESDC’s web-based consolidated learning and labour market information is considered authoritative by users

The learning and labour market information on Job Bank is increasingly relied upon by its users. Findings from the telephone survey revealed that usage of Job Bank is relatively high, with 87% of employers with a Job Bank account and more than 60% of job seekers who used information on the site in the past two years. Results of the telephone survey also indicated that the Job Bank site was the most frequently visited place for posting and searching for jobs (56% employers with a Job Bank account, and more than 40% job seekers who had a recent job separation).

In addition, the results of the telephone survey indicated that while Job Bank users also consult other sources of information, Job Bank is their main source for information on career planning and labour market information for all its user categories. Some Job Bank users consider it as a place they can find almost all information they need – a one-stop shop. For example, 55% of education seekers and 23% of job seekers reported that Job Bank is their most common source of information on career planning and approximately half of both job seekers and employers agree that Job Bank is a one-stop-shop (54% of job seekers agree versus 49% for employers).

Job Bank is the most commonly used site among different LLMI providers. Table 2 shows that, for respondents to the telephone survey, Job Bank is the primary place for posting jobs for employers with a Job Bank account and a place for searching a job for job seekers.

Table 2: Most common places for posting jobs and searching for work, 2016

	Employers with a Job Bank Account	Job Seekers
Number of survey participants	414	1,702
	%	%
Job Bank	56	40
Indeed.ca	25	35
Kijiji/Craigslist	24	2
Industry specific websites	8	20
Workopolis	8	17
Linkedin.ca	7	11
Monster	4	11
In-person employment assistance services	3	8
Union websites or employment resources	1	10
Jobpostings.ca	3	8
Other methods	8	5

Source: Telephone survey conducted January to March 2016.

4.2.6 Information on career planning and labour market trends was useful

Overall, employers and education seekers responding to the pop-up survey conducted in February 2016 found the information on career planning, job market trends, and the information that helps in posting a job to be useful.

Over half (56%) of employers responding to the pop-up survey found information on career planning to be useful (including 31% very useful and 25% somewhat useful); further, 61% of them found information on job market trends to be useful (including 33% very useful and 28% somewhat useful). More than half of employers who responded to the telephone survey found the information dedicated to helping them post jobs useful, compared to employers from the pop-up survey (60% useful versus 52%).

Career Tool and Explore Careers

Career Tool and Explore Careers are generally consulted by education counsellors and recent immigrants. Job Bank provides information under Career Tool and Explore Careers that could be useful for youth, but youth are often not aware of this information. A Web analytics study of the Career Tool conducted in 2015 found that 49% of the 101,710 visitors to Career Tool were returning visitors, and the average length of visit was 15 minutes. The 2016 pop-up survey findings indicated that education seekers most commonly sought information on education requirements for certain professions (66%) and forecasts of job openings (56%) through the Explore Careers module, but only 14% of education seekers consulted the Career Tool under Explore Career Module.

To increase awareness and usage, expert panelists suggested promoting Job Bank and the Career Tool as well as Explore Careers Module to youth and their influencers (parents, guidance and career counsellors, youth service providers, band councils, and immigrant service agencies, etc.). Using the Job Bank website of June and July 2016 as a reference, expert panelists observed that a more functional mobile version of the Job Bank website could help youth access information in a more immediate manner.

Information on career planning and labour market trends

Job seekers, students and employers were all satisfied with the site as a whole, as well as with specific information on the site such as career information, job market trends and job listings. The pop-up survey indicated that job seekers (69%) were satisfied with the job postings but less satisfied with information on career planning (55%) and labour market trends and news (49%). Employers were satisfied with information on career planning (62%) and labour market trends as were job seekers (60%) with that information.

4.2.7 Overall satisfaction with Job Bank

At the time of the evaluation, Job Bank users were satisfied with the website as a whole. For instance, 72% of job seekers responding to the 2016 telephone survey were satisfied with Job Bank overall (including 33% very satisfied and 39% somewhat satisfied).

In the pop-up survey, 70% and 71% of job seekers and students were satisfied, respectively; and 66% of employers from the pop-up survey and 68% of employers from the telephone survey were satisfied.

4.2.7.1 Opportunity for greater awareness of Job Bank and LLMI it provides

Results from the 2016 telephone survey indicate that Job Bank is well known by job seekers who experienced a recent job separation (76%). These job seekers were made aware of Job Bank through the Service Canada website or one of its local offices (52%), or through an Internet search (24%). On the other hand, job seekers who have not been in a situation that connected them with ESDC services related to employment have a low awareness of Job Bank. Findings from the 2013 and 2014 Canadian Out-of-Employment Panel surveys show that the reason given by more than half of job seekers who did not use Working in Canada or Job Bank was that they were not aware of the two sites.

The results from the 2016 telephone survey also revealed that although Job Bank is recognized as a place that provides information on career planning and job market trends for all users, awareness and use of this information beyond career counsellors is lower.

For example, while 99% of job seeker respondents were aware that Job Bank provides information related to job postings, only about two-thirds of them knew Job Bank also provided career planning information.

In addition, while there were about 1.27 million^{Footnote 17} active businesses with employees in Canada as reported in December 2016, it was estimated that only 6%, or 75,000, have a registered account to post a job on Job Bank. In comparison, Workopolis and Monster have 2% of Canadian businesses registered on their websites. The findings from the 2015 Web analytics study show that most of the visits (79%) to the Job Bank site were related to Job Search function (see Figure 2).

Figure 2 – Text description

Percentage of share	2015	2017
Explore Career	13%	9%
Job market trends and news	2%	0.27%
Job search	79%	76%
Other	6%	15%

Source: Google Analytics of Job Bank, extracted December 2015 and April 2017.

The importance of awareness of the use of LLMI was illustrated by participants in focus groups. They indicated that after they were informed of the existence of information on career planning and labour market trends on Job Bank through the pop-up survey (which was also used to recruit focus group participants), they consulted it before their focus group sessions and reported to be interested in using the information in the future.

Career counsellors in the expert panel indicated that the youth level of awareness of LLMI in general, and of the Job Bank site as a place to access such information in particular, was fairly low. This was confirmed by the results from the pop-up survey. As shown in Figure 3, youth (aged 15-24) were less aware (compared to other target groups) of career planning, job market trends and news available through Job Bank.

Figure 3 – Text description

	Total	Female	Youth	Aboriginal	People with activity limitation	Recent immigrant
Provides information about career planning	68%	66%	57%	67%	69%	71%
Provides information on job market trends	61%	60%	49%	63%	62%	69%

Source: Pop-up Survey of Job Bank users, February to March, 2016.

All three career counsellor participants who took part in the expert panel discussion in summer 2016 expressed the view that students in grade 8 and 9 should be exposed to Explore Careers and the Career Tool. One of them even suggested that these career planning tools and information should be included in the curriculum through high school and post-secondary. This would ensure that, as new entrants to the job market, youth are aware of the resources and information to help them make informed education and career choices.

Although knowledge of the Job Bank’s job searching services is generally high, many Canadians are not aware of the career planning and job market information that is also available. It is clear that Job Bank would benefit from a more robust promotion and communication strategy to increase awareness and to expose youth early on in their education. This is valuable in terms of helping Canadians better understand how to navigate the labour market, and supporting them in making informed decisions related to their education and career.

4.2.7.2 Good user satisfaction

Overall, users found their experience of Job Bank^{Footnote 18} to be satisfying. For instance, 72% of job seekers and 68% of employers responding to the 2016 telephone survey were satisfied with Job Bank. When specifically asked about improving the site, the issue of design was raised as an area where further improvement could be made.

Most users participating in the focus group discussions in March 2016 found the site attractive and the information well-organized, easy to find and easy to use. This was confirmed by most expert panelists who, in summer 2016, found navigating to other pages on the site relatively easy. However, they also indicated that the navigation experience could be improved and that the mobile version of Job Bank needs more work to provide the same level of experience as with the non-mobile version.

According to expert panel discussions held in summer 2016, the Job Bank website’s homepage look, its content arrangement, and the navigation from the homepage to sub-pages was overall satisfactory but there is room for improvement. The panelists have recommended sites (for example Career one stop ) that could be examined to improve the Job Bank site design, the content offered, its information presentation style, and media used to provide information to various user groups. Subsequent to the conduct of the expert panel discussions, the homepage was redesigned.

Navigation tools

While the expert panel (economists and career counsellors) were generally pleased with the Job Bank site’s navigation, the webmasters group indicated that the use of icons supported with text would help users know more about different products posted on the site. Some of the suggestions given by the focus group participants and expert panelists suggested included offering a tutorial video, adding bubbles to show what the icons are for, and adding embedded descriptions under icon titles. Users also suggested adding online chat as well as in-person help for some other types of users such as new visitors and less-educated users.

Search function and filters

Overall, users were satisfied with the search function but would like to be directed to the information they are seeking more quickly and not be obliged to go through multiple clicks before getting the information they are looking for. They also expressed a need for more accurate filter features that allow excluding un‑needed information from being retrieved. For example, “Healthcare” but not “Nurse”; “Driver” but not “Taxi”.

Job Bank mobile version

Although panelists and other users in summer 2016 were satisfied with the Job Bank website, the mobile version was found to be confusing and lacking. Results from the 2015 Web analytics study showed that users increasingly access Job Bank through a mobile device or tablet (Figure 4). Webmaster panelists recommended that the mobile version (as of summer 2016) of Job Bank be improved to provide users and especially youth with complete features and functionalities comparable to those available on the Job Bank website accessible by computer or tablet. Job Bank improved its mobile version during the time the evaluation was underway.

Figure 4 – Text Description

Percentage of Job Bank users accessing the site

	Desktop	Mobile	Tablet
Mar-2014	73.7%	18.3%	8.0%
Apr-2014	74.0%	17.7%	8.3%
May-2014	71.6%	19.2%	9.2%
Jun-2014	70.2%	21.8%	8.0%
Jul-2014	67.6%	24.1%	8.3%
Aug-2014	65.8%	25.9%	8.3%
Sep-2014	65.3%	25.6%	9.1%
Oct-2014	65.9%	25.7%	8.3%
Nov-2014	64.3%	26.4%	9.3%
Dec-2014	62.7%	27.0%	10.3%
Jan-2015	63.5%	27.3%	9.2%
Feb-2015	64.5%	26.2%	9.4%
Mar-2015	64.5%	26.1%	9.4%
Apr-2015	63.1%	27.7%	9.2%
May-2015	62.5%	28.2%	9.3%
Jun-2015	63.2%	28.0%	8.8%
Jul-2015	61.2%	29.6%	9.3%
Aug-2015	60.5%	30.7%	8.7%
Sep-2015	60.6%	30.0%	9.4%
Oct-2015	60.9%	29.6%	9.4%
Nov-2015	59.5%	31.2%	9.3%

Source: Google Analytics of Job Bank, extracted December 21, 2015.

Job Match and Job Alert^{Footnote 19}

Although, Job Alert and Job Match functions are not covered by this evaluation and are discussed in detail by two other evaluations underway, user views related to these two functions are reported herein as they are relevant to the experience of using Job Bank.

Employers in the Western Canada focus group used Job Match tool but had concerns with its results. Employers in others focus group discussions concurred and suggested either improving or elimination the Job Match tool, claiming that the algorithms currently in use are too simplistic and result in numerous inappropriate referrals for postings. For example, one employer had posted a job for which one of the requirements was a high school diploma. He received 76 matches that all have completed high school but did not meet other required skills and competencies.

Job seekers who participated in focus group discussions suggested that Job Alert limits the number of emails sent per day by amalgamating multiple job alert emails in a given day into one. Participants in focus groups in Montreal noted the poor quality of results of the job matching with Job Alert feature. For example, one client searched for “Chef de service” but received alerts for “culinary chef.”

4.2.8 Users experience with Job Bank

The overall user experience of Job Bank was positive. However, the evaluation found that the experiences of different target groups and user types varied depending on the extent to which the group was aware of the learning and labour market information available and the website’s ability to provide information that meets their specific needs.

4.2.8.1 Youth

Youth were less aware that Job Bank provides information about careering planning and job market trends. However, the data from the Canadian Out-of-Employment Panel survey (2013, 2014) show that youth who used Job Bank or WiC found these websites to be useful 77.7% in 2014 (see Table 3). These findings suggest that youth lack awareness of Job Bank and the information and products it provides, but when they do become aware of it, they find it useful.

Table 3: Usage and usefulness of Working in Canada or Job Bank website by target groups, 2014

	Use of Working in Canada website or Job Bank			Overall usefulness of Working in Canada website or Job Bank
	Users	Non-users	Difference in proportions	Not at all useful or not very useful (%)	Somewhat useful (%)	Useful (%)	Very useful (%)
Population	53.1	46.9	6.20	20.8	17.4	44.6	17.1
Women	55.8	44.2	11.60	21.5	22.6	41.7	14.2
Aboriginal people	64.8	35.2	29.6	23.1	12.7	47.1	17.2
Persons with limitations	65.2	34.8	30.4**	16.3	21.0	53.7	9.0
Visible minorities	45.5	54.5	9.00	17.9	11.0	58.8	12.3
Recent immigrants	n/a	n/a	n/a	n/a	n/a	n/a	n/a
Youth	30.2	69.8	39.6***	3.7	18.6	55.1	22.6
Older workers	56.4	43.6	12.80	33.3	19.7	29.1	17.9

Percentages are calculated relative to total users in the group. ×: Deletions are made when figures are below 30.
Percentages are calculated relative to the group total; weighted figures were used. Degree of significance *** 1% **5% *10%
Source: ESDC calculation using 2014 COEP survey data (cohorts 18) (the period for which the most recent data are available).

4.2.8.2 Recent immigrants

Based on the pop-up survey results, recent immigrants (in Canada for less than 5 years) have good experiences with the information disseminated through Job Bank. For instance, a quarter of recent immigrant job seekers (25%) consulted information on career planning during their session, as opposed to 16% of non-recent immigrants. Further, 41% of immigrants strongly agreed that the information they consulted motivated them to seek more information about the labour market, as compared to 27% of non-recent immigrants. However, expert panelists indicated that recent immigrants could benefit from links to support programs tailored to newcomers, similar to employers for whom resources are dedicated specifically to address their needs.

4.2.8.3 People with disabilities

As previously discussed, job postings with information on the degree of accessibility would be helpful information for persons with disabilities.

4.2.8.4 Indigenous people

The Canadian Out-of-Employment Panel survey results show that Indigenous people who use Job Bank are relatively satisfied. As shown in Table 3, 64.3% of Indigenous people who used Working in Canada or Job Bank website found they are useful or very useful.

Indigenous people who live or wish to work on reserve face multiple barriers (for example low education attainment, computer access, Internet and library services) when accessing information. These barriers might limit their experience in using Job Bank website and the learning and labour market information it provides.

4.2.8.5 Older workers

The Canadian Out-of-Employment Panel survey indicated that among older workers (aged 55 and older) who had recent job separation, approximately 56% used Working in Canada or Job Bank to search for a job. According to expert panelists, some older workers are not aware of Job Bank and could benefit from greater awareness of available learning and labour market information and knowing how to access, navigate, and understand it. Both 2013 and 2014 Canadian Out-of-Employment Panel surveys indicate that about half of older worker users found the Working in Canada and Job Bank websites useful or very useful (at 58% and 47%, respectively).

4.3 Efficiency of the web-based consolidated approach to disseminating learning and labour market information

The objective of this section is to discuss efficiency issues related to the production and dissemination of LLMI through the implementation of a web-based consolidated approach. This is done by analysing:

• Changes in resources allocated to providing LLMI and impacts on outputs (quantity, quality, speed of dissemination, etc.) and its effectiveness
• Alternative service delivery models in the provision of LLMI

4.3.1 Spending on production and dissemination of LLMI remains steady as satisfaction, quantity and quality increase

For this evaluation, financial information of various LLMI activities over the years was examined, especially during the pre- and post-consolidation of Job Bank periods. Given that LLMI activities are conducted by several groups, it was difficult to identify all of the corresponding resource centres (RC) responsible for these activities. These resource centres have been changing over time, making comparisons difficult. Finally, it is challenging to distinguish LLMI-related activities from others at ESDC (such as the foreign credential recognition program). For these reasons, the evaluation focused on aggregate financial information when addressing the question of efficiency.

Table 4 presents ESDC’s expenditures on various broad LLMI-related expenditures from 2010 to 2011 to 2015 to 2016. The spending on production and dissemination of learning and labour market information remained steady during those 6 years. The average spending was about $80 million per year, with the exception of 2013 to 2014 and 2014 to 2015 when it was reduced by over $20 million per year.

As shown in Table 4, ESDC spends over 90% of its LLMI budget on data (for example funding surveys administrated by Statistics Canada and regional LMI data collection and dissemination) and intelligence (for example funding to sector council and sectorial initiative program). There was a $23 million increase from 2014 to 2015 to 2015 to 2016 due to additional funding to data collection, namely the Job Vacancy and Wage Survey ($12.5 million) and National Apprenticeship Survey ($11.6 million).

Table 4: ESDC learning and labour market information expenditures, 2010-2011 to 2015-2016

(in millions $)	2010 to 2011	2011 to 2012	2012 to 2013	2013 to 2014	2014 to 2015	2015 to 2016 at year-end
Data	40.14	43.78	40.47	33.35	35.63	59.84
Intelligence	34.08	34.77	35.81	22.28	26.09	23.68
Dissemination	5.40	6.23	4.51	3.55	4.10	5.27
Governance NHQ-LMI (FLMM)	0.28	0.14	0.06	0.06	0.07	0.08
Total	79.90	84.92	80.85	59.24	65.88	88.87

Source: ESDC, Chief Financial Officer Branch, internal document.

Table 4 also shows that ESDC’s expenditure on dissemination has been steady at around $5 million over the period from 2010-2011 to 2015-2016. Figure 5 gives a distribution of this expenditure category over various ESDC websites. Given that Job Bank has expanded over the years due to the merger and the addition of new tools and linkages to other websites, the dissemination of more and easier to access information has been achieved with about the same resources as previous to the consolidation.

Figure 5 – Text description

Expenses in millions	2010 to 2011	2011 to 2012	2012 to 2013	2013 to 2014	2014 to 2015	2015 to 2016
NHQ  LLMI data collection	2.30	2.24	2.56	0.74	0.83	0.93
Regional LLMI data collection	12.65	12.48	11.43	9.16	8.15	8.99
NHQ  LLMI data dissemination*	1.35	0.79	0.85	0.24	0.21	1.03
Regional LLMI data dissemination	1.40	1.36	1.27	1.02	0.91	1.00
COPS**/Labour Market monitoring	0.17	0.21	0.14	0.19	0.20	0.23

* Includes LMI Council and Data purchase effective 2016.
** COPS is for Canadian Occupational Projection System.
Source: ESDC, CFOB internal document.

Figure 5 illustrates the distribution of ESDC expenditures on various data collection activities. Regional data collection expenditures have remained a significant component of this expenditure category.^{Footnote 20}

4.3.2 Alternative service delivery models in the provision of learning and labour market information

Overall, the evaluation findings indicate that the current delivery model successfully provides learning and labour market information to Canadians. In addressing one of the evaluation questions and to gather knowledge on various alternative delivery approaches, three models were identified.

Open Data

An alternate delivery approach is an open data model such as the one launched in 2015 by the UK Commission for Employment and Skills, LMI for All.^{Footnote 21} In this model, the federal government focuses on the provision of LLMI through open data formats for use by other organizations that develop tools and products to disseminate. Such a model would be challenging in the Canadian context given that the federal government will still retain the responsibility of ensuring that all segments of the population have equal access to learning and labour market information.

Separate agency

Another model is that of a separate federal agency to coordinate the collection, analysis and dissemination of learning and labour market information. This model exists in the United Kingdom, where Unistats provides information to students about education through a partnership of government agencies. Experts and key informants saw the Forum of Labour Market Ministers potentially playing a similar role as the agency in the United Kingdom. The main concern with this approach is its governance and whether such an agency would be able to meet the needs of federal and provincial/territorial governments. Some key informant interviewees saw a hybrid model as the future delivery model of LLMI, with Provinces and Territories continuing to provide the information directly to certain categories of job seekers. It is worth mentioning that a Labour Market Information Council has been recently created by the federal, provincial and territorial governments to move forward with the Forum of Labour Market Ministers.

Sector-driven delivery of Labour Market Information

Taking a specialized view focusing on a single industry is exemplified by BuildForce Canada (formerly the Construction Sector Council). This national industry-led organization funded by ESDC has been working with the construction industry to provide accurate and timely LMI for over a decade.^{Footnote 22} It uses a scenario-based forecasting system to assess future labour market conditions, tracking measures for 34 trades and occupations. To validate the scenario assumptions, BuildForce consults with industry, including owners, contractors and labour groups, and seeks input from government on related analysis. Construction Forecasts,^{Footnote 23} BuildForce Canada’s online LMI delivery system, provides access to the data series that make up forecast scenario. All data produced by BuildForce Canada is available to members and strategic partner organizations. However, others must purchase access to the data and/or reports.

5.0 Conclusion and recommendations

The evaluation findings revealed that ESDC’s decision to disseminate learning and labour market information using a web-based consolidated approach was appropriate and timely. Overall, Job Bank users were satisfied with their overall experience with the site, but the experience of target groups (especially recent immigrants, youth, Indigenous people) could be ameliorated by providing them with learning and labour market information tailored to their specific needs, in a variety of formats, and using plainer language. It is important to promote Job Bank and the various types of learning and labour market information available on its site to Canadians and in particular for youth who have little experience in the workforce.

Job Bank, as its name suggests, is primarily known and used as a job board where job seekers can search for employment and employers can post job vacancies. Still, there are an increasing number of private job boards (for example “Indeed.ca”) providing similar types of labour exchange information and services. To remain relevant as a provider of LLMI, Job Bank may consider raising its awareness among job seekers and employers.

The awareness and use of Job Bank as a provider of other forms of learning and labour market information (such as career planning or labour market trends and news) remain low. In particular, youth could benefit from greater awareness of Explore Careers and the Career Tool to make informed decisions related to their education, immediate job prospects, and long term career planning.

Overall, Job Bank users found that the information made available to them is up-to-date and easy to access. The quality and the quantity of the information are improving, but focus in near future is to ensure more availability and accessibility of relevant and up to date local information.

To provide comprehensive learning and labour market information, Job Bank could continue to provide links to external information sources as well as continue to seek partnerships with external stakeholders. Performance monitoring is another area in which Job Bank could improve by developing user satisfaction and process management standards, supported by targets, to ensure users’ needs are adequately met.

Recommendations

1. Further develop awareness (especially among students, youth and their influencers) of the Job Bank website and the learning and labour market information it provides for various users and target groups
2. Continue to consider the needs of various target groups (for example given differences in their literacy levels, preferred presentation styles, unique information needs) in the development and delivery of learning and labour market information in formats which are easy to access and use
3. Continue to find innovative ways to collect and provide users with learning and labour market information which reflects local realities (for example, data based on smaller geographic locations)
4. Continue to find ways to support employers in posting jobs on Job Bank so as to more effectively meet their hiring needs in a timely manner.
5. Monitor client satisfaction on a regular basis to continuously improve user experience and to ensure that feedback mechanisms are in place to inform further product development.

Annex 1: Evaluation questions

1. Does the learning and labour market information (LLMI) provided by Employment and Social Development Canada (ESDC) address a demonstrated need among Canadians generally and specifically among target groups?
2. Is ESDC’s web-based consolidated delivery approach to disseminating LLMI to Canadians an appropriate mechanism in meeting their needs?
3. How is the dissemination of LLMI through a web-based consolidated approach related to departmental (ESDC) and government priorities?
4. Is the dissemination of LLMI consistent with the federal government’s roles and responsibilities?
   1. What is the role of the federal government vis-à-vis other orders of government, private sector/industry and other LLMI intermediaries in the provision of LLMI to the public?
   2. How is the role of the federal government changing in the provision of LLMI to the public?
5. To what extent is ESDC’s web-based consolidated LLMI information is:
   1. Up-to-date?
   2. Standardized?
   3. Easy (for users) to find and use?
   4. Providing integrated, quality and comprehensive LLMI to Canadians, including target groups?
   5. Considered authoritative by Canadians?
   6. Building awareness of educational choices and labour market options? (for example exploring careers by occupation, education module, suggestions of related occupations on each job search, etc.)
6. What is the client experience, including satisfaction, of the provision of LLMI and its various products? Are there differences between target groups?
7. How does the current model of delivery of LLMI impact the provision of information (for example by region, occupation, etc.)?
8. Have there been unintended outcomes of the consolidated approach to providing LLMI?
9. What is the effectiveness and efficiency of ESDC’s web-based consolidated approach to providing LLMI?
   1. Were there changes in resources allocated to providing LLMI?
   2. Did these changes impact outputs (for example quantity of information, quality of information, speed of disseminating information, etc.)?
   3. What was the effectiveness of the web-based consolidated approach’s data integration, retrieval and management mechanisms?
10. Are there alternative service delivery models in the provision of LLMI? (for example devolving services to P/Ts, private sector/industry, etc.)

Annex 2: Evaluation technical reports and data sources

Technical report	Line of evidence
1) Surveys on Job Bank Technical Report	Telephone survey of users and non –users of employers with a Job Bank account (n=456) and job seekers who experienced a recent job separation (n=1749). The telephone survey was complemented with a survey of employers (n=45) obtained from Info Canada – a private business directory and an intercept survey of job seekers at the exit of employment service centres (n=41) The surveys were conducted as part of this evaluation
	Pop-up survey of users of Job Bank site (n=26673) invited to complete a survey during their visits on the site. Among them: 23,274 were job seekers; 1,672 students; 618 education seekers;Footnote 24 and 1,109 employers.
2) Focus Group Technical Report	Focus group discussions (16 in total) with employers (17 participants), job seekers (24 participants), students (24 participants)  and education seekers (17 participants) who use Job Bank site;
3) Expert Panel Technical Report	Expert panels with leading economists, web designers and youth/education counsellors:
4) Key Informant Interview Technical Report	Key informant interviews with 29 ESDC program representatives at Headquarters and in Services Canada regions as well as with 12 provincial and territorial representatives.
5) Technical Analysis of Working in Canada/Job Bank Web Analytics	Analysis of the Web analytics of Job Bank web site
6) Document Review of the Learning and Labour Market Information (LLMI) Evaluation	Document review of several domestics and international sources
7) A Quantitative Data Analysis of the 2013 and 2014 Canadian Out-Of-Employment Panel Surveys Technical Report	Quantitative data analyses of the 2013 (n=3469) and the 2014 (n= 1763) Canadian Out-of-Employment Panel (COEP) Surveys. The COEP survey was administered annually by a third party on behalf of ESDC between 1995 and 2015.  The survey collected responses from a relatively large sample of recently unemployed individuals on a wide range of labour-related information

Annex 3: Major changes made to Job Bank and learning and labour market information activities since 2005

Since the last 2005 evaluation, ESDC has under taken the following initiatives till the data collection for this evaluation was completed in August 2016:

1. Launch of Working in Canada in 2007;
2. In 2009, publication of results of the “Advisory Panel on Labour Market Information (2009). Working Together to Build a Better Labour Market Information System for Canada”;
3. Realigning Working in Canada to provide LLMI to Canadians as opposed to focusing only on potential immigrants (2010) increased information on labour market information to all Canadians;
4. Developing the National Work Plan (NWP) for LMI. The goal of the NWP was to ensure that Canadians had access to regional LMI that is standardized, high quality, and easy to use (2010); Standardization of methodologies is now used to produce LLMI that is comparable across regions and at national level;
5. Launching the Job Alert tool (2012) which provides job seekers with notifications of available jobs according to user’s profile and improve the Employment Insurance system;
6. Bringing higher paid jobs to Job Bank and increase the range of job types accessible by job seekers on the site by developing partnerships with Workopolis and Monster (2013);
7. Partnering with provinces (Quebec, British Columbia, Newfoundland and Labrador, and Saskatchewan) to facilitate access to jobs available in those provinces through the Job Bank;
8. Merging the WiC and Job Bank under the Job Bank name in order to assist users to access LLMI through one consolidated website (2014);
9. Working to make Job Bank’s LLMI accessible to all categories of users by ensuring that materials are in both official languages and available in various accessible formats. (Ongoing);
   1. Launching Job Match and Career Tool;
   2. Working with provincial and territorial governments as well as private sector partners to coordinate improvements to LLMI in Canada; and
   3. Funding the Education Policy Research Initiative (EPRI) to produce new information on the returns to education by using existing administrative data.

Annex 4: References to learning and labour market information in Budgets 2012 to 2016

• Budget 2012 devoted $21 million over two years for enhancing the content (type of content, quantity of information and accuracy); and timeliness of the job and labour market information for job-seekers;
• Budget 2013 reallocated $19 million over two years offering Job Alert service to Employment Insurance recipients; and informing young people about fields of study that are relevant to existing and forecasted demand for labour in particular occupations;
• Budget 2014 invested $11.8 million over two years and $3.3 million ongoing to enhance Job Matching to job seekers and employers;
• Budget 2015 confirmed new investments in enhanced LLMI, with the reallocation of $4M over two years to support the launch of a new one stop national labour market information portal, the launch of a new labour mobility portal and the new Career Tool, and an improved Job Bank that will make it easier for workers to find information on job opportunities outside their local geographic area; and
• Budget 2016 stressed the fact that access to timely, reliable, and comprehensive labour market information is critical to ensure that all Canadians, including students, workers, employers and educators, have the information they need to make informed decisions.