Evaluation of the Museums Assistance Program 2013-14 to 2017-18

Evaluation Services Directorate
July 19, 2021

List of tables
List of figures
List of acronyms and abbreviations
Executive summary
1. Introduction
2. Program profile
3. Approach and Methodology
4. Findings
5. Conclusions
6. Recommendations, management response and action plan
Annex A: the current state of museums - looking towards the Future
Annex B: evaluation framework
Annex C: program logic model
Annex D: bibliography

List of tables

Table 1: program objectives and expected outcomes
Table 2: MAP’s target population, stakeholders and delivery partners
Table 3: MAP Grants and Contributions (Vote 5) actual spending by component, 2013-14 to 2017-18 ($)
Table 4: MAP Operational funds, (Vote 1), including all Heritage Programs, 2013-14 to 2017-18 ($)
Table 5: core evaluation issues and questions
Table 6: evaluation limitations
Table 7: number of funded and non-funded applications, 2013-14 to 2017-18
Table 8: number of projects funded by the Indigenous Heritage component
Table 9: number of projects that enhanced professional knowledge, skills and practices, by component, 2013-14 to 2017-18
Table 10: MAP administrative ratio for the regions, based on actual operational and administrative costs (Vote 1) and G&Cs (Vote 5), Big MAP, 2013-14 to 2017-18 ($)
Table 11: MAP planned and actual expenditures (Vote 5), 2013-14 to 2017-18 ($)
Table 12: achievement of service standards for acknowledge of receipt of funding applications and making funding decisions
Table 13: percentage of requested amounts approved before/after implementation of notional allocations, by region
Table 14: recommendation 1 – action plan
Table 15: recommendation 2 – action plan
Table 16: recommendation 3 – action plan

List of figures

Figure 1: MAP components and subcomponents
Figure 2: number of travelling exhibitions compared to population distribution, 2013-14 to 2017-18
Figure 3: number of exhibitions presented, 2013-14 to 2017-18
Figure 4: number of host venues that took place in Canada, by province/territory
Figure 5: number of visitors to MAP-funded exhibitions, 2013-14 to 2017-18

Alternate format

Evaluation of the Museums Assistance Program 2013-14 to 2017-18 [PDF version - 1.19 MB]

List of acronyms and abbreviations

AH: Access to Heritage
CCI: Canadian Conservation Institute
CM: Collections Management
CMA: Canadian Museums Association
DRF: Departmental Results Framework
EA: Emergency Assistance
EBP: Employee Benefit Plan
ECF: Exhibition Circulation Fund
ESD: Evaluation Services Directorate
EWG: Evaluation Working Group
EXCOM: Executive Committee
FTEs: Full-time equivalents
GBA Plus: Gender-Based Analysis Plus
G&Cs: Grants and contributions
GCIMS: Grants and Contributions Information Management System
HPPB: Heritage Policy and Programs Branch
MAP: (aka “small” MAP) Museums Assistance Program
IH: Indigenous Heritage
MAP: (aka “big” MAP) Museums Assistance Transfer Payment Program
O&M: Operations and maintenance
OLMC: Official Language in Minority Community
PCH: Canadian Heritage
PIP: Program Information Profile
PMERS: Performance Measurement, Evaluation and Risk Strategy
PRG: Policy Research Group
PTMAs: Provincial and territorial museum associations
T&Cs: Terms and conditions
TB: Treasury Board
TRC: Truth and Reconciliation Commission
VNR: Vote-netted revenue

Executive summary

Program description

"The Museums Assistance Program (MAP) fosters the preservation of Indigenous culture and facilitates access to heritage collections for all Canadians. It promotes professional knowledge, skills and practices related to key museum functions."^{Footnote 1} MAP provides financial assistance ($6.7 million annually) to Canadian museums and related institutions for activities that aim to:

Facilitate Canadians’ access to their heritage.
Ensure the preservation, management and presentation of Indigenous cultural heritage.
Strengthen professional standards and enhance competencies in the management of key museum functions.^{Footnote 2}

Evaluation approach and methodology

The evaluation was undertaken in accordance with the Departmental Evaluation Plan for 2019-20 to 2023-24, approved by the Deputy Minister, and covered a five-year period, from 2013-14 to 2017-18. The evaluation was originally scheduled to be completed in May 2020, but finalization of the report was delayed due to the COVID-19 pandemic.

The evaluation examined the core issues addressed by most evaluations: relevance, effectiveness and efficiency. The evaluation methodology involved a review of the program’s administrative and performance data; a review of relevant documents; interviews with program personnel, external stakeholders and funded applicants; the two case studies; and, a review of relevant literature.

With the COVID-19 pandemic and subsequent lockdowns for cultural and heritage industries, a brief environmental scan of the museum sector was added to the project to ensure the relevance of evaluation findings to current and emerging realities (Annex A of the evaluation report).

Findings

Relevance

MAP is responding in part to the current and evolving needs of museums. Canadian museums face a variety of challenges, including those related to operational funding and human resources pressures, adopting new technologies and digitization of collections, and building closer relationships with Indigenous communities. MAP’s limited budget envelope, its narrow focus and its restrictive eligibility criteria for small museums are factors that impact its ability to completely address needs.

MAP supports such government priorities as reconciliation, GBA Plus and supporting official language minority communities (OLMCs). Since 1990, MAP has administered an Indigenous Heritage component and following the 2015 Truth and Reconciliation report, Canada’s residential schools: the final report of the Truth and Reconciliation Commission of Canada, it increased its flexibility and funding limits to this component. As a result of call to action 67 of the report, the program also has provided funding to the Canadian Museums Association to conduct, in collaboration with Indigenous peoples, a national review of museums’ policies and best practices to determine the level of compliance with the United Nations Declaration on the Rights of Indigenous Peoples and to make recommendations. The program funds projects undertaken by museums located in OLMCs and the funded exhibitions feature bilingual content.

MAP performance data on the department’s GBA Plus priorities is not available in PCH’s Grants and Contributions Information Management System (GCIMS). It is not possible to assess the extent to which MAP and the funded projects are serving various priority, aside from OLMCs and Indigenous groups and projects.

Effectiveness

Overall, MAP is achieving expected results and continued to contribute to the achievement of the program’s ultimate outcome of making Canada’s heritage accessible to the Canadian public.

A total of 269 projects to produce and/or circulate, exhibitions were approved for funding during the five-year evaluation period. MAP funded a total of 384 heritage exhibitions over the five years, travelling and non-travelling combined. A total of 719 venues hosted exhibitions, of which 60% were located in Ontario and Quebec. Over 6.6 million visitors experienced these exhibitions over the five years, an average of over 1.33 million visitors per year. More than 6,500 public programming activities and over 1,700 public programming products accompanied the exhibitions during the five years.

MAP’s Indigenous Heritage component contributed to improving the preservation and presentation of Indigenous heritage through funding of 82 projects over the five years. The evaluation identified challenges related to the awareness of the programming and communication with Indigenous communities as well as the great number of needs within this sector, many of which are not eligible for MAP funding.

MAP funding has led to over 1,000 quality training opportunities for professionals in the Canadian museum sector aimed at increasing knowledge and improving practices. Funding for professional development, skills and practices is provided primarily through the Collections Management component, and to a lesser extent through the Indigenous Heritage component and the Canada-France agreement. The RE-ORG: Canada initiative with the Canadian Conservation Institute has helped several museums improve their museum storage practices. Over 9,000 heritage workers from just over 3,000 institutions participated in the training opportunities over the five years. Surveys administered after these activities have found that three-quarters (76%) of these workers reported that their level of professional knowledge, skills and practices had improved.

MAP funding has had a considerable impact on the improvement of collections management and preservation of objects in general. Projects were funded by the Collections Management and Indigenous Heritage components. MAP funding for training and professional development capacity had an indirect impact on this same objective. Approximatively 1.5 million objects were managed and/or preserved under the Collection Management and Indigenous Heritage components over the five years, for a total of 86 approved projects to improve the management of heritage objects and collections, and 141 approved projects to improve the preservation of heritage objects and collections.

Virtually all clients reported at least some improvement in the management and preservation of heritage objects and collections, with approximately one-third reporting substantial improvements.

Efficiency

MAP was delivered in an efficient manner. It has an administrative ratio of 15%, which is an improvement from 22%, the ratio from the period covered by the previous evaluation. The program has largely met the service standards set by the Department. Delegation of authority to the Regional Directors General may have contributed to improvements in efficiency.

The evaluation found that the decision by the program to allocate funding to the regions, using a notional allocation method based on population distribution by region, is an efficient approach. This method resulted in improved financial management and facilitated the work of review committees.

The program has made progress in performance measurement, notably by developing a methodology to report on program results. However, the data analysis methodology is complex and does not fully align with data in GCIMS, the program logic model and Performance Information Profile (PIP). As well, the program does not report on all priority groups, other than OLMCs and Indigenous groups and projects.

Other – Innovation

The RE-ORG: Canada project had a limited scope but had a positive impact on museums and their collections. The innovative nature of this project fostered the acquisition of practical knowledge and networking among members of the museum community. MAP’s involvement was deemed to be essential to museum participation. A few issues were identified by key informants, such as the considerable amount of time required to complete the training. It was suggested that the initiative be broadened to address other needs of museums.

Recommendations

Recommendation 1:

The ADM, Official Languages, Heritage and Regions, should build upon the findings of this evaluation to further examine and identify mechanisms to address the current and emerging contexts and priorities affecting museums, including through enhanced engagement and communication with Indigenous communities to better understand their needs and realities.

Recommendation 2:

The ADM, Official Languages, Heritage and Regions, should work with the respective stakeholders to:

Simplify the application process as part of the move to an online portal and;
Examine the intake cycle.

Recommendation 3:

The ADM, Official Languages, Heritage and Regions, should work with Strategic Planning and other internal stakeholders to:

Review the MAP performance indicators to align with government orientation with respect to priority groups and implement them;
Ensure that the program logic model is aligned with the program results and performance set out in the PIP and reporting methodology, and;
Review the program’s reporting methodology to ensure that is reproducible and clear.

1. Introduction

This report presents the findings, conclusions and recommendations resulting from an evaluation of the Museums Assistance Program (MAP), a program administered by the Department of Canadian Heritage (PCH). The evaluation was conducted as prescribed in the Departmental Evaluation Plan 2019-20 to 2023-24. It covers the five-year period 2013-14 to 2017-18 and examines the relevance, effectiveness and efficiency of the program.

The study was undertaken by the Evaluation Services Directorate of PCH and was designed and conducted in accordance with the Treasury Board Policy on Results (2016) and the Directive on Results (2016). It was originally scheduled to be completed in May 2020, but finalization of the report was delayed due to COVID-19.

This report describes the program, the evaluation scope, questions and methodology, as well as the evaluation findings, conclusions, and the recommendations.

2. Program profile

2.1. Program history

Public interest in Canada’s heritage increased following the Centennial in 1967. In response, the Government of Canada introduced the National Museum Act in 1968, which created Canada’s first museum policy and a museum corporation to expand the role of museums nationally.^{Footnote 3} In 1972, the implementation of the National Museum Policy formalized the role of the national museums and had, as its primary objective:

"To better distribute those cultural resources which are obtainable through Canadian museums both national and regional to the end that the greatest possible number of Canadians be exposed to our national heritage."^{Footnote 4}

The new National Museums of Canada Corporation administered the policy, which provided national museums with grants to enable them, for example, to build new facilities and to purchase artifacts.^{Footnote 5} Along with the National Museum Policy, MAP was created in 1972 in order to allocate grants to museums in Canada for different activities including travelling exhibitions and professional development.

In 1990, the Museums Act dismantled the National Museums of Canada Corporation and created four operationally independent Crown corporations: the National Gallery of Canada, the Canadian Museum of Civilization, the National Museum of Natural Sciences and the National Museum of Science and Technology. A new museum policy, released that same year, re-affirmed the government’s commitment to increase Canadians’ access to their heritage and the necessity for ongoing government support. The objectives of the new policy were to foster Canadians’ access to and awareness of their heritage, to encourage the management, development and preservation of Canadian museum collections, and to enhance excellence in museum activities through a variety of means.

Within the 1990 policy, the government committed to increasing the funding for museums, including an increase to MAP funding, from $10.4 million in 1990 to $18 million in 1994-95. The policy stated the importance of the MAP program as the “Government’s main instrument of support for museums.” There has not been a new museum policy since 1990, and MAP funding, while increased, never reached the intended $18 million. Funding has been $6.7 million since 2011-12.

The Calls to Action from the Truth and Reconciliation Commission in 2015 include Action #67, which calls on the federal government to fund the Canadian Museums Association (CMA), with the collaboration of Indigenous peoples, to undertake a national review of museum policies and best practices. As part of the project, the CMA established a Reconciliation Council to provide guidance in determining compliance with the United Nations Declaration on the Rights of Indigenous peoples, and to produce clear and realistic recommendations for Canada’s heritage sector. The program also raised its funding limits within its Indigenous Heritage component.

In 2018, the Standing Committee on Canadian Heritage for the House of Commons prepared a report on the state of museums in Canada^{Footnote 6}. Among 15 recommendations tabled for consideration by the House of Commons, the committee recommended that PCH review and modernize the National Museum Policy (recommendation 3), and that PCH expand MAP to create more sustainable funding (recommendation 5).

2.2. Program objectives, expected outcomes and activities

MAP "supports heritage institutions and workers in the preservation and presentation of heritage collections. The Program fosters the preservation of Indigenous culture and facilitates access to heritage collections for all Canadians. It also promotes professional knowledge, skills and practices related to key museum functions."^{Footnote 7} It is one of three programs under the Museums Assistance Transfer Payment Program (often referred to as the “big MAP”) that provides support to the museum community. The other two programs are Young Canada Works-Heritage, and Movable Cultural Property Grants. While the three programs are grouped together under one transfer payment program to streamline program administration, they are evaluated independently of one another.

The program’s objectives and immediate and intermediate outcomes are listed in Table 1. The program’s long term or ultimate result is "Canada’s heritage is accessible to Canadians over time".

Table 1: program objectives and expected outcomes

Objectives	Immediate outcomes	Intermediate outcomes
Canada’s Heritage is accessible to Canadians over time	Travelling exhibitions are produced	Heritage exhibitions and other public programming products/activities are presented to the public
	Projects to improve the preservation and presentation of Indigenous heritage are implemented	Heritage objects and collections are better managed and preserved
	Opportunities are created for heritage institutions and workers to enhance their professional knowledge, skills and practices	Heritage institutions and workers have enhanced their professional knowledge, skills and practices

Source: 2012 PCH PMERS

As shown in Figure 1, MAP provides financial assistance to Canadian museums and related institutions for three main types of activities that aim to:

Facilitate Canadians’ access to their heritage.
Ensure the preservation, management and presentation of Indigenous cultural heritage.
Strengthen professional standards and enhance competencies in the management of key museum functions.^{Footnote 8}

**Figure 1: MAP components and subcomponents**

Access to Heritage

This component "promotes access to heritage across different geographic regions of Canada."^{Footnote 9} It "provides funding to heritage organizations for travelling exhibitions in Canada to promote access to heritage across different geographic regions."^{Footnote 10} This component normally funds a maximum of 70% of eligible expenses, up to $200,000 per fiscal year.

This component also includes a subcomponent, the Exhibition Circulation Fund, which assists museums with the costs of hosting travelling exhibitions originating from another museum or from a federal heritage institution, including borrowing artefacts for exhibitions from the Canadian War Museum and the Canadian Museum of History.^{Footnote 11} The Exhibition Circulation Fund subcomponent supports a maximum of 70% of eligible expenses, up to $15,000 per project.

Indigenous Heritage^{Footnote 12}

This component provides funding for projects related to the preservation, presentation, and management of Canada's Indigenous cultural heritage. Eligible activities include research and interpretation of Indigenous cultural heritage, internships and exchanges between Indigenous and museum institutions for professional development initiatives; and creation, promotion and presentation of exhibitions and associated interpretive material. Applications from non-Indigenous heritage organizations are considered if close collaboration with an Indigenous group is demonstrated. This component supports a maximum of 70% of eligible expenses and up to $100,000 per fiscal year.

Collections Management^{Footnote 13}

This component provides funding for projects that aim to improve professional knowledge, skills and practices and to strengthen professional standards related to key museum functions for collections management in Canada. It also supports the development and delivery of resources or services that will benefit multiple museums.^{Footnote 14} This component supports a maximum of 50% of eligible expenses, with up to $50,000 for collections management systems projects and up to $100,000 for best practices projects per fiscal year.

This component includes the Canada-France Agreement^{Footnote 15}, which aims to develop special, lasting ties between museums in Canada and France, as well as enhancing the skills and competencies of museum professionals. It also assists Canadian organizations in developing new international partnerships and in reaching new potential audiences for Canadian heritage collections."^{Footnote 16} Partnerships may relate to research on development and diversification of audiences. Projects that further the development of official language minority communities are given a priority for funding support.

MAP’s target population, Canadian museums and related organizations, as well as its stakeholders and delivery partners, are outlined in Table 2.

Table 2: MAP’s target population, stakeholders and delivery partners

Target population	Professionally-managed Canadian museums and related organizations that meet the eligibility criteria as defined in MAP’s Terms and Conditions. They include: Incorporated non-profit Canadian museums and related service organizations and other heritage organizations. Indigenous Governing Bodies and Indigenous organizations with a mandate related to heritage.
Key stakeholders	Recipient heritage institutions, museum associations and networks, and related service organizations. Heritage workers who benefit from professional development opportunities.
Delivery partners	PCH’s five regional offices are responsible for establishing and maintaining relationships with MAP clients and key stakeholders within their regions. The PCH National office is responsible for MAP’s overall integrity and accountability.

Target population

Professionally-managed Canadian museums and related organizations that meet the eligibility criteria as defined in MAP’s Terms and Conditions. They include:

Incorporated non-profit Canadian museums and related service organizations and other heritage organizations.
Indigenous Governing Bodies and Indigenous organizations with a mandate related to heritage.

Key stakeholders

Recipient heritage institutions, museum associations and networks, and related service organizations.
Heritage workers who benefit from professional development opportunities.

Delivery partners

PCH’s five regional offices are responsible for establishing and maintaining relationships with MAP clients and key stakeholders within their regions. The PCH National office is responsible for MAP’s overall integrity and accountability.

Sources: 2019-20 Performance Information Profile and 2015 MAP Evaluation Report.

2.3. Program management and governance

MAP is under the responsibility of the Heritage Policy and Programs Branch (HPPB) of the Heritage Group, within the Official Languages, Heritage and Regions sector of PCH. The program is managed at PCH headquarters and is coordinated and delivered through the five regional offices, with support of the district offices.^{Footnote 17}

PCH headquarters is responsible for MAP’s overall accountability for the grants and contributions budget and for the initial allocation of funds to the regions. It also manages projects led by national clients. The National Office develops, reviews and distributes program application material, including the common administration and communication tools and procedures to meet PCH requirements. It provides guidance to the regional offices to ensure a consistent interpretation of program guideline. It undertakes policy and program development, monitors trends and issues in the heritage sector and is responsible for program analysis, performance measurement and reporting, as well as the coordination of corporate requests related to the program.^{Footnote 18} The overall accountability for MAP is with the Director General of the Heritage Group.^{Footnote 19}

The regional offices are responsible for establishing and maintaining relationships with MAP clients and key stakeholders within their regions, promoting the program to potential applicants, assessing funding applications based on national evaluation instruments, managing and monitoring funded projects, collecting funding recipients’ reports and essential data for results measurement, and monitoring funding recipient audits conducted in their respective regions.^{Footnote 20} The accountability for MAP’s notional allocations, meaning its operations and maintenance (O&M) and grants and contributions (G&Cs), is with the five Regional Directors General.^{Footnote 21}

MAP operates through the combined efforts of HPPB and regional offices. Information is shared, tools are developed in collaboration, and strengthening program capacity is a collective exercise. Working groups are created on an ad hoc basis to improve different aspects of the program such as guidelines and evaluation tools.^{Footnote 22}

Prior to 2016, MAP’s objectives and activities contributed to the strategic outcome "Canadian artistic expressions and cultural content are created and accessible at home and abroad" of the Program Alignment Architecture for the Heritage Program. Since 2016, the Program has aligned with the Departmental Results Framework (DRF). Hence, Preservation of and Access to Heritage is MAP’s link to PCH’s Program Inventory, and Heritage and Celebration as its Core Responsibility.^{Footnote 23}

2.4. Program resources

Table 3 provides a breakdown of the program’s G&Cs expenditures (Vote 5), while Table 4 presents the Vote 1 Operational expenditures, consisting of salaries, Employee Benefits Plan (EBP), O&M and vote-netted revenue (VNR).

Table 3: MAP Grants and Contributions (Vote 5) actual spending by component, 2013-14 to 2017-18 ($)

Fiscal Year	Access to Heritage^{Table 3 note *}	Indigenous Heritage	Collections Management^{Table 3 note **}	Total
2013-14	3,340,781	1,463,584	1,927,744	6,732,109
2014-15	3,284,585	1,782,397	1,689,452	6,756,434
2015-16	3,844,601	1,063,431	1,708,654	6,616,686
2016-17	4,034,986	1,589,428	1,317,917	6,942,331
2017-18	3,909,864	1,337,201	1,496,093	6,743,158
Total	18,414,817	7,236,041	8,139,860	33,790,718

Source: PCH Departmental Performance Reports and Departmental Results Report from 2013-14 to 2017-18; PCH Financial Management Branch Data

Table 3 notes

Table 3 note *

Access to Heritage includes the Exhibitions Circulation Fund subcomponent for all years as well as Borrowing of Artefacts projects from 2015-16 to 2017-18.

Return to table 3 note * referrer

Table 3 note **

Collections Management includes Canada-France Agreement subcomponent for all years as well as the Canadian Museums Association commitment item from 2013-14 to 2014-15.

Return to table 3 note ** referrer

Table 4: MAP Operational funds, (Vote 1), including all Heritage Programs, 2013-14 to 2017-18 ($)

Fiscal Year	Salary	EBP	O&M	VNR	Total
2013-14	956,749	191,350	46,942	4,476	1,199,517
2014-15	951,015	190,203	42,539	903	1,184,660
2015-16	889,693	177,939	34,980	4,335	1,106,947
2016-17	902,500	180,500	44,100	3,198	1,130,298
2017-18	874,064	174,813	44,823	9,934	1,103,634

Source: MAP financial data

Note: figures are not available only for MAP, as program officers, managers and directors work on multiple programs.

2.5. Results of the previous evaluation

The previous evaluation of the program was conducted in 2014-15 and covered the period from 2008-09 to 2012-13. Three recommendations were made to the Assistant Deputy Minister of the Citizenship, Heritage and Regions sector:

Undertake an environmental scan to ensure that the MAP's priority areas are aligned with prevalent challenges of the museum community with respect to project funding in an effort to optimize the impact of program funds.
Explore opportunities to increase the efficiency of the program.
Improve activities linked to the performance measurement strategy to ensure a more timely and effective means of collecting and analyzing performance information for decision-making purposes.^{Footnote 24}

All deliverables resulting from these recommendations have since been implemented, except for recommendation 2, which requires online integration of the application process.^{Footnote 25} Unfortunately, due to the discontinuation of the Grants and Contributions Modernization Project (GCMP), an initiative to support moving program applications online, this recommendation remains pending. It is planned that the program will be included in the move of the application process to an online portal scheduled for 2021-22.

3. Approach and Methodology

This evaluation was conducted by ESD in accordance with the requirements of the Treasury Board Secretariat Policy on Results (2016). It used multiple methodologies to gather quantitative as well as qualitative data and triangulate findings.

PCH senior management requested that the evaluation take place on an accelerated schedule – within eight months – to achieve the targeted completion date of May 2020. However, finalization of the evaluation report was delayed due to COVID-19.

3.1. Scope, timeline and quality control

The evaluation was undertaken in accordance with the standards with the Treasury Board’s 2016 Policy on Results, and covered a five-year period, from 2013-14 to 2017-18.

Scoping interviews were held with program representatives to determine their specific information needs and to refine the evaluation scope. As a result of these interviews, the evaluation examined two subjects in particular: the RE-ORG: Canada initiative with the Canadian Conservation Institute (CCI), and a process used by the program to allocate G&Cs budgets to the regions, called “notional allocation”. The evaluation team chose the case study method to examine these two areas, which involved a review of documents and administrative data and key informant interviews.

The evaluation did not examine the other two programs associated with “big MAP”: Young Canada Works-Heritage and the Movable Cultural Property Grants program. Furthermore, given that the previous evaluation undertook a comparison of programs that fund museums in other countries, this task was not repeated for the present evaluation. Due to the short timelines, a survey of non-applicants was not repeated for this evaluation.

The evaluation was conducted in-house by ESD evaluators. The work was conducted in a manner that is consistent with the Treasury Board Secretariat Policy on Results (2016) and the Directive on Results (2016) and was executed in a neutral fashion and with integrity in its interactions with stakeholders.

The following quality assurance measures were undertaken during the evaluation:

ESD held an introductory meeting for MAP representatives who were the primary contacts throughout the evaluation (“Evaluation 101”), in order to provide a clear understanding of all aspects of the evaluation process.
A variety of data collection methods were designed and implemented, in order to obtain information from multiple targeted sources of primary and secondary data, which helped to ensure that findings were reliable and defensible.
A systematic approach to the synthesis of evidence on each evaluation question was used.
The evaluation working group and ESD peers reviewed some of the deliverables. It should be noted that due to the pandemic lockdown, the evaluation working group met once at the beginning of the project to review the Terms of reference, including the evaluation questions and methodology, and then for preliminary findings presentation after the lockdown.

3.2. Calibration

Calibration refers to the process of adjusting how evaluations are conducted, based on a number of different factors, to produce quality evaluations cost-effectively and within timelines. Depending on the particular evaluation, calibration can involve adjustments that increase or decrease the required level of effort, scope or depth of analysis. It enables the evaluation team to validate data across a number of sources, and to increase the legitimacy of the results of the evaluation. Calibration can help evaluations focus on particular questions based on risks and needs.

As this evaluation was undertaken on shortened timeline, calibration was needed in the following areas:

As much as possible, existing data sources were used to provide information to support the analysis of evaluation questions and where gaps were identified, additional, targeted data collection was undertaken;
There was a reduced number of questions in the area of relevance, given this issue was thoroughly examined by the previous evaluation;
The lines of evidence were chosen to target specific areas of inquiry. For example, key informant interviews were targeted to address particular evaluation issues and questions and were limited in number;
Non-funded applicants and non-applicants were not surveyed. Therefore, national, provincial and territorial museum associations (PTMAs) were interviewed to capture the perspectives of their clients and members;
Data collection for the various lines of evidence were combined where possible for efficiency (for example, case study questions were included in key informant interviews with representatives of the organizations associated with each case study, so that one interview was conducted with the same individual instead of two);
Lines of evidence have been chosen in order to cross-reference questions, and validate any findings during the triangulation process;
The length of the final report was kept to under 50 pages.

3.3. Evaluation questions

Based on the scoping consultations and preliminary research, the questions addressed by this evaluation are listed in Table 5. Further details on the indicators, data sources and data collection methods associated with these questions are contained in the evaluation framework (Annex A).

Table 5: core evaluation issues and questions
Core Issues	Evaluation Questions
Relevance	Is MAP responding to the current and changing needs of museums and heritage institutions of varying sizes, and to their professionals’ requirements? Does MAP support government priorities such as reconciliation, GBA Plus and OLMCs?
Effectiveness	Did the MAP achieve its expected short-term, medium term and long-term outcomes? Were there any unanticipated outcomes or challenges as a result of MAP activities?
Efficiency	Was MAP delivered efficiently? Was the notional allocation of funding efficient? Does the performance measurement framework for MAP allow for efficient reporting on the program’s results?
Other	What lessons or best practices have been learned from MAP’s innovation projects? (RE-ORG: Canada initiative)

3.4. Data collection methods

The evaluation methodology included a combination of qualitative and quantitative data collection methods and used primary and secondary sources of information designed to address the evaluation issues and questions. The methods included the following:

Review of the program’s administrative and file data, including performance data, Grants and Contributions Information Management System (GCIMS) data, financial data (STAR-SAP), and program databases.
Review of relevant program and Government of Canada documents including Budgets, Speeches from the Throne, Treasury Board submissions, directives and procedures, as well as relevant policies and legislation, and the results of internal PCH research.
Interviews with PCH program personnel, external stakeholders (including the PTMAs), and funded applicants.
Two case studies focused on the RE-ORG: Canada initiative conducted in partnership with the CCI and the program’s approach to allocating G&Cs budgets to the regions.
A targeted review of relevant literature, including mainstream and social media, related to current museum context and environment.

These methods are briefly described below.

Administrative review

The program provided excel spreadsheets with performance data covering the evaluation period. These reports included a program performance analysis from the program, covering the entire five-year period. The evaluation team received financial data on program Vote 1 and 5 expenditures from PCH Finance and the Hub, and gathered other administrative data including roll-ups of performance data through GCIMS, in collaboration with the Centre of Expertise. The team reviewed specific files to view reports which responded to particular evaluation questions. The methodology used by the program in their data analysis was not available at the time of the administrative review, though it was received by the evaluation team at the time of the writing of the report.

Document review

The evaluation team used the document review primarily for questions related to relevance and effectiveness. The documents included program policy documents and statements, Treasury Board submissions, Memoranda to Cabinet, terms and conditions, related legislation, as well as research and analysis undertaken by the program, department, or related organizations, and presentations and outreach communications, as well as program application guidelines and forms. The team used research findings from 2015 and 2017 Surveys of Heritage Institutions as well as the program’s report on two focus groups of external stakeholders. The goal of the focus group research was to explore the extent that MAP responded to the museum community’s needs. The team also reviewed an environmental scan of challenges faced by the museum community conducted by the program as a result of a recommendation from the previous MAP evaluation.

Literature review

The literature reviewed included external studies, articles, opinion editorials, media releases, and others, on the status of the museum sector in Canada and on the relevance and performance of MAP. A key source was the 2018 report prepared by the House of Commons Standing Committee on Canadian Heritage that examined the state of Canada’s museums. That report made 15 recommendations, including expanding MAP to create more sustainable funding and reviewing and modernizing the National Museum Policy. The report was a primary document for understanding the current challenges and issues being faced by museums of all sizes. It allowed the evaluation team to triangulate findings with the key informant interviews undertaken with MAP clients and the PTMAs. The bibliography and notes are provided in Annex D and Annex E respectively.

Key informant interviews

Key informant interviews were undertaken with funded applicants to capture perceptions and opinions of MAP. Internal interviews were also undertaken with PCH national and regional program staff. The accelerated timeline for this evaluation did not allow for a survey of funded and non-funded applicants; therefore, interviews were conducted with the PTMAs, as well as the national association. The interviews with the PTMAs provided the evaluation team with data related to the current context for small to large museums, for non-funded and non-applicants, and for the needs of the community in general. They also provided an overview of any regional variances in the community context. This approach was chosen in order to provide a range of perceptions on the program’s overall delivery and outcomes, to enable triangulation of results between clients, staff, and PTMAs, and to enable evaluators to measure any gap or improvements in delivery from the last evaluation period. In total, the evaluation team conducted 41 key informant interviews.

Case studies

Case studies were undertaken to address two specific areas of inquiry identified as important through scoping interviews with programs. The first case study examined the RE-ORG: Canada initiative, which was the result of a partnership between MAP and the PCH Canadian Conservation Institute (CCI). This case study provided the evaluators with information related to the reach and impact of MAP, as these pertain to collections management and training. The second case study was associated with the decision of the program to change the delivery of funds from a demand-based system, with all recommendations from the regions then being approved at the national HQ level, to notional regional allocations made by National Office to the Regional Offices. This case study compared the efficiency of the two approaches. It enabled the evaluation team to assess trends in efficiency, using analysis from the previous MAP evaluation, completed in 2015.

3.5. Constraints, limitations and mitigation strategies

The following table outlines the main challenges faced by this evaluation and the resulting mitigation strategies.

Table 6: evaluation limitations
Limitation	Mitigation Strategy
Targeted data collection methods	The evaluation relied on a limited number of lines of evidence. Targeted data collection methods may limit the quantity of evidence found for each evaluation question. Triangulation of data for each question helped to ensure that the findings are valid.
Non-applicants and non-funded applicants were not consulted	The perspectives presented in this report may not be representative of the entire museum sector. However, conducting interviews with PTMAs in the Canadian heritage context helped to mitigate this limitation. These interviews provided information on the issues faced by their museum members, including MAP clients and non-clients.
The COVID-19 pandemic impacted the evaluation team’s access to the program and to the PCH network	As a result of the COVID 19 pandemic, and the work-from-home mandate for all federal workers, PCH enacted their Business Continuity Plan (BCP) to enable essential services within the department to have access to the network, and to enable programs to deliver emergency funding to their program clients. The evaluation team was unable to contact programs or use the network from mid-March 2020 to September 2020. The team continued to work on the evaluation, which had collected all data with the exception of data related to the program’s methodology for analyzing their performance results.
The impacts of COVID-19 on the museums sector were not examined	Given that the evaluation covered the period of 2013-14 to 2017-18, the COVID-19 pandemic fell outside of the examination period. However, the evaluation team worked with consultant to complete an environmental scan that focused on the impacts of COVID-19 on the museums sector (Annex A).
It was not possible to validate the program’s data and data analysis methodology prior to the drafting of the report.	The detailed data analysis methodology used by the program for its five year performance analysis was not available to the evaluation team prior to the COVID-19 lockdown. While the data used in the analysis was provided, it was not possible to validate the data without the methodology as a guide. Therefore, the analysis draws from data that the evaluation team gathered from GCIMS (planned numbers), as well as the actual numbers provided by the program via their methodology. The variations in numbers do not affect the analysis and sources are identified in the tables and figures for clarity and context.

4. Findings

This section presents the findings on each of the evaluation issues and questions. A summary of the findings on each evaluation question is provided at the beginning of each sub-section.

4.1. Relevance

Evaluation Question 1.1. Is MAP responding to the current and changing needs of museums and heritage institutions of varying sizes, and to their professionals’ requirements?

MAP is responding in part to the current and evolving needs of museums. Canadian museums face a variety of challenges, including insufficient funding for operations, a lack of human resources capacity for both paid and volunteer staff, issues associated with adopting new technologies and digitization of collections, and the need to build bridges with Indigenous communities. MAP's limited budget envelope, its narrow focus and its restrictive eligibility criteria for small museums are factors that impact its ability to completely address needs.

4.1.1. Trends and issues facing museums

Museums and other heritage institutions (archives, art galleries, historic sites, etc.) are central to preserving and sharing Canada’s heritage, but they face many competing demands with limited resources. Museums and other heritage institutions preserve Canada’s over 70 million artefacts and in 2015 made these objects and related information accessible to visitors through approximately 15,000 permanent exhibitions and over 3,600 travelling exhibitions.^{Footnote 26} The number of online exhibitions has grown dramatically in the past decade, reaching over 3,600 in 2015.

Most museums and heritage institutions are predominantly medium and smaller size. In fact, most museums have fewer than ten employees and the number of part-time employees outnumber full-time employees. They rely heavily on volunteer staff, who typically outnumber paid staff. They also depend heavily on funding from various levels of government along with private sector support. Grants, contributions and donations make up the majority of museum revenues and come primarily from government, with the remainder coming from private donors.

Five central issues were mentioned by one-half or more of external key informants: museums lack sufficient operational funding; the opportunities and challenges associated with technology and digitization; a variety of human resources challenges, including the retention of volunteer and paid staff; the growing divide between large and small museums; and, the need to build bridges with Indigenous communities.

A majority of museum representatives stated that obtaining sufficient operating funding is a major challenge – an issue also highlighted in the 2018 museums report of the Standing Committee on Canadian Heritage. Many museums struggle to fulfil their basic mandates, lacking the capacity and resources to manage their collections, to undertake organizational tasks such as policy development, to stage new exhibitions and to do research.

Many key informants noted that new technologies and digitization of collections are playing increasingly important roles in meeting changing consumer tastes. Visitors are expecting technology to play a part in their museum experience. While the use of technology can enable museums to reach new audiences, very few museums have digitized their collections. Museums face the dual challenge of preserving their current collections while taking advantage of digital technologies.

Human resources issues were raised by approximately one-half of external key informants and were a central theme of the Standing Committee report on the state of museums. With constrained operating budgets, museums are reducing the number of curatorial and conservation positions to permit the hiring of specialists in marketing, fundraising, communications/social media and new technologies. A major challenge, particularly for small to mid-size museums, is retaining professional staff, caused by such issues as lack of job security, low salaries and lack of benefits.

Attracting young professionals to work in the museum sector is a particular challenge in rural and remote communities. The Young Canada Works – Heritage program plays an important role in enabling many museums to bring in young people with unique expertise. While museums rely heavily on volunteers, they represent an aging workforce.

The “growing divide” between the largest and smallest museums continues to be a significant trend identified by the previous evaluation. Larger museums typically have the capacity to stage “blockbuster” exhibitions that attract large audiences and provide a revenue stream to support ongoing operations. Small museums play an important role in hundreds of communities by representing local history and heritage in a way that knits together the social fabric. Smaller museums face many of the same issues as larger institutions but many lack the capacity to apply for funding to MAP and other government programs. Many smaller museums are also only open seasonally, and primarily staffed and managed by volunteers.

Finally, the internal interviews conducted within PCH identified many of the same trends and issues voiced by MAP clients and stakeholders. In addition, they highlighted the need to accommodate the government priority of reconciliation with Indigenous peoples and the opportunity for museums to build bridges between Indigenous and non-Indigenous communities.

4.1.2. Extent to which MAP meets the needs of museums

From 2013-14 to 2017-18, MAP received an average of 181 applications per year, of which 144 on average were approved, for an approval rate of 80% (Table 7). In the period covered by the previous evaluation (2010-11 to 2012-13), the numbers of applications received and approved were slightly lower: on average, 162 applications were received and 129 approved, and the approval rate was the same (80%).

The number of applications received each year remained relatively stable over the five years, which suggests the program is of benefit to the museum community. If the program was viewed as not being beneficial, then presumably the volume of applications would have declined.

Table 7: number of funded and non-funded applications, 2013-14 to 2017-18

Fiscal Year	Number of Applications	Number of Applications Approved	Number of Applications Rejected	% Approved	% Rejected
2013-14	183	127	56	69	31
2014-15	174	143	31	82	18
2015-16	182	157	25	86	14
2016-17	181	140	41	77	23
2017-18	185	154	31	83	17
Total	905	721	184	80	20
Average	181	144	37	-	-

Source: GCIMS Data

Over the five-year period, 20% of the applications were not approved, for reasons such as the applications were weak or ineligible or there were insufficient MAP funds.

Further analysis revealed that the approval rate was highest for the Access to Heritage component of the programming (95%) which is for larger museums. The program indicated that larger museums tend to have greater capacity and thus submit strong applications.

During the first three years (2013-14 to 2015-16), the Minister rejected 6% of applications, but rejected none over the other two years (2016-17 and 2017-18). In 2016-17, the program was granted delegated authority, which allowed the Directors General of the Regional Offices to approve projects valued up to $75K.

Smaller museums are less likely to be funded, due to eligibility requirements (for example, must be open to the public year-round and employ at least one full-time paid staff member) or an inability to apply (for example, too complex, costly or time-consuming to apply).

MAP funding was allocated to an average of 114 institutions annually during the five-year evaluation period (2013-14 to 2017-18). In any given year, MAP funding provides support to approximately 4% of the more than 2,700 museums and heritage institutions in Canada. However, given the program’s eligibility requirements, it is estimated that approximately 800 of the 2,700 museums and heritage institutions are eligible for funding.

Over the five years, 59% of new approved projects were from the Ontario and Quebec regions. The Collections Management and Exhibitions Circulation Fund accounted for 71% of all new approved projects.^{Footnote 27} Projects funded by these two components are smaller in size but the barriers to entry are as substantial as the Access to Heritage component.

Over the past few years, PCH has made changes to MAP to ensure the program remains relevant in the cultural heritage domain. Notably, MAP’s Terms and Conditions were updated to better reflect the needs and issues which the museum sector are facing in Canada. MAP formed a partnership with the CCI to support the RE-ORG: Canada initiative that addresses the significant storage issues faced by museums. MAP also granted funding to the Canadian Museums Association to support reconciliation with Indigenous peoples.

While most key informants indicated that MAP is aligned with some of the museum sector’s needs, approximately one-half of key informants identified core issues, especially:

Small and remote museums either do not apply or do not qualify to apply – Respondents commented that MAP’s eligibility criteria are too restrictive, there is a lack of resources in small museums that can be dedicated to application processes and there is a perception that MAP is oriented to larger museums. It is difficult for small museums to find matching funds, and there is no capacity-building component for small museums.
MAP’s narrow focus does not cover many critical museum needs – Many examples were provided of needs that are not covered by MAP, including digitization, operational funding and ongoing sustainability of museums, engaging with new communities, volunteer management, succession planning and development, governance, organization and management restructuring, collection care, and collaborative regionally-based projects. Note: the program has confirmed that many of these areas are funded, however it would appear that they are poorly known by funded clients.

The case study of the RE-ORG: Canada initiative found that it responded to the museum’s sectors needs related to the storage of collections and provided training of staff. However, MAP’s level of financial support was very limited ($10,000 annually). Transportation and accommodation costs incurred by participants were not covered, which may have discouraged some institutions from participating. RE-ORG: Canada initiative was a one-time project that delivered one round of training in each region and is no longer in operation.

Key informants indicated that MAP has an important impact on museums, even with limited financial supports, which have remained stagnant for many years. During the five-year evaluation period, MAP project funding accounted for 12% of the total budgets of institutions.

Evaluation question 1.2. Does MAP support such government priorities as reconciliation, GBA Plus and Official Language Minority Communities (OLMCs)?

MAP supports government priorities of reconciliation, GBA Plus and OLMCs. Since 1990, MAP has administered an Indigenous Heritage component and following the 2015 Truth and Reconciliation report, it increased its flexibility and funding limits to this component. The program also has provided funding to the Canadian Museums Association to conduct a national review of museums policies and best practices and provide recommendations for the inclusion and representation of Indigenous communities within museums. The program funds projects undertaken by museums located in OLMCs and the funded exhibitions feature bilingual content. While there is evidence that the Department is committed to GBA Plus, it is not possible to assess the extent to which MAP and the funded projects are serving various priority groups. Performance data on these priorities is not available in GCIMS, and the program only reports on reconciliation and OLMCs.

MAP has a long history of supporting the preservation of and access to Indigenous heritage. The 1990 Museum Policy committed to the development, preservation, protection and management of Indigenous collections and to the professional development of Indigenous museum personnel.^{Footnote 28} Since that time, MAP has had a component, Indigenous Heritage, dedicated to this purpose.

MAP has also responded to the government’s commitments related to reconciliation arising from the 2015 Truth and Reconciliation report. MAP has increased its flexibility and funding limits for applicants to the Indigenous Heritage component. A total of 82 projects were approved as part of this component during the five-year evaluation period. Furthermore, exhibitions funded by the Access to Heritage component and the Exhibitions Circulation Fund sub-component, including Borrowing of Artefacts projects, can be related to Indigenous heritage. In total, 151 projects related to Indigenous heritage over the five years were completed.

In addition, the program transferred funds to the CMA in compliance with the Truth and Reconciliation Commission’s Call to Action #67 for a national review of museum policies and best practices by the CMA in collaboration with Indigenous peoples, in order to determine the level of compliance with the United Nations Declaration on the Rights of Indigenous peoples and to make recommendations. In 2018, the CMA convened a Reconciliation Council to examine this issue.

There is also evidence that the programming supports OLMCs. The PCH Heritage Group’s commitments under Section 41 of the Official Languages Act are to ensure that strong links are developed and maintained with OLMCs; including that these communities continue to be regular clients and have access to the Heritage Group’s programs and services (including MAP) in the language of their choice.

The document review and interviews with PCH key informants revealed that MAP meets these commitments in several ways. For example, through: ongoing consultations between the regional offices and institutions located within OLMCs; contributions to MAP projects undertaken by museums located in OLMCs; and, travelling exhibits that reach museums located in OLMCs. In addition, the program funds projects that foster the full recognition and use of English and French including through bilingual exhibitions and programming materials.

There is evidence that PCH is also committed to ensuring that GBA Plus is integrated in its decision-making processes. The department has created a GBA Plus Responsibility Centre, located within the Strategic Policy and International Affairs Sector, and a community of practice, represented by all parts of the organization. However, through interviews, the program indicated that it struggles to target visible minority groups. Furthermore, no specific priorities are identified in the MAP guidelines.

Finally, the administrative data review was not able to identify the number of priority groups reached by MAP-funded projects, aside from Indigenous applicants and OLMC projects. The program does not set annual targets related to funding of projects within OLMCs, that target visible minorities or with GBA Plus communities. It does not report on program reach or impact with visible minority or other GBA Plus groups.

4.2. Effectiveness

This section presents the major evaluation findings and conclusions related to program effectiveness, organized by the immediate, intermediate and ultimate outcomes presented in the MAP logic model.

Evaluation question 2.1. Achievement of immediate outcome – Travelling exhibitions are produced

A total of 269 projects to produce and-or circulate exhibitions were approved for MAP funding during the five-year evaluation period. The number of projects funded in each region was roughly proportional to the distribution of the Canadian population.

MAP supported a large number of travelling exhibitions during the five-year period covered by the evaluation (2013-14 to 2017-18). A total of 269 projects to produce and-or circulate travelling exhibitions were approved over this period through both the Access to Heritage and the Indigenous Heritage component.

As shown in Figure 2, the number of projects funded in each region was roughly proportional to Canada’s population distribution. For example, 65% of the travelling exhibitions were funded in Ontario and Quebec and these two provinces account for approximately 60% of Canada’s population (2016).

**Figure 2: number of travelling exhibitions compared to population distribution, 2013-14 to 2017-18**

PCH administrative region	% of travelling exhibits per region	% of Canadian population in 2016
Atlantic	5	7
National Capital Region (N.C.R.)	2	4
Ontario	33	39
Prairies and North	8	1
Quebec	32	24
West	21	26

As per previous MAP evaluations, the funding was critical to these projects, which was re-affirmed by the present evaluation through the key informant interviews. In the absence of MAP funding, virtually all funded exhibitions would either not have been produced at all or would have been carried out but reduced in size or scope or delayed.

Evaluation question 2.2. Achievement of immediate outcome – Projects to improve the preservation and presentation of Indigenous heritage are implemented

MAP’s Indigenous Heritage component contributed to improving the preservation and presentation of Indigenous heritage. Over the five-year period, the program approved 82 projects for the repatriation of cultural objects, to build connections with Indigenous communities, and to raise awareness and understanding of Canada’s Indigenous heritage. Several issues were raised with this programming component, including that it does not seem to be well known within Indigenous communities, that there are so many unmet needs within this sector, and that many of these needs would not eligible for MAP funding. A greater communication with Indigenous communities to better understand their needs was also identified.

In total, 151 projects that involved Indigenous heritage were completed over the five year period, across all MAP components. Most of the projects (58%) were funded through the Indigenous Heritage component, 27% through the Exhibition Circulation Fund and, 11% through Access to Heritage. There were fewer projects supported through the Collections Management (3%), and Borrowing of Artefacts (1%) components.

MAP’s Indigenous Heritage component supports the preservation of, and access to Indigenous heritage. As shown in Table 8, this component received a total of 119 applications for funding during the five-year evaluation period, of which 82 were approved (69%). This approval rate was lower than for MAP as a whole.

Table 8: number of projects funded by the Indigenous Heritage component

Fiscal Year	Applications	Approvals	% Approved for IH	% Approved for MAP
2013-14	31	20	64	69
2014-15	19	14	74	82
2015-16	22	17	77	86
2016-17	25	16	64	77
2017-18	22	15	68	83
Total	119	82	69	80

Source: GCIMS Data

It should be noted that the Access to Heritage component also supports projects that are related to Indigenous heritage, as well as the Borrowing of Artefacts component, a subcomponent of the Exhibition Circulation Fund. Non-Indigenous institutions can also preserve their Indigenous cultural objects and collections in the Collections Management component.

Most key informants (mainly within PCH) indicated that the Indigenous Heritage component funds worthwhile projects with positive impacts. These projects have helped with the repatriation of cultural objects, helped build connections with Indigenous communities, and raised awareness and the level of understanding of Canada’s Indigenous heritage. Several projects were completed that transferred cultural skills from Elders to youth. Internships have also been provided to Indigenous youth. The program has been able to work with the same partners over many years, which is a significant benefit.

However, the evaluation identifies areas for improving program results related to Indigenous heritage, particularly in light of the importance of government’s commitments to reconciliation. Many external key informants (non-Indigenous) were concerned about increasing communication with Indigenous communities. This was supported by the literature review, which also found that museums were not sure how to support reconciliation and repatriation. Several key informants indicated that the Indigenous Heritage component is not well known, with suggestions to increase the level of communications with Indigenous communities to better understand their needs.

Some key informants suggested that the program better promote how it can support the translation of documents into Cree and other Indigenous languages. Though translation into Indigenous languages is already permitted under the current guidelines, eligibility guidelines are not well known by some potential applicants. Others stated that this component should be broadened and the level of funding increased, given the many unmet needs of Indigenous organizations, such as the need for new infrastructure. Another suggestion was to permit small communities in addition to Indigenous organizations to apply for funding, so that they could develop projects that enable them to enhance their understanding of the world around them.

Evaluation question 2.3. Immediate outcome – Opportunities are created for heritage institutions and workers to enhance their professional knowledge, skills and practices

MAP funding has led to close to 1,000 quality training opportunities for professionals in the Canadian museum to improve knowledge and improve professional practices. Funding for professional development, skills and practices is provided primarily through the Collections Management component and to a lesser extent by the Indigenous Heritage component and the Canada-France agreement. The RE-ORG: Canada initiative with the Canadian Conservation Institute has helped several museums improve their museum storage practices.

MAP activities have contributed to developing knowledge, skills and professional practices of heritage institutions and workers. Over the five-year evaluation period, a total of 969 opportunities were offered to enhance professional knowledge, skills and practices (Table 9). The projects supported a mix of in-person workshops and webinars, as well as written tools. The level of professional development activity peaked in 2015-16 and declined in the following two years.

There were of 763 opportunities offered as part of Collection Management projects and 79 opportunities offered as part of Indigenous Heritage projects. In addition, 127 opportunities were offered as part of projects funded via the Canada-France Agreement, which contributed to the sharing of knowledge and skills between institutions in Canada and France. Projects are funded primarily by the Collections Management component, which funds projects designed to create learning opportunities and to improve the management of collections. Collections Management funds two types of projects: those aimed directly at improving the care and management of collections and those focused on improving workers’ and institutions’ knowledge, skills and professional practices.

Table 9: number of projects that enhanced professional knowledge, skills and practices, by component, 2013-14 to 2017-18

Fiscal Year	CM	IH	Canada-France Agreement	Total
2013-14	145	43	2	190
2014-15	141	1	69	211
2015-16	234	25	20	279
2016-17	140	6	22	168
2017-18	103	4	14	121
Total	763	79	127	969

Source: MAP Program Performance Analysis 2013-14 to 2017-18

MAP also partnered with the Canadian Conservation Institute (CCI) to develop and implement RE-ORG: Canada initiative, a storage reorganization training initiative that helped museums address their storage issues and inspire the next generation of leaders in storage organization. This initiative addressed the key challenge of museum storage faced by museums in Canada and all over the world. When museum storage is severely disorganized, collections are no longer accessible, which limits their benefit to the public. During the evaluation period, a total of 12 RE-ORG: Canada projects were completed by CCI.

Several observations were made by key informants:

The level of funding does not cover all training needs and limits the number of employees from each institution who can participate.
While some stated that the training was accessible, others noted that geographic distance made it less accessible for some. The increasing availability of remote learning is helping to address this accessibility challenge.
While there are perceptions that training is restricted to collections management, museums have a need for training in other areas, including marketing, sponsorships, and digitization. The program noted that at one time MAP had an Organizational Development component that supported training in areas beyond collections management.

Evaluation question 2.4. Achievement of intermediate outcome – Heritage exhibitions and other public programming products/activities are presented to the public

MAP funded a total of 384 heritage exhibitions over the five years, most of which were travelling rather than non-travelling. These exhibitions were accompanied by related public programming products and activities. A total of 719 venues hosted exhibitions, of which 60% were located in Ontario and Quebec. Over 6.6 million visitors experienced these exhibitions over the five years, an average of over 1.33 million visitors per year. More than 6,500 public programming activities and over 1,700 public programming products were produced during the five years.

MAP supported the presentation of a large number of heritage exhibitions and other related public programming products and activities over the five years. Mostly, these were travelling exhibitions that were either produced or borrowed with funding from the Access to Heritage, Exhibition Circulation Fund and Indigenous Heritage components.

As shown in Figure 3, a total of 384 exhibitions were presented to the public through MAP funding over the five years, an average of 77 per year. The volume of exhibition activity increased compared to the period examined by the previous evaluation (2010-11 to 2012-13), when 51 exhibitions were presented annually on average.

**Figure 3: number of exhibitions presented, 2013-14 to 2017-18**

Fiscal year	Number of travelling exhibitions presented	Number of non-travelling exhibitions presented
2013-2014	53	5
2014-2015	56	7
2015-2016	77	6
2016-2017	85	3
2017-2018	91	1

In addition, more than 6,500 public programming activities and over 1,700 public programming products were produced during the five years to accompany exhibits or provide public access to collections. Public programming activities include lectures, special guided tours, educational programming and online activities. Public programming products include multimedia presentations, interactive displays, object reproductions and self-guided tour aids.

Exhibitions were hosted by a large number of venues across Canada. A total of 719 venues were the sites for exhibitions over the five years, although this figure includes some funded projects whose exhibitions were held in the same location (Figure 4). Ontario and Quebec accounted for the majority of venues (60%), followed by British Columbia (14%) and Alberta (7%). A few exhibitions travelled outside of Canada, mainly to the US. The concentration of venues in central Canada is explained by the presence of many larger museums, which have the resources necessary to apply to MAP and to plan and circulate travelling exhibitions.

**Figure 4: number of host venues that took place in Canada, by province/territory**

	2013-2014	2014-2015	2015-2016	2016-2017	2017-2018
Alberta	11	6	10	12	14
British-Columbia	11	16	18	37	17
Manitoba	10	1	3	9	9
New-Brunswick	1	4	4	4	6
Newfoundland and Labrador	0	3	3	0	1
Nova Scotia	7	3	6	0	2
Northwest Territories	0	0	0	0	1
Nunavut	1	2	2	0	0
Ontario	19	23	41	38	62
Prince-Edward-Island	4	1	0	0	0
Quebec	39	41	59	57	51
Saskatchewan	6	6	14	4	3
Yukon	1	5	1	1	3
Other	2	2	0	1	1

MAP-funded exhibitions and related public programming products/activities have been experienced by millions of Canadians. During the evaluation period, the review of administrative data revealed that there were over 6.6 million (6,632,236) visitors to exhibitions funded by MAP across Canada (Figure 5), an average of 1.33 million visitors per year. The number of visitors peaked in 2015-16, which may have been due to pre-Canada 150 activities. The number of visitors increased compared to the period examined by the previous evaluation (2010-11 to 2012-13), when just over 1 million visitors experienced MAP-funded exhibitions and other public programming products/activities.

Most key informants stated that MAP funding has helped increased attendance of visitors to their institutions. Some also indicated that MAP funding helped them to bring high quality and/or major exhibits to their organization, to bring prestige and regional awareness of their organization, and to have a longer time period to develop their collections.

**Figure 5: number of visitors to MAP-funded exhibitions, 2013-14 to 2017-18**

Fiscal year	Number of visitors to exhibitions
2013-2014	718 868
2014-2015	843 806
2015-2016	2 351 155
2016-2017	1 200 838
2017-2018	1 517 569

Evaluation question 2.5. Achievement of medium-term outcome – Heritage objects and collections are better managed and preserved

MAP funding has had a considerable impact on the improvement of collections management and preservation of objects in general. Projects were funded by the Collections Management and Indigenous Heritage components. MAP funding for training and professional development capacity had an indirect impact on this same objective. Approximatively 1.5 million objects were managed and/or preserved under the Collection Management and Indigenous components over the five years, for a total of 86 projects to improve the management of heritage objects and collections, as well as 141 projects to improve the preservation of heritage objects and collections, were approved. As part of their final project reports, virtually all clients reported at least some improvement in the management and preservation of heritage objects and collections, with approximately one-third reporting substantial improvements.

Some key informants noted that MAP’s funding of professional development capacity, such as through the RE-ORG: Canada initiative, indirectly contributed to the achievement of this outcome.

As part of the program’s final reports, clients reported on their institution’s improvement pertaining to five aspects of heritage management or preservation, depending on the type of project. Over the five years, just over one-half (55%) of clients reported that correction of records was the main aspect of heritage management improved, followed by databases (49%), digitization and making objects accessible via the web (both at 46%), and cataloguing (39%). With respect to preservation projects, the main improvement was in storage spaces (59% of clients), followed by shelving (55%), conservation materials (53%, climate control (48%) and security (25%).

Clients were also asked to indicate the institution’s status before and after the completion of each project, using a five-point scale (none, poor, average, good and very good). For the management of heritage objects and collection projects, 95% of clients reported at least some improvement, with 34% reporting an large improvement of 3 or 4 levels.

Some key informants stated that it would be helpful if MAP could fund other major needs of museums, including research, digitization of collections and in supporting specialized conservation and intervention requirements.

Evaluation question 2.6. Achievement of intermediate outcome – Institutions and workers in the heritage sector have improved their knowledge, skills and professional practices

Over 9,000 heritage workers from just over 3,000 institutions have participated in the hundreds of MAP-funded professional development activities over the five years. Surveys administered after these activities have found that three-quarters (76%) of these workers reported that their level of professional knowledge, skills and practices had improved.

As discussed earlier, MAP funding enabled hundreds of professional development opportunities to be offered over the five years, consisting of workshops, webinars and written tools. Program administrative data indicates that over nine thousand (9,039) professionals from over three thousand (3,055) institutions have taken advantage of these opportunities.

There is evidence that the training led to improved knowledge, skills and practices. In surveys administered to heritage workers, they are asked to report their level of knowledge, skills and practices before and after attending each professional development activity, using a five-point scale. Over 5,000 workers completed the survey during the five years, and three-quarters (76%) of workers reported improvements by at least one level due to the activities. Approximately 1 in 5 (22%) participants reported an improvement of 2 levels, while 1 in 10 (11%) reported an improvement of 3 or 4 levels.

Key informants confirmed that the professional development activities have a positive impact on their knowledge, skills and practices. The RE-ORG: Canada initiative and, to a lesser extent, the Canada-France Agreement were mentioned in positive terms. The funded professional development activities have the added benefit of helping participants to form networks and partnerships.

Some common observations by key informants:

The level of MAP funding does not cover all training needs and also limits the number of employees from the same institution who can participate in a particular activity.
While some key informants indicated that the professional development activities were accessible to them, others stated that high costs and geographic distance made these activities less accessible to others.
The emergence of remote learning opportunities has partially addressed the accessibility challenge, however internet connectivity remains a challenge.
Training content is perceived by clients as being restricted to collections management, though the program does fund other activities. Comments from clients mentioned that it would benefit from being broadened to include other museum management activities, such as marketing, sponsorships, digitization, etc.

Evaluation question 2.7. Achievement of long-term outcome – Canadian heritage is accessible to Canadians over time

Over the evaluation period MAP continued to contribute to the achievement of the program’s ultimate outcome of making Canada’s heritage accessible to the Canadian public.

MAP contributes to the accessibility of Canadian heritage to Canadians. As outlined in the preceding pages, the review of program administrative shows that MAP-funded exhibitions and other public programming activities/products have reached millions of Canadians. An estimated 1.33 million visitors on average experienced MAP-funded exhibitions during each year of the evaluation period.

A majority of stakeholders interviewed indicated that MAP funding has had a specific positive impact on the capacity of Canada’s heritage institutions to provide long-term access to Canada’s heritage.

The evaluation confirmed a finding of previous MAP evaluations that MAP funding was important for the success of funded projects, though it is a small percent of overall museum budgets. In the absence of MAP funding, most projects would not have gone ahead, or would have been scaled back or delayed.

Overall, program administrative data shows that 97% of institutions that completed projects to better manage or preserve heritage objects reported improvements, thus increasing their ability to provide long-term access to Canada’s heritage.

In addition, 81% of heritage workers who participated in professional development opportunities reported an improvement in their knowledge, skills and practices, thus increasing their ability to ensure long-term access to Canada’s heritage.

Evaluation question 2.8. Were there any unexpected results or challenges as a result of MAP activities?

One unexpected result from the RE-ORG: Canada initiative, is that MAP funded activities have helped to create networks among members of the museum community.

An unintended result of the RE-ORG: Canada initiative was the development of networks and relationships among members of the Canadian museum community. Relatively few comments were made by key informants on whether MAP activities had resulted in any unexpected impacts or challenges.

4.3. Efficiency

Evaluation question 3.1. Has MAP been implemented efficiently?

MAP was delivered in an efficient manner. It has an administrative ratio of 15%, which is an improvement from the period covered by the previous evaluation. The program has largely met the service standards set by the department. Delegation of authority to the Regional Directors General may have contributed to improvements in efficiency.

4.3.1. Administrative costs

The program’s administrative cost ratio over the five-year evaluation period was 15%, as shown in Table 10, which showed a slight decline over the five years. This represents an improvement compared to the 22% administrative ratio reported in the previous evaluation (2010-11 to 2013-14), perhaps due to the notional allocation of funds to the regions, which permits greater flexibility to the regions in awarding funds.

Table 10: MAP administrative ratio for the regions, based on actual operational and administrative costs (Vote 1) and G&Cs (Vote 5), Big MAP, 2013-14 to 2017-18 ($)

	2013-14	2014-15	2015-16	2016-17	2017-18	Total
Salaries and O&M	1,199,517	1,184,660	1,106,947	1,130,298	1,103,634	5,725,056
Grants and contributions	6,732,109	6,756,434	6,616,686	6,942,331	6,743,158	33,790,718
Total	7,931,626	7,941,094	7,723,632	8,072,629	7,846,792	39,515,773
Administrative ratio	15%	15%	14%	14%	14%	15%

Source: MAP financial data.

Note: Vote 1 costs are tracked as a whole from headquarters by the program for the following programs: Museum Assistance Program, Young Canada Works-Heritage, and Movable Cultural Property Grants. Therefore there was no regional and program specific data available.

A comparison of planned with actual expenditures over the five years shows that actual human resources expenditures (salaries and EBP) were much greater than planned, due to higher average salary costs. Conversely, actual G&C expenditures (Vote 5) were very close to planned expenditures (Table 11).

Table 11: MAP planned and actual expenditures (Vote 5), 2013-14 to 2017-18 ($)

Year	Planned			Actual
Year	Grants	Contributions	TOTAL	Grants	Contributions	TOTAL
2013-14	$3,500,000	$3,193,284	$6,693,284	$1,939,915	$4,792,194	$6,732,109
2014-15	$3,500,000	$3,193,284	$6,693,284	$1,952,817	$4,803,617	$6,756,434
2015-16	$3,500,000	$3,193,284	$6,693,284	$2,283,371	$4,333,315	$6,616,686
2016-17	$3,500,000	$3,193,284	$6,693,284	$1,892,778	$5,049,553	$6,942,331
2017-18	$3,500,000	$3,193,284	$6,693,284	$2,343,543	$4,399,615	$6,743,158

Source: PCH Finance

An examination of the program’s performance in achieving the service standards set by the department revealed that most of the standards have been met (Table 12). The program was able to acknowledge receipt of 92.7% of applications within 15 days during the evaluation period.

The program greatly improved its performance in achieving the standard for making funding decisions over the period of evaluation, decreasing timelines. For the first two years, the standard was 29 weeks and the program achieved this standard 62% of the time. For the last three years, the standard was reduced by 3 weeks to 26 weeks, and the program met this standard 96% of the time. No data was available on the program’s performance on the number of days required to issue payments to recipients.

Interviews conducted within PCH indicated that delegation of authority contributed to the program’s ability to meet its standards.

Table 12: achievement of service standards for acknowledge of receipt of funding applications and making funding decisions

Components and sub-components	Type of activity	2013-14	2014-15	2015-16	2016-17	2017-18
Access to Heritage	Acknowledgment of receipt % Met	78%	97%	96%	97%	100%
Access to Heritage	Decision % Met	100%	23%	44%	85%	100%
Canada-France Agreement	Acknowledgment of receipt % Met	100%	100%	100%	75%	100%
Canada-France Agreement	Decision % Met	100%	100%	100%	75%	100%
Exhibition Circulation Fund	Acknowledgment of receipt % Met	85%	85%	85%	93%	95%
Exhibition Circulation Fund	Decision % Met	83%	98%	89%	95%	98%
Collections Management	Acknowledgment of receipt % Met	88%	95%	95%	93%	99%
Collections Management	Decision % Met	69%	27%	52%	91%	97%
Indigenous Heritage	Acknowledgment of receipt % Met	90%	100%	95%	76%	100%
Indigenous Heritage	Decision % Met	86%	11%	48%	93%	100%

Source: Departmental Data

4.3.2. Design and delivery

When external stakeholders were asked whether any improvements are required to MAP’s design and delivery, most stated that application forms should be simplified. Some questioned why applications must be submitted to a program officer instead of via an online portal. As per the previous MAP evaluation, the application process was meant to be moved online via the GCMP project, which was subsequently cancelled. The application process is planned to move onto the PCH Online portal, currently under development.

Some key informants suggested that the program should provide multi-year funding, however, the program noted to the evaluation team that it can consider funding for projects that take place over more than one year. This would suggest that this possibility is not generally known by applicants. Some would like to see more flexibility with cut-off dates, e.g., to have two application intake rounds annually instead of one. Finally, some mentioned an issue discussed earlier, i.e., the project eligibility criteria are not aligned with many of the needs of museums.

The case study of the RE-ORG: Canada initiative identified other issues with the timing and coordination between the application process for the initiative, and the MAP deadlines and processes. Some participants stated that the processing of applications by CCI and MAP, and the timing of budget allocation by MAP could have been more closely aligned, to avoid problems with coordination, communication and overlapping of grants. Potential RE-ORG: Canada initiative participants applied to CCI, which then sent applicants to MAP for funding. The timeline for the delivery of funding through MAP, meant that successful applicants to RE-ORG: Canada initiative had already started their projects by the time the MAP funding was confirmed.

Evaluation question 3.2. Was the notional allocation of funding method more efficient?

The notional allocation of funding resulted in efficiencies, and improvements in financial management. Most internal key informants indicated that using a notional allocation method based on population distribution by region is a more efficient approach than that used previously.

In 2015, PCH regional offices requested increased regional equity in the distribution of MAP funds and the National Office undertook an extensive review for all MAP components, covering 2012-13 to 2014-15. As a result, a new approach for budget allocation was launched beginning in the 2016-17 cycle. From that point, MAP’s budget was distributed to the regions based on a notional allocation, related to the percentage of population per region, after removing national and multi-year commitments, the Exhibition Circulation Fund and Canada France Agreement envelopes, and priority projects.

Table 13 shows the ratio of requested and approved budget amounts in the regional offices following the implementation of the notional allocation method. In all regions except for Quebec, the ratio of requested amounts approved increased with the new approach. For instance, for the Atlantic region, only 51% of requested amounts were approved before its implementation compared to 80% after. The probability that the total amount requested be accepted has increased since the notional allocation coming into effect in all regions, to various degrees, except for Québec. Quebec region was allocated $2.1 million less in the two years following adoption of the new method.

The program noted that factors other than the notional allocation also influenced the increase in the percentage of requested amounts approved. For example, in the Prairies and North Region, there was decrease in applications because the region began actively discouraging multiple applications from the same client (a practice that other regional offices have also adopted) and because a major art gallery in the region closed, removing a major MAP applicant.

Table 13: percentage of requested amounts approved before/after implementation of notional allocations, by region

Region	Ratio of requested- approved amounts before^{Table 13 note *} notional allocations (2014-15 to 2015-16)	Ratio of requested-approved amounts after^{Table 13 note *} notional allocations (2016-17 to 2017-18)	Difference in amounts received by the regions after notional allocations (over the 4 years)
Atlantic	51%	80%	$301,429
Ontario	57%	67%	$596,795
Prairies & North	47%	64%	$1,078,544
Quebec	64%	48%	($2,100,079)
West	78%	80%	$1,735

Source: Analysis of MAP data obtained from GCIMS.

Table 13 notes

Table 13 note *

In order to assess the impact of the notional allocation approach, data were compared for the two years preceding implementation of the new approach (2014-15 and 2015-16) and the two years following use of the new approach (2016-17 and 2017-18). The first year (2013-14) was excluded so that the same number of years was compared before and after implementation of the new approach. Percentages do not represent regions’ demographic weight, but reflect the increased availability of funds by application following the notional allocations, which is based on the demographic weight. As well, requested amounts by region vary yearly, which influences the ratios illustrated.

Return to table 13 first note * referrer

In 2013-14, the regions returned 1.0% of their budgets to the PCH corporate reserve, and this figure dropped to 0.13% in 2017-18 with regional notional allocations. However, program key informants stated that this improvement could be due less to notional allocation but more to a change in policy to prohibit the re-profiling of funds between fiscal years, and the flexibility introduced by the PCH Centre of Expertise in the contribution agreements to allow clients to retain funds already paid up to September 30th of the following fiscal year of the project.

Evaluation question 3.3. Does the performance measurement framework for MAP allow for the efficient reporting on programs results?

To a certain extent, the program has a performance measurement framework that allows efficient reporting on program results. The program has developed a data analysis methodology to report on program results. However, the full methodology was not available to the evaluation at the time of data analysis. The methodology may need simplification to allow for more alignment with existing data in GCIMS. Also, the program logic model and program information profile (PIP) do not appear to be completely aligned with how the program is currently reporting. The program does not report on priority groups.

As per requirements of TBS, MAP has a PIP, which defines expected program results and indicators. The program produced a performance analysis report covering the evaluation period. Unfortunately, the detailed data analysis methodology to support the program’s performance report was not available to the evaluation team until final phases of the project, in part due to restricted activities during the early stages of the COVID-19 pandemic. When the team received the methodology in December 2020, it was not able to replicate the results produced by the program.

Given the professional standards that ESD follows in its methodologies, as well as these challenges, the evaluation team was not able to fully leverage the program’s performance analysis.

In addition to the complexity of the performance measurement approach, the evaluation identified some gaps in the alignment of outcomes and indicators between the program’s logic model, the PIP, and the data analysis methodology. The evaluation noted that the methodology reports on outcomes across a combination of components, with repeating indicators, while these relationships were not clearly represented within the logic model nor within the current PIP.

The program does collect some data on priority groups, however, these are not reported. There is no data regarding priority groups in GCIMS, aside from projects submitted by Indigenous applicants or with Indigenous content collaborations, and data is currently tracked manually by the program via recipients’ final reports.

PCH internal key informants had mixed views on the usefulness of the performance measurement framework. A few stated it was strong, while a majority of internal informants were not familiar with it. Although MAP has made efforts to simplify the reporting required of recipients, some stated that further improvements could be made to reduce the burden placed on recipients.

4.4. Other – Innovation

Evaluation question 4.1. What lessons or best practices have been learned from MAP’s innovation projects?

The RE-ORG: Canada initiative had a limited scope but had a positive impact on museums and their collections. The innovative nature of this project fostered the acquisition of practical knowledge and networking among members of the museum community. MAP’s involvement was deemed to be essential to museum participation. A few issues were identified by key informants, such as the considerable amount of time required to complete the training. It was suggested that the initiative be broadened to address other needs of museums.

The case study of the RE-ORG: Canada initiative found that it responded to a growing need from museums to address their storage issues. All key informants indicated that it had a positive impact on their institutions.

The RE-ORG: Canada initiative was based on a methodology developed by ICCROM and UNESCO for reorganizing severely disorganized storage in small to medium-size museums with limited resources or access to outside expertise. CCI used this methodology in its regional workshops focused on storage reorganization.

A total of 12 museums from Ontario, Atlantic and Quebec regions participated in the program between 2015-16 and 2017-18. Approximately three representatives on average participated from each museum. Each year a different region of Canada was selected to participate in the program.

MAP provided financial assistance in the form of grants for acquiring equipment or hiring human resources. An online survey conducted by the CCI showed that MAP funding was very significant for 19 out of 23 respondents.

The joint promotion and processing of candidate applications showed that the MAP has developed a synergy with the CCI in delivering RE-ORG: Canada projects. However, MAP’s coordination with the CCI, in terms of the timing of funding and communications with clients, could be further developed in the event that the project is repeated.

A few issues were raised by key informants. Some noted the considerable amount of time required to participate in the workshops, which had an impact on other museum priorities. The suggestion was made to broaden the initiative to address other training needs related to such topics as digitization and management of computer software, item conservation methods and volunteer training, as examples.

MAP’s very limited level of funding ($10,000) and the restrictions linked to transportation and accommodation costs had a deterrent effect on some museums who would have liked to have participated in the RE-ORG: Canada initiative but could not.

5. Conclusions

5.1. Relevance

MAP continues to respond to some of the pressing needs faced by museums and other heritage institutions in Canada. The program helps these organizations to improve the management and preservation of their collections and to create, circulate and borrow exhibits that enable millions of Canadians to experience Canada’s cultural heritage. The Indigenous Heritage component responds to an urgent need of museums to build bridges with Indigenous communities. MAP has also granted funding to the Canadian Museums Association to support reconciliation. Further, MAP has made changes to its Terms and Conditions over the years in an attempt to respond to the evolving needs of museums.

However, the evaluation found that museums and other heritage institutions have many other needs that are not being met by MAP. Some of these pressing issues are insufficient funding for operations, a lack of human resources capacity, the challenges associated with adopting new technologies and digitizing collections, and the need to build bridges with Indigenous communities.

MAP has been constrained in its ability to respond to these unmet needs, largely because its budget has remained stagnant for many years. Furthermore, MAP’s project-based funding model is not suited to addressing the sector’s significant need for funding stability.

This conclusion echoes a theme of the 2018 report on the state of museums by the Standing Committee of Canadian Heritage. That report stated the MAP no longer reflects the needs and realities of museums in the 21st century and noted that MAP’s project-based premise does not provide any financial support for basic operations. The need for operational funding is an issue faced in particular by many smaller museums, which tend not to apply to MAP. The Standing Committee report recommended that PCH expand MAP to provide more sustainable funding.

The evaluation identifies that the program has made some efforts to support such government priorities as reconciliation and OLMCs. MAP has long supported the need to preserve and provide access to Canada’s Indigenous heritage. It has also responded to the 2015 Truth and Reconciliation report by awarding funding to the Canadian Museums Association to review museum policies and practices and provide recommendations for the inclusion and representation of Indigenous communities within museums. The program funds and tracks projects in the OLMCs.

However, little performance data is available on efforts related to these priorities, aside from information on Indigenous heritage and OLMC projects. The program has noted that it does track this data, however does not report on it. While there is evidence that the department is committed to GBA Plus, it was not possible to assess the extent to which MAP is serving priority groups.

5.2. Effectiveness

During the five-year period (2013-14 to 2017-18) MAP continued to make progress in achieving its intended immediate, intermediate and ultimate outcomes. A total of 269 projects to produce and or circulate exhibitions were approved for funding during the five-year evaluation period. These exhibitions were produced and circulated in all regions of Canada. MAP funding was critical to the production of these exhibitions. These exhibitions have been experienced by millions of Canadians over the five years.

MAP’s Indigenous Heritage component continued to support the preservation of, and access to Indigenous heritage. A total of 82 projects were approved during the evaluation period. The Access to Heritage component, Collections Management component and the Exhibition Circulation Fund subcomponent, including Borrowing of Artefacts projects, also supported projects related to Indigenous Heritage. MAP’s support for Indigenous heritage is critical to ensuring cultural representation in museums and preserving Indigenous cultural heritage and belongings within communities. The evaluation also found that museums are unsure how to create links with Indigenous communities, and how to implement repatriation of Indigenous cultural heritage and belongings, and that further communication with Indigenous communities would be useful to better understand their realities.

MAP also continued to support heritage institutions and their workers to enhance their professional knowledge, skills and practices. A total of 969 opportunities were offered as part of projects funded that were devoted to this area, including 763 opportunities offered as part of Collection Management projects and 79 opportunities offered as part of Indigenous Heritage projects. MAP program data notes that over 9,000 heritage workers from just over 3,000 institutions participated in these professional development opportunities and there was strong data to support that they led to improvements in knowledge, skills and practices.

The RE-ORG: Canada initiative with the Canadian Conservation Institute has helped several museums improve their museum storage practices. In addition, 127 opportunities were offered as part of projects funded via the Canada-France Agreement, which contributed to the sharing of knowledge and skills between institutions in Canada and France.

MAP funding has had a considerable impact on the improvement of collections management and preservation of objects. Approximatively 1.5 million objects were managed and/or preserved under the Collection Management and Indigenous components over the five years, for a total of 86 approved projects to improve the management of heritage objects and collections, and 141 approved projects to improve the preservation of heritage objects and collections. Funded institutions reported that they experienced improvement in the management and preservation of their collections and objects, with approximately one-third reporting substantial improvements.

5.3. Efficiency

MAP has improved its operational efficiency over the years. The program’s ratio of administrative costs to total expenditures was approximately 15% over the five years, which represents an improvement compared to the 22% administrative ratio reported by the previous evaluation (2010-11 to 2013-14). The program largely achieved the service standards set by the department.

A departmental decision to delegate authority to the Regional Directors General may have contributed to the improvement in efficiency. A case study concluded that the decision by the program to allocate funding using a notional allocation method based on population distribution by region is a more efficient approach. This approach led to improved financial management and facilitated the work of review committees.

Application packages are on request, and are distributed by MAP program officers after applicants call for more information. Applications are currently submitted directly to program officers. It was also noted by informants during the evaluation that the application process would be more efficient if it was online. The last evaluation noted the need to move applications online, and was planned as part of the GCMP roll-out. As that project was not completed, the move of applications to an online portal is now planned within the PCH Online project.

The availability of performance data is an important consideration for program efficiency as it supports evidence-based decisions making by program management. The assessment of the program’s performance measurement documents received determined that there could be better alignment between the current PIP, the logic model, and the performance reporting methodology. The program data analysis methodology for reporting results was not available to the evaluation team during the conduct of the project due in part to restrictions caused by the COVID-19 pandemic. The evaluation team was not able to replicate the analysis and found that the methodology report could be simplified, to allow for more aligned reporting on the programs results.

Furthermore, PCH staff had mixed views on the usefulness of the performance measures in place. Some suggested that further improvements could be made to reporting, to reduce the burden placed on recipients.

5.4. Other – innovation

The RE-ORG: Canada initiative conducted in partnership with the CCI had a limited scope but a positive impact on museums. The innovative nature of this project fostered the acquisition of practical knowledge and networking among members of the museum community. It was suggested that the initiative be broadened to address other needs of museums.

6. Recommendations, management response and action plan

MAP remains relevant to the museum community, however there are ongoing and emerging needs that are not currently fully addressed.

The findings of this evaluation note that overall, MAP is filling many important needs of museums in Canada. However, there are emerging needs that are preoccupying the museums sector, including adopting new technologies, digitization of collections, and continuing to build bridges with Indigenous communities, among others. The recommendations of the Truth and Reconciliation Commission are a critical government priority for museums and heritage organizations. In particular, the evaluation raises opportunities to improve engagement with Indigenous communities and Indigenous organizations on questions such as the repatriation of cultural objects. Also, the repercussions of COVID-19 are and will continue to be critical for museums, though this was not fully assessed through this evaluation. An environmental scan of museums during the pandemic was subsequently conducted and can be found in Annex A.

Recommendation 1

Management response - Recommendation 1

The Heritage Policy and Programs Branch (HPPB) Management accepts this recommendation.

The Museums Assistance Program (MAP) works collaboratively with its regional teams to examine and identify feasible ways to address current and emerging museum priorities. For example, as part of the COVID-19 Emergency Support Fund for Cultural, Heritage and Sport Organizations, MAP delivered $34 million to approximately 1,300 heritage organizations to allow for the continuous care of their heritage collections in light of the current context. While this emergency funding provided some temporary relief and broadened the reach of MAP’s client base, heritage organizations continue to face many challenges. Budget 2021 introduced a number of new measures to help heritage organizations on the road to recovery.

In addition, Budget 2021 announced new funding to support the development of a new digital access sub-component under MAP. These new funds will address a key museum priority by supporting the creation of online content by non-national museums. However, HPPB did not receive any additional funding in Budget 2021 for the renewal of the Museum Policy, which currently dates to 1990, and the co-development of a framework for repatriating Indigenous cultural property and ancestral remains. HPPB will continue to address current and emerging contexts within MAP’s existing resources while aligning with government priorities.

Reaching Indigenous communities remains a key priority for MAP. Since the mid-1990s, MAP has funded projects with an Indigenous focus through a dedicated and flexible Indigenous Heritage component.

The Heritage Policy and Programs Branch will work with MAP’s regional teams and key stakeholders to identify and implement an engagement strategy to meet this recommendation. The framework will build upon the work already done by the regions, which has resulted in increased support for Indigenous organizations in recent years. Furthermore, the framework will take into consideration the limited resources available at HQ and in the region as well as the expectations and pressures on existing program budgets resulting from broader engagement.

Table 14: recommendation 1 – action plan

Action Plan Item	Deliverables	Timeline	Responsible
1.1: HPPB to develop and submit the required documentation for approval of a new digital access component under MAP	1.1.1. Digital Access Component is developed and approved	October 31, 2021	Deputy Director General, Heritage Policy and Programs Branch
1.2: MAP to launch the new digital access component to support the creation of online content by non-national museums.	1.2.1. Program launch of digital access component	November 30, 2021	Deputy Director General, Heritage Policy and Programs Branch
1.3. MAP HQ to develop an Indigenous Engagement Strategy in close collaboration with the regional offices, PCH Communications, and key stakeholders.	1.3.1. Indigenous Engagement Strategy framework	March 31, 2022	Deputy Director General, Heritage Policy and Programs Branch
1.4. Regional offices, with the support of MAP HQ and PCH Communications, to implement the Indigenous Engagement Strategy.	1.4.1. Implementation of Indigenous Engagement Strategy	October 31, 2023	Deputy Director General, Heritage Policy and Programs Branch

Full implementation date: October 31, 2023

There are ways to continue to improve the efficiency of MAP delivery.

MAP has successfully responded to a recommendation of the previous evaluation to reduce the administrative costs and to improve achievement of service standards. As per a recommendation of the previous evaluation of MAP, the program was working on modernizing the application process as part of the Department’s Grants and Contributions Modernization Project (GCMP). However, GCMP was discontinued prior to any progress being made. However, as part of a new initiative, My PCH Online, MAP is scheduled to have its application process moved to an online portal in 2021-22.

In addition to continuing to support the transition to a digital application process, the evaluation has identified that intake cycles are an area for further attention. MAP has only one intake cycle annually with the exception of the Exhibition Circulation Fund (which includes the Borrowing Artefacts subcomponent) and the Canada-France Agreement. It affects the ability of applicants to complete time-sensitive projects.

Recommendation 2

The ADM, Official Languages, Heritage and Regions, should work with the respective stakeholders to:

Simplify the application process as part of the move to an online portal and;
Examine the intake cycle.

Management response - Recommendation 2

The Heritage Policy and Programs Branch (HPPB) Management accepts this recommendation.

The Museums Assistance Program (MAP) is scheduled to be one of the first programs to be part of the My PCH Online funding portal in 2021-22. As part of the onboarding process, and to ensure alignment, the program will examine simplification of the application process. MAP will build upon the success of the delivery of the COVID-19 Emergency Funds for Heritage Organizations in 2020-21 where the program worked with the My PCH Online team and the Centre of Expertise to develop a simplified application.

Specific to the examination of the application cycle and the impact on time-sensitive projects, MAP will work collaboratively with its regional teams to examine the intake cycle for its various program components and will prepare a list of recommendations. The analysis will take into consideration the impact on both program and external stakeholders while taking into account staffing implications and program delivery costs.

Table 15: recommendation 2 – action plan

Action Plan Item	Deliverables	Timeline	Responsible
2.1. MAP to work with My PCH Online, the Centre of Expertise and other PCH partners to onboard MAP’s main funding components to a new online portal. The onboarding schedule is tentative and will be adjusted based on operational need and context and, therefore, changes in timing are possible and beyond the program’s control.	2.1.1. MAP’s main funding components integrated into MY PCH Online	March 31, 2023	Deputy Director General, Heritage Policy and Programs Branch; Director General, Modernization
2.2. MAP HQ and MAP regional offices to examine MAP’s intake cycle and provide recommendations.	2.2.1. Recommendations to the Deputy Director General	March 31, 2023	Deputy Director General, Heritage Policy and Programs Branch
2.3. MAP HQ and MAP regional offices to implement the approved intake cycle recommendation	2.3.1. Implementation of Recommendation	March 31, 2024	Deputy Director General, Heritage Policy and Programs Branch

Action Plan Item

Deliverables

Timeline

Responsible

2.1. MAP to work with My PCH Online, the Centre of Expertise and other PCH partners to onboard MAP’s main funding components to a new online portal.

The onboarding schedule is tentative and will be adjusted based on operational need and context and, therefore, changes in timing are possible and beyond the program’s control.

2.1.1. MAP’s main funding components integrated into MY PCH Online

March 31, 2023

Deputy Director General, Heritage Policy and Programs Branch; Director General, Modernization

2.2. MAP HQ and MAP regional offices to examine MAP’s intake cycle and provide recommendations.

2.2.1. Recommendations to the Deputy Director General

March 31, 2023

Deputy Director General, Heritage Policy and Programs Branch

2.3. MAP HQ and MAP regional offices to implement the approved intake cycle recommendation

2.3.1. Implementation of Recommendation

March 31, 2024

Deputy Director General, Heritage Policy and Programs Branch

Full implementation date: March 31, 2024

While improvements have been made to performance measurement, there are inconsistencies between the MAP logic model and PIP, as well as gaps in data for certain priority groups.

Since the last evaluation exercise, MAP has made progress in performance measurement, developing and implementing a detailed, program specific reporting methodology to track results. However, while it is acknowledged that challenges caused by the COVID-10 pandemic prevented close collaboration with the program during parts of the project, the evaluation team was unable to reproduce the program data analysis on results. In addition, the evaluation found that the collection and reporting of data shown in the reporting methodology did not clearly align with the program’s current logic model and subsequently the PIP.

The evaluation team also noted that the program tracked a number of priority groups but only reported on, and analyzed the data from OLMCs, Indigenous groups and Indigenous projects.

Recommendation 3

The ADM, Official Languages, Heritage and Regions, should work with Strategic Planning and other internal stakeholders to:

Review the MAP performance indicators to align with government orientation with respect to priority groups and implement them;
Ensure that the program logic model is aligned with the program results and performance set out in the PIP and reporting methodology, and;
Review the program’s reporting methodology to ensure that it is reproducible and clearer.

Management response - Recommendation 3

The Heritage Policy and Programs Branch (HPPB) Management accepts this recommendation.

HPPB will review the MAP Performance Indicators and make appropriate adjustments where needed, taking into consideration:

any program changes that may result from Recommendations 1 and 2;
the Department’s ongoing Departmental Results Framework (DRF) Renewal exercise to integrate the Sustainable Development Goals (SDGs) and review the other area of reporting;
the expected Department’s PIP review resulting to the new DRF;
the Department’s orientation to report on diversity and inclusion, and how the priority groups are being reached and/or represented by the programs;
current and new systems limitations in data extraction and production of reports; and
the practicality and/or cost efficiency of different data collections or reporting strategies.

HPPB will review the program logic model to ensure it is aligned with the program results and performance, taking into consideration the DRF, PIP and program performance indicators as set out in the previous item of the current recommendation.

Furthermore, HPPB will review the program’s reporting methodology, taking into consideration the data being collected, the limitations of the current systems, the My PCH Online Pilot project, and the reporting tools.

Table 16: recommendation 3 – action plan

Action Plan Item	Deliverables	Timeline	Responsible
3.1. In collaboration with the regions and Strategic Planning, HPPB to explore and review the data collection of and reporting on priority groups.	3.1.1. Review and recommend on data collected and priority groups	October 31, 2022	Deputy Director General, Heritage Policy and Programs Branch
3.2. HPPB to review the MAP performance indicators and program logic model : Review expected results Review performance indicators Review data collection	3.2.1. Updated MAP Performance Indicators and logic model	March 31, 2023	Deputy Director General, Heritage Policy and Programs Branch
3.3. HPPB to review the programs reporting methodology	3.3.1. Revised reporting methodology as approved by Director General	October 31, 2023	Deputy Director General, Heritage Policy and Programs Branch

Action Plan Item

Deliverables

Timeline

Responsible

3.1. In collaboration with the regions and Strategic Planning, HPPB to explore and review the data collection of and reporting on priority groups.

3.1.1. Review and recommend on data collected and priority groups

October 31, 2022

Deputy Director General, Heritage Policy and Programs Branch

3.2. HPPB to review the MAP performance indicators and program logic model :

Review expected results
Review performance indicators
Review data collection

3.2.1. Updated MAP Performance Indicators and logic model

March 31, 2023

Deputy Director General, Heritage Policy and Programs Branch

3.3. HPPB to review the programs reporting methodology

3.3.1. Revised reporting methodology as approved by Director General

October 31, 2023

Deputy Director General, Heritage Policy and Programs Branch

Full implementation date: October 31, 2023

Annex A: the current state of museums - looking towards the Future

The evaluation of the Museum Assistance Program covers the period of 2013-14 to 2017-18. The evaluation activity itself began in 2019-2020, and took place during the period of the COVID-19 pandemic and subsequent lockdowns. The pandemic and lockdowns affected not only the evaluation’s timeline, but, more importantly, the museum and heritage sectors, which are key stakeholders for the Museum Assistance Program. It was deemed important to present a summary of the main trends and issues facing Canada’s museum community, including a discussion of the impact of the COVID-19 pandemic, in order to bring the evaluation’s findings into the current context. To this end, an outside expert was engaged to provide an environmental scan of the sector.

Introduction

This chapter reflects on the findings of the evaluation and the current state of museums in Canada. The impacts of COVID-19 on the sector were not analyzed in this evaluation because it covers the pre-pandemic period, but they cannot be ignored in this response, which looks to the future. The sector was fragile before the pandemic and has been one of the most negatively affected sectors globally. We cannot predict the repercussions of repeated closures, and travel restrictions, on the future of Canadian museums.

Impacts of COVID-19 on the heritage sector

It is estimated that one third of museums in the United States and two thirds in the United Kingdom will close permanently.^{Footnote 29} Some museum staff have been laid off or had hours reduced, others able to work on site or from home during the pandemic. Some museums have been able to reopen for weeks or months between lockdowns and some have been able to respond with innovative online programming. However, many museums lack the capacity to develop digital resources and the need to do so has become more urgent. Museums that have provided resources online have developed new audiences, while those unable to do so have missed an opportunity. Museums often lack a basic understanding, beyond the number of virtual visits, of how their websites are being used, what various audiences want and need, and how digital resources can be monetized. It is also important to remember that not all Canadians have equal access to digital technology. Museums and individuals in northern and remote communities, and lower income Canadians in cities, do not have equal access.

The degree to which museums have been able to continue operating varies from one museum, one community, to another. These disruptions have resulted in stress and feelings of isolation. Museum professionals have changed retirement plans (to retire either earlier or later) and it has become more difficult for those entering the workforce to get, or keep, a job. The closures have caused interruptions in relationships between staff, volunteers, and audiences. Where possible, museum professionals that have been working from home may expect to continue doing so part-time post-pandemic. However, staff that require regular access to collections or visitors have been laid off, particularly registrars, conservators, educators, and sales staff.^{Footnote 30} Collections preservation may be at risk due to the lack of regular monitoring.

Closures have also caused disruptions in relationships in and with Indigenous and migrant communities. Where traditional forms of community consultation have been impossible, projects may have been delayed. Global social upheaval, and growing awareness of systemic racism and the disparity between rich and poor, have reinforced the need for museums to address decolonization and develop programming for marginalized communities. Some museums have not yet begun the road to reconciliation. The inability of museums to communicate effectively with Indigenous and migrant communities also impacts their ability to respond to the Truth and Reconciliation Commission’s (TRC) Calls to Action,^{Footnote 31} as well as the government priority of reaching diverse communities.

Museums have lost a significant amount of earned revenues. Earned revenues accounted for over $2.6 billion (33%) of institution budgets in 2017 (for example, sales, admissions, fundraising, registrations, and memberships).^{Footnote 32} Museum operating expenses increased when museums reopened between lockdowns because of measures required to ensure staff and visitor safety. There is concern, particularly in museums reliant on tourism, about when tourism will return to 2019 levels. It is expected that international tourism will not fully return until 2023 at the earliest. The Vancouver Art Gallery notes that earned revenue has dropped by 75% and that the wage subsidy is “literally what’s keeping us alive.”^{Footnote 33} Museum professionals question the feasibility of future blockbuster/travelling exhibitions due to many of these factors, as well as increased concern about climate change. As a result, museums may need to develop more temporary exhibitions and programming to attract repeat visitation by local audiences.

A survey conducted by Hill Strategies in May 2020 indicates that it will also take years for local visitors to return to museums:

Only 30% of previous visitors expected to visit immediately when museums reopen
15% planned to return between one and five months after they reopen
12% planned to wait six months or more, until a vaccine is available
42% of visitors were unsure when they will return.

A follow up study in August 2020 suggests that exhibiting organizations have lost 63% of their visitation and will not reach 2019 levels until at least 2022.^{Footnote 34} School groups, a significant museum audience, are not planning field trips. Some museums have distributed materials for classroom use alongside online programming. Science centre and children’s museums wonder how the public will respond to interactive exhibitions in the future and are introducing more programming for use on visitors’ own devices.

The changing needs of museums

The 2019 Survey of Heritage Institutions indicates that there were more than 2,700 Canadian museums and heritage institutions in 2017.^{Footnote 35} The 2016 census records 5,162 municipalities in Canada, 90% of which are located out of census metropolitan areas (CMAs).^{Footnote 36} Small cities, towns, villages, and rural areas throughout Canada have museums.

Many new museums that opened in recent years, and others currently in the planning stages, relate to specific communities identified by race, religion, ethnicity, or culture. They may not see themselves in existing museums and want to tell their own stories about their communities within Canada. For example, the Museum of Jewish Montreal began telling the stories of Montreal’s Jewish community online a decade ago, and while it had a project space for a few years, continues to offer virtual programming. The Chinese-Canadian Museum has ambitious plans for the future and recently opened its inaugural exhibition in a temporary space.^{Footnote 37} New forms of museums, virtual museums and pop-up museums, for example, reflect global trends in museology. Museums have incomparable reach. These organizations need access to funding to ensure sustainability, and MAP funding is not accessible to them, nor would it be sufficient. The gig economy, which the government began to recognize during the pandemic, includes museum professionals as well as artists. Granted, without an increase in funding, broadening eligibility would lead to other challenges.

Funding to museums

In 1990, the government stated the importance of MAP as its primary support for museums and committed to increasing MAP funding from $10.39 million to $18 million by 1994-95. However, MAP funding never reached $18 million and was instead reduced to its current $6.7 million. The committed $10.39 million would now be worth over $20 million; MAP’s 1972 funding would now be worth over $40 million^{Footnote 38}. While the amount of funding provided through MAP is relatively small (just 12% of overall museum budgets), it is essential to those institutions that do receive it.

Contrast this with the Canada Council for the Arts (CCA), which doubled its budget between 2015 and 2021 to $360 million, with an increase in grants to the arts sector from $150 million to $310 million.^{Footnote 39} Funding to museums is paltry by comparison. Canada Council funding is equally split between core operations and projects. Art galleries are eligible for operational funding from the CCA, but other types of museums cannot access operational funding. In 2019-20, arts organizations received $195.7 million from the CCA; 379 recipients received funding for their first time (a total of $14.5 million).^{Footnote 40} The CCA is also able to fund projects that bridge fiscal years, something MAP is unable to do as a federal program.

The CCA has always funded a broader range of projects but in recent years, in addition to increasing dollar amounts, it has been strategic in supporting First Nations, Métis and Inuit arts, first time recipients, international activities, and digital technology. The CCA is more flexible in terms of eligibility, providing operational funding to arts organizations as well as project funding to individual artists and collectives. The difference in the historical approach to funding arts and heritage has benefitted arts organizations and artists more so than heritage organizations and practitioners. Museums can access some federal programs other than MAP^{Footnote 41} so it is difficult to provide an accurate comparison with the CCA without including other programs – except to say that Canada Council funding has increased dramatically while MAP funding has decreased significantly.

In May 2020, the Department announced $53 million in emergency funding to museums and heritage organizations, with grants of $5,000-$100,000. Eligibility criteria were relaxed (for example, no requirement for full-time professional staff, thus allowing smaller organizations to apply) and the application and reporting processes simplified to allocate funds quickly. The success of this program points to the need within the sector and the ability of the department to expedite funding while ensuring accountability.

Museums: a shared responsibility

Investment in museums, museums associations, and independent museologists is a shared responsibility of federal, provincial/territorial, and municipal governments. Indigenous governments and organizations also provide funding for cultural heritage preservation. However, communication between the various jurisdictions is limited, the rationale for who funds what is unclear, and funding applications, deadlines, and reporting processes are not aligned resulting in too much time being wasted in providing the same information in different ways. Projects may be jeopardized by the failure of funding streams to align, particularly when matching funding is required. Better coordination between funders is required.

Museums have difficulty securing adequate operational funds. Neither MAP nor major funders like Foundations nor corporate sponsors provide core funds. Large museums can access more diverse income sources (e.g., earned revenues and donations by major philanthropists) than small museums, and the gap between them grows. In the post-pandemic world, it will take years for museums to get back to their 2019 level of earned revenues, as it did, particularly in Toronto, after the 2003 SARS epidemic.^{Footnote 42} Unlike SARS, the coronavirus pandemic has impacted museums throughout the country, and the world, and for a longer period.

The 2018 Standing Committee report Moving Forward – Towards a Stronger Canadian Museum Sector identified funding priorities.^{Footnote 43} Yet the funding scope for museums has not changed significantly. The collection, while still the cornerstone for many museums, is not for all. Today’s museums focus on stories and ideas as much as collections which was brought into focus by ICOM’s current debate about a new museum definition.^{Footnote 44} Museums need support for temporary, non-travelling exhibitions, and for alternative content delivery and engagement. Museums also need funding for research, specialized conservation and intervention, intangible cultural heritage, community collaborations, audience development, improving access and inclusion, and to build bridges with Indigenous communities and migrants.

Indigenous heritage

The 1994 Task Force Report on Museums and First Peoples raised expectations that the relationship between museums and Indigenous peoples would improve significantly^{Footnote 45} but, the need to improve relationships with Indigenous communities remains and has been amplified in the years since, particularly with publication of the Truth and Reconciliation Commission’s (TRC) Calls to Action in 2015.^{Footnote 46} MAP increased its flexibility and funding limits for the Indigenous heritage component following the TRC report. Indigenous Heritage funding has helped with Elder/youth programming, repatriation, building connections with Indigenous communities, and improving understanding of Indigenous heritage.

Canada did not immediately endorse the United Nations Declaration on the Rights of Indigenous peoples (UNDRIP) in 2007.^{Footnote 47} On December 3, 2020, Canada introduced Bill C-15, to “chart a path forward for implementing the individual and collective rights set out in the declaration…the bill is not written to make UNDRIP a part of federal law, but instead identifies the declaration as a human rights instrument that governments and courts can use to guide the development and interpretation of Canadian law.”^{Footnote 48} UNDRIP has implications for museums that have not yet been addressed.

There has been both an improvement in the way Indigenous heritage is managed within some traditional museums and an increase in the number and variety of First Nations, Métis and Inuit-owned museums, cultural and heritage centres. However, more needs to be done to decolonize museums. As noted in this evaluation report, PCH is providing funding to the Canadian Museums Association’s (CMA) Reconciliation Council established in response to Call to Action 67 (PDF format), which called upon the federal government to provide funding to the CMA “to undertake, in collaboration with Aboriginal peoples, a national review of museum policies and best practices” to determine the level of compliance with UNDRIP and make recommendations.^{Footnote 49}

Marginalized communities

Museums can and do facilitate the settlement process by developing programming for migrants and refugees but have not yet fully addressed concerns about systemic racism raised during the pandemic and the need to develop anti-racism projects. Other marginalized communities include the disability, mental health and justice communities, youth, and seniors. Some museums have done great work in developing programs for specific communities, often in partnership with social service organizations. However, there has been no concerted focus on these communities in Canada as there has been in other countries, particularly the United Kingdom and United States.^{Footnote 50}

Technology

Since the National Museums Policy was revised in 1990, audience expectations of museums have risen dramatically. Technology has changed the way all museum functions are performed, from collections management to preservation, research, and interpretation. Audiences expect physical exhibitions to incorporate moving images and creative technologies that enhance interactivity. They expect online programming to include access to collections through websites, virtual exhibitions, interactive databases, learning opportunities, and social media. While the number of virtual exhibitions has grown significantly, most museums continue to have limited capacity to develop online programming – whether exhibition, collections resources, or educational programs. Museum staff do not always have the expertise required to fully integrate technology in exhibitions and/or develop a sophisticated social media presence to meet the expectations of a technology-savvy audience.

Human resources

Human resources challenges include an aging/retiring workforce and difficulty attracting and retaining young professionals due to low salaries and increasingly demanding workloads. It is difficult for young professionals to enter the workplace because of few/sporadic opportunities, most of which are term or project-related positions. Many museums remain heavily dependant on volunteers and continue to expect them to provide the same level and type of support they have historically when it has been well documented that volunteerism itself has changed. Museums are struggling to attract more culturally and socially diverse volunteers who are representative of the Canadian population. Today’s volunteers are more interested in episodic volunteering than in regular weekly commitments and are providing fewer hours.

International programs

The recent Senate report Cultural Diplomacy at the Front Stage of Canada's Foreign Policy^{Footnote 51} largely overlooked the potential for museums and other heritage institutions to contribute to cultural diplomacy. However, Canadian museums, museums associations, and individuals do support cultural diplomacy in many ways. A few examples:

With membership in nearly fifty of the fifty-four Commonwealth nations, the Edmonton-based Commonwealth Association of Museums reflects on colonial legacies and develops new international relationships and working practices, in part by profiling Canadian heritage practice and approaches to decolonization;^{Footnote 52}
The Coalition of Museums for Climate Justice “mobilizes and supports Canadian museum workers and their organizations in building public awareness, mitigation and resilience in the face of climate change,”^{Footnote 53} demonstrating leadership on climate justice for museums globally;
Yukon's ongoing Ice Patch Research Project has stimulated international collaborations between Indigenous governments, scientists, and conservators since the 1990s and the patches were recently added to Canada’s Tentative List for World Heritage Sites;^{Footnote 54}
And finally, since 2000, the Royal Alberta Museum and Hokkaido Museum (Sapporo, Japan) have had an ongoing relationship that involves continuing staff exchanges and associated research projects, exhibitions, scholarly activity, publications, and lectures, while promoting and maintaining cultural and research ties.^{Footnote 55}

While the focus of MAP is to expose the greatest possible number of Canadians to our national heritage, the Canada-France Agreement demonstrates commitment to developing international partnerships and reaching new audiences in France.

Conclusion

The Standing Committee on Canadian Heritage report recommended that the National Museums Policy of 1990 be modernized, and MAP expanded to provide more sustainable funding. It would take an increase to $20 million to return to 1990 levels and $35 million to reach the level promised at that time. The need to review the policy was included in the mandate letter to the Minister of Canadian Heritage^{Footnote 56} and announced with great enthusiasm by the CMA which has recommended an increase to at least $60 million.^{Footnote 57} A supplementary mandate letter, dated January 15, 2021, states that the Minister should, in addition to the priorities identified previously, “…introduce sector-specific measures for industries that have been hit hardest by the impacts of the COVID-19 pandemic, including the art, cultural, heritage and sport sectors...”^{Footnote 58} It also refers to the importance of issues such as reconciliation and climate change, and the need to address systemic injustice, the latter of which has been demonstrated recently by events in several high profile Canadian museums.

Museums have changed dramatically in the past 30 years – and more rapidly in the past year due to COVID-19, which has highlighted the need to address issues such as the ability to hire and retain young professionals, the need to improve access to new technologies in museums, to continue to address reconciliation and systematic racism within Canadian museums, and to support the work of the museum sector internationally.

Annex B: evaluation framework

Relevance: Q. 1.1. Is MAP responding to the current and changing needs of museums and heritage institutions of varying sizes, and to their professionals’ requirements?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
1.1.1. Views, trends and important issues facing museums and institutions of varying sizes and their professionals	Yes	-	Yes	Yes	Yes	Yes	Yes
1.1.2. Views, trends and important issues facing the Canadian museum industry	Yes	-	Yes	-	Yes	Yes	Yes
1.1.3. Trends in MAP applications according to MAP components and sub-components	Yes	Yes	-	-	Yes	Yes	-
1.1.4. Number of applications to MAP; Number of funded and of non-funded applications to MAP	-	Yes	-	-	-	-	-
1.1.5. Views and measurement of the level of alignment between needs in the sector and MAP support	-	-	Yes	Yes	Yes	Yes	Yes
1.1.6. Ratio between MAP financial resources and funded applicants’ budget	-	Yes	-	-	Yes	-	-

Relevance: Q. 1.2. Does MAP support government priorities such as reconciliation, GBA Plus and OLMCs?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
1.2.1. Number of MAP funded applications pertaining to priority groups	-	Yes	-	-	Yes	-	-
1.2.2. Views and extent to which needs of these priority groups have been taken into account by MAP	Yes	-	Yes	-	Yes	-	-

Effectiveness: Q. 2.1. Short-Term outcome 1 - Travelling exhibitions are produced
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.1.1. Number and locations of travelling exhibitions produced during the evaluation period	-	Yes	-	-	-	-	-
2.1.2. Number of entries to these travelling exhibitions	-	Yes	-	-	-	-	-

Effectiveness: Q. 2.2. Short-Term outcome 2– Projects to improve the preservation and presentation of Indigenous heritage implemented
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.2.1. Views on funded projects related to Indigenous heritage	-	-	-	-	Yes	Yes	Yes
2.2.2. Number of funded projects related to Indigenous heritage	-	Yes	-	-	-	-	-

Effectiveness: Q. 2.3. Short-Term outcome 3 – Opportunities are created for heritage institutions and workers to enhance their professional knowledge, skills and practices
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.3.1. Number of training opportunities funded by MAP for professionals and institutions in the Canadian museum sector	-	Yes	-	Yes	-	-	-
2.3.2. Views on training opportunities by professionals and institutions	-	-	-	Yes	Yes	Yes	Yes

Effectiveness: Q. 2.4. Medium-Term outcome 1 - Heritage exhibitions and other public programming products/activities are presented to the public
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.4.1. Views and number of funded MAP applications	-	Yes	-	-	Yes	Yes	Yes
2.4.2. Views and number of non-funded MAP applications	-	Yes	-	-	Yes	Yes	-
2.4.3. Number of travelling exhibitions and other products presented to the public	-	Yes	-	-	-	-	-
2.4.4. Locations and distribution of travelling exhibitions and other products presented to the public	-	Yes	-	-	-	-	-
2.4.5. Number of visitors of the travelling exhibitions and other products	-	Yes	-	-	-	-	Yes

Effectiveness: Q. 2.5. Medium-term outcome 2 – Heritage objects and collections are better managed and preserved
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.5.1. Views and number (and percentage) of professionals that reported an improvement in the management and preservation of heritage objects and collections	-	Yes	-	Yes	Yes	Yes	Yes
2.5.2. Number of projects better managed and preserved	-	Yes	-	-	-	-	-

Effectiveness: Q. 2.6. Medium-Term outcome 3 – Heritage institutions and workers have enhanced their professional knowledge, skills and practices
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.6.1. Views and number of professionals that improve their knowledge, skills and practices in the sector	-	Yes	-	Yes	Yes	Yes	Yes
2.6.2. Number of institutions in the sector funded by the MAP	-	Yes	-	-	-	-	-

Effectiveness: Q. 2.7. Long-term outcome – Canada’s Heritage is accessible to Canadians over time
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.7.1. Views and trends in the context of museums and heritage institutions in Canada	Yes	-	Yes	-	Yes	Yes	Yes
2.7.2. Number of funded applications by MAP	-	Yes	-	-	-	-	--
2.7.3. Number and percentage of institutions that reported an increased capacity in long term access to Canadian heritage	Yes	Yes	-	-	-	-	-

Effectiveness: Q. 2.8. Were there any unanticipated outcomes or challenges as a result of MAP activities?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
2.8.1. Evidence and views of unexpected results, outcomes or impacts of the MAP	Yes	-	-	Yes	Yes	Yes	Yes

Efficiency Q. 3.1. Was the MAP delivered efficiently?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
3.1.1. Program administrative/ operational costs in relation to overall budget (by component)	Yes	Yes	-	-	-	-	-
3.1.2. Funding applications processed by FTE (by component)	Yes	Yes	-	-	-	-	-
3.1.3. Trends in planned vs utilized (actual) financial and human resources	Yes	Yes	-	-	-	-	-
3.1.4. Achievement of service standards	Yes	Yes	-	Yes	Yes	-	-
3.1.5. Views on the management and delivery of the MAP	-	-	-	Yes	Yes	Yes	Yes

Efficiency: Q. 3.2. Was the notional allocations of funding efficient?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
3.2.1. Views and evidence on the notional allocation of funds	Yes	Yes	-	Yes	Yes	-	-

Efficiency: Q. 3.3. Does the performance measurement framework for MAP allow for the efficient reporting on the Program’s results?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
3.3.1. Performance measurement used	Yes	Yes	-	-	Yes	-	-
3.3.2. Program results to report	Yes	Yes	-	-	Yes	-	-
3.3.3. Views on the performance measurement framework and its efficiency for reporting	-	-	-	-	Yes	-	-

Other: Q.4.1. What lessons or best practices have been learned from MAP’s innovation projects?
Indicators	Document review	Admin data & file review	Literature review	Case Studies	Key informant interviews: Program managers and staff	Key informant interviews: PTMAs	Key informant interviews: Funded applicants
4.1.1. Evidence and views of lessons or best practices from MAP’s experimentation and innovation projects	-	Yes	-	Yes	Yes	Yes	Yes

Annex C: program logic model

Annex D: bibliography

American Alliance of Museums. (2020a). Lessons Learned from SARS (and Strikes) at the Ontario Science Center.
American Alliance of Museums. (2020b). National Survey of COVID-19 Impact on United States Museums.
Artlyst. (2020). “60% UK Museums Facing Permanent Closure.”
Assembly of First Nations/Canadian Museums Association. (1994). Turning the Page: Forging New Partnerships Between Museums and First Peoples. Task Force Report on Museums and First Peoples. (PDF format)
Canada Council for the Arts. (2020). Annual Report 2019-20.
Canadian Association for Conservation of Cultural Property. “Summary from the survey assessing the impact of COVID-19 on Canadian Conservation Professionals: Part I. (PDF format)
Canadian Heritage. (2015). Evaluation of the Canadian Museums Assistance Program, 2008-09 to 2012-13 [PDF version - 1.3 MB].
Canadian Heritage. (2017). 2016-17 Report on Plans and Priorities. (PDF format)
Canadian Heritage. (2019). Government of Canada Survey of Heritage Institutions: 2019.
Canadian Heritage. (2019). 2019-2020 Departmental Plan.
Catherine C. Cole & Associates. (2005). Portrait of Canadian Public Art Galleries and Museums.
Government of Canada. (1990). Canadian Museums Policy.
Government of Canada. (1990). Museums Act (S.C. 1990, c.3).
Hill Strategies. (2020). Arts Response Tracking Survey.
House of Commons Canada. (2018). Moving Forward – Towards a Stronger Museums Sector: Report of the Standing Committee on Canadian Heritage. (PDF format)
ICOM. (2020). Museums, museum professionals and COVID-19: follow-up survey. (PDF format)
Langlois, Marie-Claude. (2013). Canada’s National Museums: Background Paper. Ottawa: Library of Parliament. (PDF format)
The Art Newspaper. (2019). “Vote on Icom’s new museum definition postponed.”
Truth and Reconciliation Commission of Canada. (2015). Truth and Reconciliation Commission of Canada: Calls to Action. (PDF format)

Page details

Date modified:: 2021-11-25

Evaluation of the Museums Assistance Program 2013-14 to 2017-18

On this page

List of tables

List of figures

Alternate format

List of acronyms and abbreviations

Executive summary

Program description

Evaluation approach and methodology

Findings

Relevance

Effectiveness

Efficiency

Other – Innovation

Recommendations

1. Introduction

2. Program profile

2.1. Program history

2.2. Program objectives, expected outcomes and activities

Access to Heritage

Indigenous HeritageFootnote 12

Collections ManagementFootnote 13

2.3. Program management and governance

2.4. Program resources

Table 3 notes

2.5. Results of the previous evaluation

3. Approach and Methodology

3.1. Scope, timeline and quality control

3.2. Calibration

3.3. Evaluation questions

3.4. Data collection methods

Administrative review

Document review

Literature review

Key informant interviews

Case studies

3.5. Constraints, limitations and mitigation strategies

4. Findings

4.1. Relevance

4.1.1. Trends and issues facing museums

4.1.2. Extent to which MAP meets the needs of museums

4.2. Effectiveness

4.3. Efficiency

4.3.1. Administrative costs

4.3.2. Design and delivery

Table 13 notes

4.4. Other – Innovation

5. Conclusions

5.1. Relevance

5.2. Effectiveness

5.3. Efficiency

5.4. Other – innovation

6. Recommendations, management response and action plan

Annex A: the current state of museums - looking towards the Future

Introduction

Impacts of COVID-19 on the heritage sector

The changing needs of museums

Funding to museums

Museums: a shared responsibility

Indigenous heritage

Marginalized communities

Technology

Human resources

International programs

Conclusion

Annex B: evaluation framework

Annex C: program logic model

Component - Access to Heritage

Activity

Reach

Outputs

Reach

Short Term Results

Medium Term Results

Reach

Long Term Result

Reach

Component - Indigenous Heritage

Activity

Reach

Indigenous Heritage^{Footnote 12}

Collections Management^{Footnote 13}