This evaluation provides the Atlantic Canada Opportunities Agency (ACOA) management with systematic, neutral evidence on the relevance and performance of its Community-Based Business Development sub-program. The results of this evaluation will be used to support the future direction of the programming.

The study was managed and implemented by ACOA’s Evaluation Unit. We thank the members of the evaluation advisory committee for their advice and support throughout the evaluation process. Their involvement and assistance has helped to ensure that the study and its recommendations are both relevant and useful. Of special note are the gracious contributions of two external advisory committee members: Sylvia Alexander, Atlantic Association of Community Business Development Corporations (CBDCs), and Maurice Turcot, Canada Economic Development for Quebec Regions. Special thanks also to Basil Ryan, Chief Operating Officer of the Atlantic Association of CBDCs for his contributions.

We are also grateful to the many ACOA staff members, including members of the Evaluation Unit, evaluation colleagues from other regional development agencies and Industry Canada, ACOA recipients (CBDCs and Ulnooweg), clients and external key informants, who provided their time and essential knowledge in support of this study.

Overall, these contributions were instrumental in building our understanding and identifying best practices and areas for improvement for ACOA’s Community-Based Business Development programming.

Courtney Amo Acting Director, Evaluation and Risk Directorate (Head of Evaluation) Atlantic Canada Opportunities Agency

Executive Summary

ACOA’s Community-Based Business Development (CBBD) sub-program focuses on activities to improve access to capital and other supports needed to strengthen small and medium-sized enterprises (SMEs) in rural communities across Atlantic Canada. The programming supports the achievement of ACOA’s Community Development (CD) program goal to foster dynamic and sustainable Atlantic communities and, more broadly, the Agency’s mandate to increase opportunities for economic development in Atlantic Canada.

The majority of activities undertaken under the CBBD are supported through the Community Futures Program (CFP), a national transfer payment program currently administered by five departments and agencies. In Atlantic Canada, the CFP is administered by ACOA, supported by a network of 41 Community Business Development Corporations (CBDCs). The balance of CBBD activities utilize a combination of the Business Development Program and Innovative Communities Fund transfer payment programs.

Over the six-year period of the study (2007-2008 to 2012-2013), CBBD programming represented nearly 16 per cent of overall CD program expenditures ($769 million) and 6 per cent of total Agency expenditures ($2,171 million).

The purpose of this evaluation is to assess the relevance and performance of CBBD programming. The evaluation fulfills Government of Canada accountability requirements as well as the information needs of ACOA’s senior management. It was undertaken according to the terms of reference for the study, approved by ACOA’s executive committee in December 2012.

The evaluation methodology included interviews with 64 key informants, a CBDC recipient survey (response rate of 41 per cent), a CBDC client survey (response rate of 9 per cent), CFP performance analysis based on Statistics Canada data, as well as administrative and financial analysis incorporating data from a variety of sources. Evaluation findings are based on a high level of convergence of multiple lines of evidence and are deemed reliable and valid within the context of the study limitations.

Conclusions

Relevance

Overall, the evaluation finds that the CBBD sub-program remains relevant. The needs that the programming are expected to meet exist to at least the same extent as they did five years ago. CBBD activities are aligned with ACOA’s strategic outcome and the ACOA Act and are consistent with the Agency’s roles and responsibilities and broader federal priorities related to economic development.

Access to capital remains an important need of rural entrepreneurs in Atlantic Canada. ACOA is aware of existing and emerging needs and is adjusting its programming in response, largely through the implementation and monitoring of the Community Futures of Tomorrow (CFoT) model, introduced in 2011-2012.

Activities undertaken as part of the CBBD programming are relevant for ACOA and the federal government and to the needs of local, rural communities. CBBD activities continue to be aligned with ACOA and broader federal priorities related to economic development. CBDC activities align with the needs of local, rural communities. Financial as well as non-financial services are important to SMEs.

CBBD programming is consistent with the roles, responsibilities and mandate of similar programming delivered by other federal departments and agencies across Canada. Within Atlantic Canada, CBBD programming complements rather than duplicates similar programming delivered by other organizations. The programming has unique qualities: local community-based governance and delivery; suite of support services in addition to financing; and the ability to support higher risk SMEs.

Performance -Effectiveness

Overall, the evaluation finds that the CBBD programming is effective in that it is incremental to the implementation of projects and the achievement of outcomes. Factors impeding the success of CBBD programming are known and many are being mitigated. Significant progress has been made in ensuring availability and use of performance measurement information, allowing for better and timelier programming decisions and improved communication of results.

ACOA provides important financial support to community economic development projects in Atlantic Canada through its CBBD sub-program. Its funding is incremental to the activities of the CBDCs and to the associations that support them as well as to Ulnooweg. Program funding positively impacts the creation and expansion/modernization of SMEs in rural Atlantic Canada, and many projects would not have proceeded without the Agency’s support.

CBBD programming is achieving expected outcomes by providing SMEs with access to capital to start, modernize and expand businesses. The sub-program has contributed to the creation and the expansion/modernization of SMEs in rural Atlantic Canada.

ACOA and CBDCs are aware of the internal and external contextual factors that impede the success of CBBD programming, and it is evident that many are being mitigated. Some factors would benefit from further investigation: opportunities for consistency in service delivery; succession planning; and gaps in skills development.

Early evidence suggests that the CFoT is on track to meet its stated objectives. Further time will be required in order to assess its impact on program effectiveness. There are questions related to the risk of unintended behaviours resulting from the CFoT as well as the ongoing need for and composition of a governance committee.

ACOA and funding recipients have made significant progress in ensuring the availability and use of performance measurement information, allowing for better and timelier programming decisions and improved communication of results. Gaps include client contact information, data on non-financial services and certain Community Futures (CF) performance measurement strategy (PMS) collection data.

Performance – Efficiency and Economy

The evaluation finds that the CBBD programming demonstrates efficiency and economy in the utilization of resources. The existing model of delivering the majority of CBBD programming through CBDCs and Ulnooweg allows the Agency to achieve an important part of its mandate in an efficient and economical fashion.

CBDC associations add efficiencies through coordination and cost-saving measures, including the CFoT funding model, the Atlantic Canada Community Business Investment Fund, management information systems and group purchasing initiatives. The CFoT funding model appears to have improved governance and investment fund management tools, though it is too early to fully assess its impact. Since the previous evaluation, ACOA and the CBDCs have adopted a continuous improvement approach to program management and monitoring. This has allowed them to measure and monitor performance and continue to explore delivery alternatives. CBDC management of loan funds is now supported by more consistent measures. Better collaboration between Ulnooweg and the CBDCs holds potential for additional efficiencies.

Recommendations

The evaluation has issued three recommendations aimed at building upon progress made to program delivery, performance and efficiency since the previous evaluation while ensuring that ACOA identifies and considers emerging programming needs on an ongoing basis.

Recognizing the progress that has been made since the previous evaluation, continue to improve the performance of the CBBD programming. There have been significant improvements to the delivery of the programming, particularly through the recent implementation of the CFoT model. In the spirit of continuous improvement, it is recommended that ACOA program management:
1. work in co-operation with the CBDCs and their associations to increase understanding of the CFoT model among CBDC and ACOA staff and to foster the achievement of objectives and expected behaviours while minimizing risks. In particular, it is recommended that governance mechanisms be reviewed to better understand and reflect the complex relationships between ACOA and the CBDC network and the ongoing work of the CFoT Oversight Committee;
2. support the CBDCs and their associations in exploring opportunities for greater consistency of service delivery in areas that hold the greatest potential for enhancing effectiveness and efficiency of the programming; and
3. engage in dialogue with the CBDCs, their associations and Ulnooweg to identify areas for enhanced collaboration, with the aim of improving performance and efficiency of programming for Aboriginal entrepreneurs across Atlantic Canada.
Further advance policy research and knowledge of economic development in rural regions to ensure that emerging programming needs are identified and addressed and that programming remains relevant and effective. It is recommended that ACOA program management, in co-operation with the CBDCs and their associations, consider ongoing research on economic development in rural regions to further support the strategic direction of the CBDCs and Ulnooweg. Particular areas of knowledge development could include the geographical evolution of economic activities, changing demographics, access to capital and the extent and impacts of the loss of banking institutions in rural communities, business succession planning and other supports for SMEs in rural communities.
Continue to strengthen the availability and use of performance measurement information in support of results-based management and decision making. It is recommended that ACOA program management work in collaboration with CBDC associations (in support of their members) to improve information technology and tools as well as to address remaining gaps in performance information by:
1. supporting the standardized collection by all the CBDCs of client contact information necessary for program management, performance measurement and evaluation purposes. This should include contact information for clients receiving financial and/or non-financial services;
2. examining the non-financial services offered by CBDCs (e.g. provision of business information, counselling and skills development) in order to identify data that could be used to describe, track and better understand the need for and utilization of these services. Data collection should be focused on value-added information for the management of the CBDCs and should be integrated into the existing reporting platform; and
3. supporting ongoing work through the National Community Futures Performance Measurement Strategy Committee to review the implementation of the strategy, including the validation of indicators, collection of data and setting of targets.

Management has agreed with this evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations, can be found in Appendix A.

1. Introduction

This report presents the findings, conclusions and recommendations of the evaluation of ACOA’s Community-Based Business Development sub-program. The evaluation reports on the relevance and performance of CBBD programming as per the Agency’s approved evaluation plan and the study’s approved terms of reference, which take into consideration the needs of senior management. It also fulfills accountability requirements set forth by the Treasury Board (TB) Policy on Evaluation, the Directive on the Evaluation Function, and the Standard on Evaluation for the Government of Canada.

The CBBD sub-program focuses on activities to improve access to capital and other supports needed to strengthen SMEs in rural communities across Atlantic Canada. The programming supports the achievement of ACOA’s Community Development program goal to foster dynamic and sustainable Atlantic communities and, more broadly, the Agency’s mandate to increase opportunities for economic development in Atlantic Canada.

The majority of activities undertaken under the CBBD are supported through the CFP, a national program currently administered by five departments and agencies including ACOA.[See Footnote 1] Each partnering department/agency is required to complete an evaluation of their CFP by June 2014. In Atlantic Canada, the CFP is mainly used to support Community Business Development Corporations (CBDCs) as well as CBDC associations.

The balance of CBBD activities utilize a combination of the Business Development Program (BDP) and Innovative Communities Fund (ICF) transfer payment programs. Until February 2011, ACOA used the BDP to provide contributions to CBDCs and other organizations to help Atlantic Canadians (at any age) acquire the business expertise, advice, training and capital they needed to start a business under the Seed Capital Initiative (SCI).[See Footnote 2] The initiative also enabled these organizations to provide assistance to young Atlantic Canadians (under 35 years of age) to modernize and expand their businesses. Since April 2011, the SCI no longer exists as a stand-alone initiative in rural Atlantic Canada, but funding continues to be accessible as part of the CBDCs’ new series of loan products and training fund targeting the same clientele supported by the SCI. Through the CBBD, the Agency continues to support the operations of five urban, not-for-profit organizations, to address investment capital and skills development gaps for small businesses, as well as the Ulnooweg Development Group, which assists Aboriginal entrepreneurs through support services and access to capital financing.

The evaluation covered a six-year period, from 2007-2008 to 2012-2013.[See Footnote 3] Research was undertaken between October 2012 and January 2014. ACOA’s Evaluation Unit planned and conducted the evaluation with a consulting firm providing assistance in the development and implementation of the surveys.[See Footnote 4] An evaluation advisory committee (EAC), with representatives from ACOA management and staff as well as external stakeholders with specialized knowledge in economic development and evaluation, provided advice and guidance throughout the evaluation.

The report is structured as follows: sections 1 and 2 provide an overview of the evaluation approach and a profile of the CBBD sub-program. Sections 3 to 5 present the evaluation study’s findings by broad area of relevance, performance-effectiveness, performance-efficiency and economy. Section 6 presents the conclusions and recommendations.

Management has agreed with the evaluation’s recommendations. The management action plan, which contains ACOA’s response to and planned actions for each of the evaluation’s recommendations can be found in Appendix A.

1.1 Evaluation Overview

This evaluation provides timely, credible and neutral information on the relevance and performance of CBBD programming in order to support decision making, continuous improvement and results-based management.

Two key initiatives contributed to the planning phase of this evaluation. First, an extensive planning study for the CD program conducted in 2012-2013 helped to clarify the objectives and scope of the CBBD evaluation. It also resulted in the development of a draft Community Economic Development Conceptual/Analytical Framework, which contributed to the development of methodologies and the interpretation of findings (Appendix B). Second, the national Community Futures PMS developed in 2010 in collaboration with departments and agencies who deliver the program was a key source of information used in the development of the CBBD evaluation framework (Appendix C).

1.2 Evaluation Design and Methodology

This is the first evaluation of the CBBD sub-program. The majority of the Agency’s CBBD-related programming was last evaluated in the context of the evaluation of the CFP in 2009. While the 2009 evaluation found that the CFP was relevant and was largely achieving its intended results, the report made seven recommendations[See Footnote 5] related to performance measurement, communication, working with partners and cost-effectiveness.

The evaluation of the CBBD sub-program addresses five core issues that fall into two broad categories – relevance and performance – in accordance with the TB Policy on Evaluation. Table 1 identifies the specific evaluation questions for each core issue as per the approved terms of reference for the study approved by the EAC.

Table 1: CBBD Evaluation Questions by Core Issue

Issue	Evaluation Question
Relevance
Issue 1: Continued Need for the Program
1.1	To what extent does the CBBD sub-program continue to address a demonstrable need?
1.2	To what extent is the CBBD sub-program responsive to existing and emerging needs of Atlantic Canadians?
Issue 2: Alignment with Government Priorities
2.1	To what extent is the CBBD sub-program aligned with federal government priorities and expectations, and to ACOA’s strategic outcome?
Issue 3: Alignment with Federal Roles and Responsibilities
3.1	To what extent does the CBBD sub-program align with federal roles and responsibilities? What other mechanisms exist to address these needs?
Performance
Issue 4: Effectiveness
4.1	Incrementality: What impact would the absence of CBBD funding/assistance have on the start-up, survival and growth of a SME?
4.2	To what extent have the CBBD outcomes been achieved? To what extent do the results achieved for the CBBD sub-program align with the immediate, intermediate and ultimate outcomes of the CFP?
4.3	To what extent have unintended impacts been achieved from CBBD programming?
4.4	What are the barriers to achieving CBBD immediate, intermediate and ultimate outcomes, and to what extent are these being mitigated?
4.5	To what extent have the CBBD performance measurement and reporting structures been implemented and effectively contributed to the reporting of CBBD outcomes? How is the performance information being used by ACOA and ACOA-funded organizations to support decision making?
Issue 5: Efficiency and Economy
5.1	To what extent is the CBBD sub-program efficient in the context of the results being achieved? To what extent do CBBD funding recipients have access to the tools and knowledge needed to operate efficiently? Has the Community Futures of Tomorrow ( CFoT) funding model had a positive impact on the effectiveness and efficiency of the CFP under CBBD?
5.2	Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned?
5.3	Are CBDC investment funds well managed? Are CBDC loan loss rates acceptable? Do the CBDCs carry an acceptable level of risk?

Judgment criteria or “benchmarks” for successful achievement for each evaluation question were developed with advice from program management. These criteria were considered in determining data needs, methodology and analysis. They were used by the evaluation team to objectively and transparently assess the relevance and performance of the programming.

As required by the TB Policy on Evaluation (2009), the CBBD evaluation was undertaken using a risk-based approach to evaluation design. Based on the results of a document review and stakeholder consultations, the level of effort associated with conducting the evaluation was calibrated to reflect risks associated with the relevance, effectiveness and efficiency/economy of the programming. In consideration of the evaluation issues, risks and judgment criteria, the evaluation team chose a mixed-methods research design involving multiple lines of evidence.

A mixed-methods approach allowed for triangulation (i.e. convergence of results across lines of evidence) and complementarity (i.e. developing better understanding by exploring different aspects of an evaluation issue). Taking into account the National Community Futures Performance Measurement Strategy Evaluation Framework, as well as the conceptual/analytical framework developed outlining the impact of the Agency’s CD programming the following methods were used for the CBBD evaluation:

Document and literature review – The evaluation team conducted a comprehensive review of relevant internal and external documents, including scholarly and grey literature. A review of background documentation related to the CD program, the CBBD sub-program, funding programs (e.g. CFP, BDP, ICF) and other documents deemed relevant to the programming was undertaken.

Administrative data review – The evaluation analyzed project data from ACOA’s administrative and financial databases as well as performance data from the CBDCs (i.e. E-Report data) related to the indicators identified in the Community Futures PMS. The Agency’s QAccess database provided an in-depth look at CBBD projects, including the types of activities funded. Project data was supplemented by resource data from ACOA’s financial (i.e. GX system) where possible.

Statistics Canada data analysis[See Footnote 6] – As specified in the National Community Futures Performance Measurement Strategy, ACOA along with CFP partnering departments/agencies entered into a contract with Statistics Canada to obtain a series of data tabulations to assist with measuring Community Futures (CF) performance. Using a list of client business numbers provided by CFP partnering departments/agencies, Statistics Canada provided data on employment, sales, and entry and exits. The data received from Statistics Canada was presented at an aggregate level, divided by sector and/or firm size if deemed necessary. Statistics Canada was only responsible for providing the data tables; each individual department/agency was responsible for carrying out their own analysis and reporting as per agreed upon methodologies.

To date, three data sets have been provided by Statistics Canada, each covering a five-year period (2003-2008, 2004-2009 and 2006-2010). Due to two-year time lags in accessing the data, 2006-2010 was the most current data available at the time of the study. The data was used to calculate the employment growth, sales growth and business survival rates of Community Futures (CF) organization clients and comparable firms and to classify by number of employees. The results from each performance indicator were presented in a manner illustrating the change in performance gap between CF organization clients and comparable firms with the objective of maintaining or improving the performance gap over time.

Interviews – In-depth interviews were conducted with 64 key informants, including a cross-section of internal and external stakeholders. In total, 56 interviews were held with 64 people, including 36 internal and 28 external key informants. The internal informants included 11 directors general, 14 directors/managers and 11 development officers. In addition to CBDC associations and other external key informants, two representatives from Ulnooweg were interviewed.

Client and recipient surveys – Recognizing that the majority of CBBD programming is focused on CBDCs, two electronic surveys were conducted.[See Footnote 7]

CBDC recipient survey: Based on total population of 416 CBDC executive directors and board members, 165 completed surveys were required to meet a 90 per cent confidence level (with a margin of error +/- 5 per cent). The target was achieved, with 169 surveys completed, representing a response rate of 41 per cent.
CBDC client survey: With a total population of 8,252 unique CBDC clients, a stratified random sample of approximately 3,410 clients from all CBDCs were invited to participate in the survey. Given the high number of clients in the sample, 265 completed surveys were required to meet a 90 per cent confidence level (with a margin of error +/- 5 per cent). The target was achieved, with 290 surveys (e-mail and telephone) completed, representing a response rate of 9 per cent.

1.3 Evaluation Strengths, Limitations and Mitigating Measures

The evaluation design and implementation are appropriate for the objectives of the study. A key strength of the evaluation design was the consideration of the broader program theory, supported by the Community Economic Development Conceptual/Analytical Framework. Another important strength was the participatory approach to stakeholder engagement, including the use of an EAC, whose members acted as “champions” in the development and implementation of particular methods. The evaluation team also facilitated a discussion of preliminary findings and conclusions with the EAC and held consultations with each regional vice-president to discuss recommendations.

As with all ACOA programming, a variety of external factors affect the achievement of expected results of the CBBD sub-program. Through an assessment of incrementality, the evaluation attempted to gauge the degree to which results could be attributable to CBBD programming. The evaluation aimed to assess the contribution of the Agency`s programming to the achievement of outcomes, especially at the intermediate and longer term levels. The Community Economic Development Conceptual/Analytical Framework, developed during the planning phase as well as the CFP and CBBD logic models were used in the analysis of results. These models, together with Statistics Canada analyses comparing SMEs that received CBDC assistance with comparable firms, serve to strengthen the contribution analysis and reduce uncertainty due to attribution limitations. Two limitations relating to the Statistics Canada analysis were noted: 1) the inability to ascertain whether the difference in the performance between CFP-assisted and comparable firms is significant and 2) possible program selection bias.

One of the limitations encountered during the study – gaps in client contact and other administrative data – was similar to what was encountered during the 2009 Evaluation of the CFP. Other limitations of this study included: challenges in comparing survey results to those of the previous evaluation period, in part due to weaknesses in the design of the 2009 survey; challenges in assessing loan management due to the merging of investment funds and changes to financial data collection during the evaluation period; timing delays related to fielding the surveys; and some inconsistent data for the period of the evaluation related to the introduction of the CFoT funding model in 2011-2012 as well as to the finalization of a national CF performance measurement strategy in 2012-2013, which resulted in the establishment of a new set of indicators.

The diversity of methods helped compensate for the inherent limitations of each data source and helped mitigate the overall study challenges. Multiple lines of evidence gathered through a mix of qualitative and quantitative methods allowed for triangulation – meaning convergence of results across methods. The approach also allowed for complementarity of findings so that a better understanding of results was gained by exploring different facets of complex issues associated with economic development.

Given these mitigation measures, the evaluation team is of the opinion that the limitations of the study were adequately addressed and the results of the CBBD sub-program evaluation are deemed to be reliable and valid. A complete list of study limitations, along with mitigation measures, are presented in Appendix D.

2. Profile of the Community-Based Business Development Sub-program

2.1 Context

Within the Agency’s program alignment architecture, CD is one of four program areas that support the Agency’s overall strategic outcome of developing “a competitive Atlantic Canadian economy.” CD programming focuses on creating dynamic and sustainable Atlantic Canadian communities by working with communities and entrepreneurs to stimulate economic growth, improve infrastructure and develop opportunities in local economies.

ACOA’s Community-Based Business Development sub-program contributes to addressing a significant barrier to economic development – the lack of business capital. This lack of capital and related supports such as advice and information impede the creation and expansion of small businesses, particularly those in rural areas. Small and medium-sized enterprises (SMEs) are essential to the vitality and sustainability of rural communities.

2.1.1 Community Futures Program

The majority of CBBD sub-program activities are funded through the CFP, which is a national program administered by five departments and agencies: ACOA; Canada Economic Development for Quebec Regions; Western Economic Diversification Canada; Federal Economic Development Agency for Southern Ontario; and, the Federal Economic Development Initiative for Northern Ontario, under Industry Canada. In Atlantic Canada, the CFP is administered by ACOA, supported by a network of 41 CBDCs.[See Footnote 8]

The CFP assists communities in developing their economies by supporting the growth of small businesses, primarily through financial assistance in the form of loans, the provision of counselling, entrepreneurship development support and training to individuals and small businesses. The objective of the CFP is to assist communities to pursue:

economic stability, growth and job creation;
diversified and competitive local and rural economies; and
economically sustainable communities.

Not all government departments/agencies delivering the CFP engage in all its activities. The CFP undertook a unique delivery structure in Atlantic Canada whereby CBDCs are primarily responsible for providing access to capital for SMEs along with counselling, business management skills training and advice, while community economic development strategic planning was coordinated by other not-for-profit organizations and funded through the BDP.[See Footnote 9]

This delivery approach existed before ACOA became involved but was largely due to rural/urban limitations of CBDC coverage, which were often not an issue for other not-for-profit organizations.

CBDCs are autonomous, not-for-profit corporations that work in co-operation with all levels of government and the private sector. Each is guided by a volunteer board of directors representing the interests of the community. Each CBDC manages an independent investment fund that is used to support SMEs. The distribution of CBDCs across Atlantic Canada is shown in Figure 1.

Figure 1: Map of CBDCs in Atlantic Canada

(D)

CBDCs assist in the creation of small businesses and in the expansion and modernization of existing businesses by providing financial and technical support to entrepreneurs as follows:

Financial assistance not normally exceeding $150,000 is available in the form of loans, loan guarantees and equity financing to existing and aspiring entrepreneurs.
Business counselling and advice is available to small businesses. CBDCs help businesses to succeed and, therefore they give high priority to the advisory role of their mandate.
Entrepreneurship development and business management skills training to individuals and small business owners/managers through many CBDC offices.
Technical assistance usually takes the form of guidance and coaching, and sometimes advocating on behalf of clients to other lending establishments or regulatory agencies.

CBDCs are supported throughout Atlantic Canada by Associations at the provincial and Atlantic levels. Each Atlantic province has a provincial CBDC association that, in turn, is represented on the Atlantic Association of CBDCs and the Community Futures Network of Canada. ACOA provides contributions to provincial associations and the Atlantic Association of CBDCs, permitting them to offer a suite of services, including coordination and liaison, marketing, training and cost-saving initiatives such as group purchasing of IT supports and benefits programs.

In addition to coordination and liaison roles, the Atlantic Association of CBDCs administers the Atlantic Canada Community Business Investment Fund (ACCBIF). The ACCBIF is a regional investment fund that CBDCs with surplus funds can invest in and those with excess demand can borrow from to meet loan demands. ACOA has invested in the ACCBIF to ensure that adequate loan capital is available for the region’s CBDCs and their clients.

A new funding model called the Community Futures of Tomorrow (CFoT)[See Footnote 10] was implemented in 2011-2012 to maximize the efficiency and effectiveness of resources within the CBDC network in order to enhance CBDCs’ ability to provide access to capital and business counselling to SMEs in rural Atlantic Canada. The model was the result of an extensive collaborative effort and was influenced, in part, by the recommendations identified in the 2009 CF evaluation and audit reports.[See Footnote 11]

The CFoT funding model was implemented in 2011-2012 to address distribution of operational funding and loan management issues raised in the 2009 CFP evaluation. The CFoT model uses results on lending activities and portfolio management to determine the acceptable level of funding needed by the CBBD organizations to accomplish the objectives and achieve the results required of the CBBD activities.

The CFoT model represents a fundamental shift in how CBDC funds are managed, moving from several fund silos to an integrated fund approach. This approach, in addition to injecting equity into “undercapitalized” CBDCs, also aims to address issues surrounding inadequate levels of investment capital to meet the demand for eligible loans. Operational funding is based on loan activity levels and the financial needs of individual CBDCs. Part of the CFoT model’s aim is to:

make access to loans more consistent across the Atlantic region;
address the needs of specific target groups (general business, youth, first-time entrepreneur, innovation and social enterprise);
use current resources effectively;
ensure large unused cash balances are put to work; and
maintain a strong relationship with the CBDCs.

One of the core mechanisms introduced to mitigate the risks related to the new suite of loan products, particularly those related to youth, first-time entrepreneur, and innovation is the Risk Mitigation Fund. The aim of this fund is to share the risk with CBDCs as an incentive to take on higher risk clients, particularly those in targeted groups, by providing a cash reimbursement to CBDCs worth 35 per cent of net loan losses for new loans approved under the CFoT for up to seven years from the date of disbursement of the loan.

2.1.2 Other CBBD programming

The BDP and the Innovative Communities Fund (ICF) are also used to support specific CBBD activities. Until February 2011, ACOA used the BDP to provide contributions to CBDCs and other organizations to help Atlantic Canadians (at any age) acquire the business expertise, advice, training and capital they needed to start a business under the Seed Capital Initiative (SCI). The initiative also provided assistance to young Atlantic Canadians (under 35 years of age) to modernize and expand their businesses.

Although the SCI no longer provides loans and business skills training in rural areas, loans continue to be accessible within the CBDCs’ new suite of loan products and the Entrepreneurial Training Fund. The CBBD sub-program continues to support SCI in urban areas through the following not-for-profit organizations: the Centre for Entrepreneurship Education and Development (Nova Scotia), Metro Business Opportunities (Newfoundland and Labrador), Enterprise Greater Moncton (New Brunswick), Enterprise Saint John (New Brunswick) and Enterprise Fredericton (New Brunswick).

ACOA uses the flexibility within the BDP to provide SMEs and entrepreneurs with business expertise, advice and training. It is also used to assist Aboriginal communities to improve their access to capital financing, particularly through the Ulnooweg Development Group, Atlantic Canada’s only Aboriginal capital corporation. The BDP is also used to provide targeted support to community-based, not-for-profit organizations to address investment capital gaps that focus on small businesses in other communities of Atlantic Canada.

From time to time, ACOA also uses funding mechanisms such as the ICF or special initiatives such as Canada’s Economic Action Plan (EAP) to support activities under the CBBD.

2.2 Program Theory

Programs such as the CBBD sub-program are developed to intervene in an existing process or situation in order to provide a service or solve a problem. A program’s theory is the explanation of how a given intervention is supposed to work, taking into consideration the assumptions, causal mechanisms and other factors that contribute to the achievement of expected outcomes.

2.2.1 Program Logic Model

The CBBD sub-program logic model (Figure 2) represents the inputs, activities, outputs and expected outcomes of the CBBD sub-program.[See Footnote 12] In summary, through CBBD programming, ACOA provides financial and other supports to CBDCs and other non-profit organizations, which in turn offer loans and other services to SMEs located in rural communities across Atlantic Canada.

One of the immediate outcomes of these activities is increased business financing for SMEs and entrepreneurs. This relates to the number of CBBD clients that have received funding or are able to secure capital in the fiscal year for start-ups and for the expansion of SMEs in Atlantic Canada. Activities are also expected to increase access to business services, based on the premise that capitalization and operating contributions, information and support, policies, plans, reports and program tools should lead to enhanced business development services for entrepreneurs and SMEs. At the next level, these outcomes are expected to lead to the development of business skills and knowledge, as well as an increased ability for businesses and social enterprises to secure and leverage capital. Combined, these immediate and intermediate outcomes are expected to help strengthen and expand businesses in the Atlantic region.

These outcomes align with the expected outcome of the CD program, dynamic and sustainable communities for Atlantic Canada with increased economic and business activity. The activities, outputs and outcomes related to the CBBD are summarized below. A CBBD performance measurement strategy, including a detailed narrative explaining the program theory and logic model is currently being finalized by ACOA.

Figure 2: Logic Model for the CBBD Sub-Program

Source: CBBD Performance Measurement Strategy, 2014-2015.

(D)

2.3 Program Accountability and Governance

Accountability for the CBBD rests with the ACOA Director General (DG) of Community Development, located at Head Office, together with regional directors of CD. The DG of Community Development reports directly to the senior Vice-President, Policy and Programs, while regional directors of CD report to the DG of Operations for their respective region. Enterprise Cape Breton Corporation (ECBC) has a DG of Community Development who reports internally within ECBC. In this sense, the CD function is decentralized in the regions, consistent with ACOA’s approach to other areas of programming.

The largest component of the CBBD, the Community Futures Program (CFP), is administered in Atlantic Canada by ACOA and supports a network of 41 autonomous, not-for-profit corporations through contribution agreements established according to the terms and conditions of the CFP. ACOA’s regional offices in New Brunswick, Nova Scotia, Newfoundland and Labrador and Prince Edward Island and the Enterprise Community Development Corporation in Cape Breton administer CBDC contribution agreements at a regional level, with community development account managers responsible for the daily management of the projects in each region. Within CBDCs, lending decisions are made by independent local boards of directors sensitive to the needs of their communities, with day-to-day operations managed by an executive director.

Each Atlantic province has a provincial CBDC association that, in turn, is represented on the Atlantic Association of CBDCs (AACBDC) and the Community Futures Network of Canada. The AACBDC is governed by a President and Executive Committee made up of volunteers and staff that provide oversight and guidance on projects and initiatives that support and assist CBDCs throughout the Atlantic region. Each Atlantic province has an association that is made up of volunteers who act as a governing body and who also serve as representatives on the AACBDC and the Community Futures Network of Canada.

The CFoT Oversight Committee was established in 2011 to oversee the implementation of the CFoT funding model by providing a governance function to ensure that the model’s objectives are being managed collaboratively among the CBDCs and ACOA. The committee, which meets quarterly, is composed of senior ACOA officials and representatives of the AACBDC and of the ACCBIF.

2.4 Expenditure Profile

2.4.1 CBBD Expenditures

Over the period of the study, the CBBD sub-program incurred over $120.7 million in expenditures (Table 2) with grant and contribution (G&C) expenditures representing the majority of program spending (89 per cent). Of the $107 million in G&C expenditures incurred over the period, $77.2 million (72 per cent) was through the CFP and $29.2 million (27 per cent) was through the BDP. Only $0.6 million (<1 per cent) was through the ICF. G&C expenditures increased in 2009-2010, largely due to the delivery of initiatives associated with the EAP.

Over the six-year period, CBBD programming represented nearly 16 per cent of overall CD program activity expenditures ($769 million) and 6 per cent of total Agency expenditures ($2,171 million).[See Footnote 13] Between 2010-2011 and 2011-2012, CBBD as a percentage of CD expenditures increased from 12.7 per cent to 19 per cent, largely due to a short term EAP Community Adjustment Fund for the lobster industry in Prince Edward Island.

The CBBD sub-program’s operation and maintenance expenditures represented a small portion of total costs, at $13.7 million (11.3 per cent). Internally, ACOA has a small number of staff that work on CBBD related activities. This accounts for the salaries and operating expenditures identified. Operational expenditures relate mainly to transportation, communication, professional and special services.

Table 2: Total Expenditures for CBBD ($000s), 2007-2008 to 2012-2013

Fiscal Year	G&C	Operations and Maintenance ( O&M)			Total
Fiscal Year	G&C	Salary	Operations	Total O&M	Total
2007-2008	18,894	1,465	415	1,880	20,774
2008-2009	16,607	1,626	669	2,295	18,902
2009-2010	22,188	1,930	282	2,212	24,400
2010-2011	17,858	2,349	313	2,663	20,521
2011-2012	17,121	2,200	288	2,489	19,610
2012-2013	14,396	1,946	215	2,161	16,557
Total	107,064	11,516	2,183	13,699	120,763
Source: GX Data June 2013 with CB Region expenditure data provided by ECBC. [See Footnote 14]

2.4.2 Project Profile

According to ACOA’s project database (QAccess), 223 CBBD projects were initiated during the period from 2007-2008 to 2012-2013, representing a total of $93.5 million in ACOA-approved funding (Table 3).[See Footnote 15] The fluctuations in assistance, particularly between 2010-2011 and 2011-2012, are the result of timing issues as multi-year agreements were entered into in 2010-2011, with the full amount of assistance recognized that year. Timing differences related to the recognition of expenditures between QAccess and ACOA’s financial system also account for the variance between the “Total ACOA Assisted” in Table 3 and the total “G&C” reported in Table 2.

Table 3: CBBD Approved Projects by Transfer Payment ($000), 2007-2008 to 2012-2013

Fiscal years	Number of Projects	BDP	CFP	ICF	Total ACOA Assisted	Total Project Cost
2007-2008	71	2,645	10,307	0	12,952	64,679
2008-2009	51	2,235	12,906	50	15,191	36,221
2009-2010	21	6,795	10,017	0	16,812	14,464
2010-2011	53	5,385	22,458	275	28,118	53,951
2011-2012	14	2,178	3,139	0	5,317	8,152
2012-2013	13	859	14,043	228	15,130	5,211
Total	223	20,097	72,870	553	93,520	182,678
Source: QAccess project data, May 2013.

Projects funded under the CBBD are non-commercial and non-repayable in nature and fall within five main types:

Operational funding to CBDCs (70 per cent of funding);
Operational funding to CBDC associations (23 per cent of funding);
Operational funding to the Ulnooweg Development Group (2 per cent of funding);
All other projects (5 per cent; includes operational funding to five not-for-profit organizations in urban areas).

According to QAccess project data (Table 4), the CFP represented the largest category of funding, accounting for approximately 75 per cent of projects and 78 per cent of ACOA assistance. Core operational support represents the largest activity[See Footnote 16] type, regardless of transfer payment program, representing 70 per cent of ACOA assistance. Provision of capital represented the second largest category of projects, at 22 per cent.

Table 4: CBBD Projects by Program and Project Type, 2007-2013

Program and Project Type	Number of Projects	Total ACOA Assistance ($000s)	Total Project Cost ($000s)
Community Futures Program sub-totals	168	72,870	119,781
Core Operational Support	128	63,382	107,600
Counselling/Mentoring	6	425	482
Infrastructure and Equipment	2	1,501	1,573
Marketing	6	1,145	1,614
Planning and Studies	3	71	94
Promotion and Awareness	1	30	95
Provision of Capital	10	3,969	5,266
Training	11	2,149	2,858
Support for Special Target Groups (not youth)	1	199	199
Business Development Program sub-totals	51	20,097	62,033
Core Operational Support	9	1,925	5,871
Planning and Studies	2	656	1,419
Provision of Capital [See Footnote 17]	37	16,436	53,558
Support for Special Target Groups (not youth)	3	1,080	1,185
Innovative Communities Fund sub-totals	4	553	864
Counselling/Mentoring	3	521	796
Training	1	32	68
Grand Total	223	93,520	182,678
Source: QAccess project data May 2013. ACOA Assistance includes amended contract amounts.

3. Findings: Relevance

Overall, this evaluation found that the CBBD sub-program remains relevant. The needs the program is expected to meet exist to at least the same extent as five years ago. CBBD activities are aligned with ACOA’s strategic outcome and the ACOA Act. They are consistent with the Agency’s roles and responsibilities and broader federal priorities related to economic development. The Agency’s CBBD programming is consistent with that being delivered by other federal departments/agencies across Canada in terms of roles, responsibilities and mandate. CBBD programming complements rather than duplicates programming being delivered by other organizations. Relevance of the CBBD sub-program was assessed by examining the continued need for the programming and the alignment between the programming, Government of Canada and Agency priorities, and federal roles and responsibilities.

3.1 Continued Need for the Programming

Judgment Criteria	Key Finding
The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.	Needs are either the same or greater than they were during the last evaluation in terms of the following: economic recovery was a key issue during the period of this evaluation; this was not the case during the 2009 CFP evaluation period; Atlantic Canada continues to experience lower participation rates, education levels, employment and wages than the rest of Canada; and factors related to out-migration and the aging population continue to challenge both urban and rural communities.
The role that CBBD funding recipients play in communities is consistent with their mandate as well as with community needs and expectations. CBBD programming responded to emerging needs.	The role that CBBD funding recipients play in communities is consistent with their mandate and with community needs and expectations. CBDCs’ volunteer boards of directors ensure that their communities’ needs are met. All CBDCs provide non-financial services that support capacity for the creation and expansion of SMEs. Emerging needs are being addressed with the implementation of the CFoT, succession planning initiatives and, in certain regions, the use of consulting advisory services[See Footnote 18] to address counselling needs.

Judgment Criteria

Key Finding

The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.

Needs are either the same or greater than they were during the last evaluation in terms of the following:

economic recovery was a key issue during the period of this evaluation; this was not the case during the 2009 CFP evaluation period;

Atlantic Canada continues to experience lower participation rates, education levels, employment and wages than the rest of Canada; and

factors related to out-migration and the aging population continue to challenge both urban and rural communities.

The role that CBBD funding recipients play in communities is consistent with their mandate as well as with community needs and expectations. CBBD programming responded to emerging needs.

The role that CBBD funding recipients play in communities is consistent with their mandate and with community needs and expectations. CBDCs’ volunteer boards of directors ensure that their communities’ needs are met. All CBDCs provide non-financial services that support capacity for the creation and expansion of SMEs.

Emerging needs are being addressed with the implementation of the CFoT, succession planning initiatives and, in certain regions, the use of consulting advisory services[See Footnote 18] to address counselling needs.

CBBD programming needs exist to at least the same extent or are greater than that reported in the previous evaluation. A literature review, key informant interviews and survey results indicate that SMEs are faced with challenges such as access to funding, population changes, economic recovery, the decline of traditional industries and unemployment.

Access to capital is an ongoing challenge for Atlantic Canadian SMEs, particularly in rural areas. In a 2011 study by the Business Development Bank of Canada, 32 per cent of SMEs cited problems accessing financing as the second highest challenge impacting their ability to expand (number one being the economic downturn).

Ulnooweg provides access to capital and business support services to all Aboriginal entrepreneurs in Atlantic Canada. Its aim is to help Aboriginal entrepreneurs manage their businesses whether as a start-up or an expansion. Ulnooweg supports partnerships to strengthen business; its commitment to the communities it services is helping Aboriginal entrepreneurs prosper, grow and develop their businesses.

The CBBD sub-program addresses this issue by supporting CBDCs and other community-based economic development organizations, which in turn provide access to loans and other services. CBDCs are considered less risk averse than traditional banking institutions. They provide loans to SMEs that may not otherwise obtain funding. Evaluation results indicate that 70 per cent of CBDC clients who responded to the survey were refused funding from other sources prior to accessing CBDC loans.

According to loan statistics from the CBDCs, a total of 13,108 applications were received (an average of 2,185 annually), and 10,089 applications were approved (an average of 1,682 annually) over the evaluation period, for an overall approval rate of 77 per cent. This resulted in over $353 million in financial assistance provided to close to 9,500 businesses.

Key informants consistently reported that CBBD programming helps to address gaps in accessing capital, an issue that continues to impact SMEs, particularly in rural areas. As banks continue to move out of rural areas, CBDCs and organizations such as Ulnooweg Development Group are often the only remaining local organizations that provide support to SMEs. The key informant interviews and the survey results support the ongoing need for both financial and non-financial services such as training, counselling, succession planning and skills development. CBDCs identified a high level of need (> 97 per cent) for all services being offered (e.g. financing, business information, referral services, business counselling, business skills training and business plan development); CBDC clients indicated their greatest needs were related to business financing (92 per cent), business skills training (92 per cent), assistance with developing a business plan (91 per cent), and business information (91 per cent).

According to the client survey, satisfaction with all CBDC services is relatively high (ranging from 75 per cent for referral services to 95 per cent for business loans). This is similar to what was reported in the previous evaluation.

The economy of Atlantic Canada continues to lag behind other parts of the country, and key informants note that recovery from the economic downturn remains an important factor. In Atlantic Canada, the real gross domestic product (GDP) per capita, the accepted measure of an economy’s standard of living, is below that of Canada as a whole. The region represents 7.0 per cent of Canada’s population but produces only 5.8 per cent of its GDP. The gap between Atlantic Canada and Canada has remained relatively stable during the past five years. Real GDP per capita in Atlantic Canada is now 85 per cent of the national average, down from 87 per cent in 2007.

Closing the labour productivity gap is fundamental to Atlantic Canada’s economic prosperity. Some of the factors contributing to this gap include population changes and high unemployment in rural areas.

Demographic changes such as declining population, mobility, aging of the population and immigration have a direct bearing on labour force development, availability, and activity. For rural communities, the loss of population and labour to larger centres constricts the capacity for growth, development, expansion and innovation.

Out-migration is a leading cause of population loss in Atlantic Canada as people leave the region for better employment opportunities in other provinces, especially in Alberta. As noted in Figure 3, the level of out-migration was highest in 2006, decreasing in 2009 and 2010 due to the economic recession as many Atlantic Canadians returned home. Out-migration levels increased once again in 2011 and 2012 as the economy began to recover from the 2008-2010 recession. Key informants reported that out-migration from rural communities to urban centres within Atlantic Canada is also affecting rural economic development.

Figure 3: Net Interprovincial Migration for Atlantic Canada, 2000-2012

Source: ACOA Policy Unit, based on Statistics Canada data, February 2013.

(D)

Atlantic Canada continues to experience lower participation rates, lower education levels, lower employment and lower wages than the rest of Canada. During the evaluation period, the labour market participation rate[See Footnote 19] for Atlantic Canada as a whole (average 63.8 per cent) consistently fell 2 to 3 percentage points below the Canadian average (67.1 per cent) annually, with three of the four provinces having lower-than-average rates. Prince Edward Island is the only province that consistently had participation rates above both the Atlantic and Canadian rates. During the same period:

Atlantic Canada consistently reported higher unemployment rates than Canada overall, with an average rate of 10.6 per cent versus 7.2 per cent nationally between 2007-2008 and 2012-2013.
The average weekly earnings of Atlantic Canadians continued to lag behind the Canadian average.[See Footnote 20] In 2012, the average earnings of Atlantic Canadians were 8.9 per cent lower than the Canadian average ($817.16 for the Atlantic versus $896.81 for the nation), a slight improvement over 11.4 per cent in 2008 ($717.94 for the Atlantic versus $810.47 for Canada).
Atlantic Canada, on a proportional basis, reports fewer university graduates and a higher percentage of people without a high school diploma than most other provinces.

Importance of the CFP: The June 2008 Senate Report, Beyond Freefall: Halting Rural Poverty, clearly indicates a continued need for the CFP by stating: “The challenges of accessing credit in rural Canada have long been recognized at the federal level. Arguably the most successful program to help address this concern has been the Community Futures Program.” Furthermore, the committee recommended that “the federal government reaffirm its long-term commitment to the Community Futures Program.” Source: Senate of Canada, 2008.

3.2 Alignment with Government Priorities

Judgment Criteria	Key Finding
There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome. The alignment is recognized and made explicit.	The alignment of CBBD programming to the Agency’s mandate is evident through its position in the program alignment architecture (PAA). The programming is well-aligned with Agency priorities to enhance the growth of earned incomes and employment opportunities in Atlantic Canada. CBBD activities are aligned with the federal government priorities related to economic growth.

The alignment between ACOA’s CBBD sub-program and the Agency’s strategic outcome is reflected in the PAA (Figure 4). The CBBD is one of two sub-programs supporting the Community Development (CD) program.[See Footnote 21]

The PAA is based on the results of policy research and analysis, the periodic assessment of program relevance and performance, ongoing dialogue with stakeholders in the region, and the priorities and directions of the Government of Canada.

Figure 4: ACOA Program Alignment Architecture 2014-2015

Strategic Outcome	Program	Sub-Program
A competitive Atlantic Canadian economy	1.1 Enterprise Development	1.1.1 Innovation and Commercialization
		1.1.2.Productivity and Growth
		1.1.3 International Business Development
	1.2 Community Development	1.2.1 Community Investment
	1.2 Community Development	1.2.2 Community-Based Business Development
	1.3 Policy, Advocacy and Coordination	1.3.1 Policy
		1.3.2 Advocacy
		1.3.3 Coordination
	1.4 Internal Services

Source: ACOA 2014-2015 Report on Plans and Priorities.

ACOA recognizes the importance of the region’s many geographic, linguistic and cultural communities and, through the CBBD and other CD programming, supports efforts to develop the resources they need to assume full responsibility for their own economic development.

CBBD programming supports “strong economic growth” – one of 16 high-level expected outcomes identified in the Government of Canada’s whole-of-government framework. This outcome is focused on increasing economic growth and development in all regions and all sectors of the economy.

In the 2010 Speech from the Throne, the Government of Canada committed to creating conditions for continued success in industries that are the foundation of Canada’s prosperity and that support thousands of communities, both rural and urban. This commitment was further recognized in Budget 2010, when the Government of Canada approved the renewal of the CFP’s terms and conditions and confirmed the Government of Canada’s commitment to bolster the program’s permanent budget with funding of $11 million per year ongoing, resulting in total program funding of $82.6 million annually. This demonstrated the government’s commitment to help communities diversify their economies and develop sustainable economic opportunities, and to help rural SMEs grow and become more competitive by, among other things, increasing their access to capital.

CBBD programming contributes to the government’s Economic Action Plan by helping to create jobs in rural communities and by supporting SMEs. It supports Advantage Canada by helping Canadian businesses compete (entrepreneurial advantage) and by supporting skills development and the innovation vital to productivity and competitiveness (knowledge advantage).

Need for the CBBD: The CBBD and, more specifically, the CFP is a cornerstone of the Government of Canada’s support for rural economic development. It plays an important role in strengthening the ability of rural communities to diversify their economic base to foster long-term prosperity and sustainability. It provides key support to the development of rural businesses. Source: CFP performance management strategy program profile, January 2013.

ACOA’s CBBD programming contributes to the Federal Framework for Aboriginal Economic Development by supporting the Ulnooweg Development Group in focusing on opportunities, responding to new and changing conditions, leveraging partnerships and fostering results for Aboriginal Canadians.

3.3 Alignment with Federal Roles and Responsibilities

Judgment Criteria	Key Finding
ACOA is mandated by law to fulfill this role. Other jurisdictions administer such programming through the federal government.	ACOA’s roles and responsibilities related to the CBBD are aligned with the ACOA Act. Like ACOA, other federal agencies and departments have a mandate to administer CFP programming across Canada.
Needs are not met by alternative services (extent of duplication, overlap or complementarity).	CBBD programming complements rather than duplicates other economic development funding targeted at SMEs, particularly those in rural areas.

The ACOA Act gives the Agency the authority to “plan, direct, manage and implement programs and projects intended to contribute directly or indirectly to the economic prosperity of the Atlantic region.” By investing in initiatives that support SMEs in rural Atlantic Canada, CBBD programming contributes to ACOA’s mandate to “increase opportunities for economic development in Atlantic Canada and, more particularly, enhance the growth of earned incomes and employment opportunities in that region.”

CBBD activities appear to complement other economic development programming targeted at SMEs in rural areas. As outlined in Table 5, CBDCs are unique in their attention to economic development outcomes for programming, their Atlantic-wide scope, a lack of demographic restrictions on programming, and approval of higher risk loans relative to traditional commercial lenders. Initiatives like the Risk Mitigation Fund (RMF), allow CBDCs to be less risk averse than other organizations by providing support to SMEs that may not typically be eligible for other financial loans or that might otherwise be ineligible for direct support from ACOA.

ACOA’s unique focus on Atlantic Canada allows the Agency to respond to specific regional needs, opportunities and contextual factors while taking advantage of synergies between the provinces. This is consistent with what was reported in the 2009 Evaluation of the CFP.

Table 5: Summary of Organizations Delivering Programming Similar to CBDCs

Organization	Atlantic-wide programming	Primary mandate support of SMEs	Eligible sector flexibility	volunteer boards of directors	Small loan focus	Higher risk loans	Aboriginal focus
CBDCs	X	X	X	X	X	X
ACOA*	X	X				X
Ulnooweg	X	X	X	X	X	X	X
Business Development Bank of Canada	X		X			X
Canada Youth Business Foundation	X	X	X		X
Banks	X		X
Credit unions / Caisses populaires	X			X	X
Provincial governments	varies by province					X
* CBDCs deliver programming similar to what ACOA does through its Enterprise Development Program but in rural areas and with fewer restrictions on eligibility requirements.

The 2009 Financing Continuum Evaluation supports the finding that CBBD funding complements rather than duplicates other programming. It stated that ACOA is not the only public-sector provider of financing for SME expansion and modernization projects in Atlantic Canada. Funding is provided by the Business Development Bank of Canada, provincial ministries and agencies, and CBDCs (funded through the Agency’s CD program) as well as credit unions/caisses populaires and private-sector banks.

The niche that CBBD programming fills compared to other providers is evident in the CBDC client survey results. In the key informant survey, clients were asked whether they had been refused funding from other sources prior to approaching a CBDC or whether funding from another source was conditional upon receipt of CBDC funding:

The survey found that 68 per cent of clients who received CBDC loans to start a new business and who attempted to obtain funding from other sources were refused funding from other sources prior to approaching a CBDC.
The survey also found that 73 per cent of clients who received a CBDC loan to expand an existing business and who had attempted to obtain funding from other sources were refused funding prior to approaching a CBDC.

4. Findings: Performance – Effectiveness

Overall, the evaluation found that CBBD programming is incremental to the implementation of projects and the achievement of outcomes. CBBD programming is achieving immediate expected outcomes by supporting organizations such as the CBDCs and Ulnooweg that provide SMEs with access to capital and other supports to start, modernize and expand businesses. The programming provides the CBDCs with the skills and knowledge to operate efficiently, which translates into business services (e.g. information, counselling, training and referrals) for their SME clients. The programming contributes to the growth and maintenance of jobs and the growth and stabilization of SMEs in Atlantic Canada.

Factors impeding the success of CBBD programming are known and it is apparent that many are being mitigated. Some factors would benefit from further investigation, particularly the potential for unintended behaviours resulting from the Community Futures of Tomorrow (CFoT) model. Progress has been made in ensuring availability and use of performance measurement information, allowing for better and timelier programming decisions and improved communication of results. However, there is a need to address specific gaps that exist in performance data.

The evaluation team assessed the effectiveness of the CBBD sub-program by examining: (1) incrementality; (2) evidence of achievement of expected outcomes; (3) unexpected outcomes; (4) barriers and facilitators to achieving results; and (5) evidence that performance information is adequate and effective.

4.1 Incrementality

Judgment Criteria	Key Findings
The impact of the absence of the programming is similar to or greater than that observed in the previous evaluation.	CBBD programming is incremental to Ulnooweg, the activities of the CBDCs and the associations that support them. Without funding from CBDCs in rural Atlantic Canada, many SMEs would not exist.

ACOA’s CBBD programming is incremental to the Ulnooweg Development Group, to the activities of the CBDCs and to the associations that support them. According to financial data, funding from ACOA supports over 60 per cent of operational expenditures incurred by CBDCs in Atlantic Canada. Key informants state that operational assistance from ACOA allows CBDCs and Ulnooweg to provide the financing and range of business services that they currently offer to rural SMEs.

Similar to the findings from the 2009 CFP evaluation, program funding positively impacts the creation and expansion/modernization of SMEs in rural Atlantic Canada. According to both client survey data and key informant interviews, many SME creation and expansion projects would not have proceeded without the Agency’s support to the CBDCs and other community-based loan organizations.

To gauge the degree to which results being reported are attributable to ACOA’s investments, CBDC clients were asked what the likelihood would have been of them starting, maintaining or expanding their business if they had not received the loan. Approximately two thirds (67 per cent) of respondents indicated this would have been “somewhat or very unlikely” to occur, suggesting that without CBDC support, the initiatives would not have proceeded.

Clients were also asked what the impact on their business would have been without CBDC loans. The respondents indicated there would have been a major negative impact on when they would have started the business (46 per cent), on the size of the business (44 per cent) and on the quality of operations (37 per cent) they planned. Clients receiving loans for existing businesses indicated there would have been a major negative impact on when they would have expanded or modernized their business (50 per cent), on the size of the operation (44 per cent) and the existing quality of operations (43 per cent).

Another indicator of incrementality is the degree to which ACOA funding influences other partners to support CBDC clients. Both start-up (33 per cent) and existing (29 per cent) SME clients indicated that other funding was secured as a result of receiving CBDC loans. This represents an increase over results presented in the 2009 CFP evaluation, where 22.5 per cent of start-ups and 21.4 per cent of existing SMEs stated they were able to leverage other funds as a result of their CBDC loans.

The incrementality of the Agency’s CBBD programming is further supported in the Statistics Canada data analysis presented in Table 7 related to CFP-assisted versus comparable firms.

4.2 Achievement of Expected Results

There was sufficient evidence to confirm that CBBD programming is achieving immediate expected outcomes. In general terms, the programming is expected to assist Atlantic Canadian businesses in obtaining access to capital to start up, expand and/or modernize. It is also intended to ensure that CBDCs have the tools and knowledge required to effectively and efficiently provide non-financial business services to SMEs.

As discussed in section 5, the network of CBDCs is considered an effective mechanism in the achievement of outcomes because of their local expertise, volunteer boards and timely decision making as well as their collaborative efforts within the communities and regions they serve. In addition, the provincial and Atlantic associations of CBDCs contribute to the achievement of outcomes through their supports to the CBDCs. While it is too early to attribute impacts to the implementation of the CFoT, all indications point to the new delivery model as contributing to increases in both the effectiveness and efficiency of CBBD programming. The move to a targeted suite of loan products (e.g. general business, youth, first-time entrepreneur, innovation) and implementation of specific initiatives such as the RMF and the Entrepreneurial Training Fund, are all targeted at enhancing the effectiveness of programming while meeting the needs of SMEs and entrepreneurs.

4.2.1 Immediate and Intermediate Outcomes

Judgment Criteria	Key Findings
Atlantic Canadian Businesses have access to capital/improved access to capital to start, modernize and/or expand their businesses. CBDCs have access to the tools and knowledge needed to operate efficiently/Atlantic Canadian businesses have access to business information and counselling. The performance of CBDC-supported SMEs is at least as good as that of comparable SMEs in terms of business survival.	Through CBBD programming, Atlantic Canadian SMEs are provided with access to capital to start, modernize and/or expand their businesses. CBDCs provided $353 million in assistance to 4,332 new start-ups and 5,162 existing businesses over the evaluation period. It is estimated that, on average, CBDCs provide over 7,900 non-financial services such as business information and counselling annually. According to Statistics Canada data, the survival rate of CFP-assisted firms was higher than that for comparable firms by 20 percentage points after the crucial fifth year following start-up.

Judgment Criteria

Key Findings

Atlantic Canadian Businesses have access to capital/improved access to capital to start, modernize and/or expand their businesses.

CBDCs have access to the tools and knowledge needed to operate efficiently/Atlantic Canadian businesses have access to business information and counselling.

The performance of CBDC-supported SMEs is at least as good as that of comparable SMEs in terms of business survival.

Through CBBD programming, Atlantic Canadian SMEs are provided with access to capital to start, modernize and/or expand their businesses.

CBDCs provided $353 million in assistance to 4,332 new start-ups and 5,162 existing businesses over the evaluation period.

It is estimated that, on average, CBDCs provide over 7,900 non-financial services such as business information and counselling annually.

According to Statistics Canada data, the survival rate of CFP-assisted firms was higher than that for comparable firms by 20 percentage points after the crucial fifth year following start-up.

CBDC E-Report data, client and recipient survey results, and key informant interviews confirm that CBBD programming contributes to the achievement of expected outcomes, most notably increased access to capital to start, modernize or expand a business. The achievement of outcomes is consistent with the 2009 CFP evaluation, which confirmed that CBDCs support Atlantic Canadian SMEs through investments and counselling activities and that the effectiveness of the CBDCs is strengthened by the network of associations that promote collaboration and sharing of best practices among the CBDCs as well as partnerships with all levels of government.

Results reported by the CBDCs are significant. During the six-year evaluation period, over 10,000 loan applications were approved by CBDCs with 9,494 separate SMEs: 4,332 new start-ups and 5,162 existing businesses. CBDCs assisted SMEs with loans valued at $353 million over this period.[See Footnote 22] Table 6 shows the number and value of loans through both the Community Futures and Seed Capital.

Table 6: Summary of Community Futures and Seed Capital Loans by Fiscal Year

Number of Loans / Business Assisted	2007	2008	2009	2010	2011	2012	Total
New Start-Ups - CF	286	274	312	312	515	490	2,189
New Start-Ups - Seed	463	420	625	635			2,143
Total Start-Ups	749	694	937	947	515	490	4,332
Existing - CF	763	734	776	730	808	798	4,609
Existing - Seed	145	125	142	141			553
Total Existing	908	859	918	871	808	798	5,162
Total Number of Loans	1,657	1,553	1,855	1,818	1,323	1,288	9,494
Value of Loans ($M)
CF	47,903	49,567	52,384	52,283	57,465	60,339	319,941
Seed	7,244	6,578	9,577	9,789			33,188
Total Value of Loans	55,147	56,145	61,961	62,072	57,465	60,339	353,129

Source: CBDC E-Report data, 2007-2012.

In addition to business loans, CBDC E-Report data indicate that, on average, 7,904 non-financial services were provided annually by CBDCs. However, it was not clear from E-Report data what types of services were provided. Furthermore, key informants reported that the data were not consistently collected or reported and are therefore at best an estimate of the number and types of services provided.

Through the funding recipient survey, all CBDCs reported success in contributing to CBBD program outcomes by ensuring SMEs have improved access to capital to create or expand their businesses. All CBDC recipients also reported that their CBDC offers a range of non-financial services.

Those CBDC clients surveyed confirmed that their CBDC contributed to the provision of access to capital, to support in starting and/or maintaining a business as well as other outcomes. In particular, the 290 clients surveyed reported CBDC contribution (to some extent or to a great extent) to the following outcomes:

accessing capital (67 per cent);
starting a new business (64 per cent);
accessing information, training and counselling, and obtaining referrals (63 per cent);
keeping business open (61 per cent);
expanding or modernizing (57 per cent);
improving business skills (53 per cent);
improving the way they do business (50 per cent); and
increasing sales/revenues (50 per cent).

Clients reported lower contribution to the success of hiring staff (27 per cent), maintaining staff (28 per cent), and increasing business productivity (49 per cent).

As previously mentioned, 70 per cent of clients indicated that they were refused funding from other sources prior to accessing CBDC loans, and 67 per cent of clients reported that CBDCs helped them achieve success in terms of access to capital to some or great extent. Thirty-three per cent of new businesses and 29 per cent of existing clients reported that their CBDC loans acted as a conduit to leveraging other funding. As is also noted in Section 5.1.2, CBDCs leverage, on average, $0.95 per CBDC investment dollar to SMEs.

While all of the clients surveyed had received loan services from a CBDC, many also received non-financial services. When asked if they accessed non-financial business services, more than half (51 per cent) of clients reported that they obtained business information, followed by business counselling (46 per cent), business skills training (44 per cent), business plan development (37 per cent) and referrals (sometimes and often: 22 per cent). Performance data related to non-financial services is not consistently tracked and reported by the CBDCs, which prevented further analysis from being undertaken in this area.

4.2.2 Ultimate Outcomes

The 2009 CFP evaluation identified a need to improve the collection and management of program performance information to ensure that success and cost-effectiveness can be measured with greater accuracy. In response to this, and as part of the Community Futures (CF) performance measurement strategy (PMS), the regional development agencies (including FedNor) entered into a contract with Statistics Canada to obtain performance indicators such as measures of employment, business entries and exits (start-ups) and financial performance indicators based on a list of CFP-assisted firms provided to Statistics Canada by each regional development agency. Three reports have been completed to date covering performance over five year time periods ending in 2008, 2009 and 2010 (there is a two year time lag in terms of when Statistics Canada can access data). As shown in Table 7, over three rounds of data collection, Statistics Canada analyses comparing CFP-assisted firms to comparable firms show that CFP-assisted firms are achieving better outcomes than comparable firms, particularly with respect to employment, growth and business survival rates.[See Footnote 23]

Over the three rounds, CFP-assisted firms were much more successful at achieving job creation, higher wages and sales than the comparable group of firms. Employment growth in CFP-assisted firms, for example, averaged 9 per cent, 6.2 per cent and 6 per cent per year over each of the five-year periods, compared to 2.5, 2.1 and 2.3 per cent for the comparable group. Probably the more significant impact relates to the business survival rates, which are available from Round 2 and 3 only. In fact, the five-year business survival rate for CFP-assisted firms for Round 2 (76 per cent) and 3 (75 per cent) is notably higher than the rate for comparable firms for Round 2 (56 per cent) and 3 (53 per cent) after the crucial fifth year following start-up. This represents a variation in the five-year business survival rate of 20 percentage points or more between CFP-assisted firms and the comparable group from each round.

These results are promising, especially given the barriers and challenges noted throughout the study that continue to impact SMEs in Atlantic Canada. Overall, evaluation results on program effectiveness provide a strong indication of the contribution of CBBD programming to the achievement of the ultimate outcome of strengthened and expanded businesses.

Table 7: Statistics Canada – SME Outcomes of CFP-assisted Versus Comparable Firms

Indicators	Round 1 March 31, 2010 (2003-2008)	Round 2 March 31, 2011 (2004-2009)	Round 3 March 31, 2012 (2005-2010)
*CFP-Assisted Firms*
Employment Dynamics
Growth in Employment	9.0%	6.2%	6.0%
Growth in Payroll	13.2%	9.7%	10.0%
Business Survival Rates	N/A	76.0%	75.0%
Financial
Growth in Sales by Industry	15.5%	16.0%	15.0%
Growth in Sales by Worker	N/A	3.5%	3.7%
*Comparable Group of Firms*
Employment Dynamics
Growth in Employment	4.5%	2.1%	2.3%
Growth in Payroll	8.1%	5.8%	6.1%
Business Survival Rates	N/A	56.0%	53.0%
Financial
Growth in Sales by Industry	10.7%	6.5%	7.4%
Growth in Sales by Worker	N/A	3.4%	3.7%

Source: CFP Performance Reports 2010, 2011, 2012 (based on data tabulations obtained from Statistics Canada).

4.3 Unintended Outcomes of CBBD Programming

Judgment Criteria	Key Findings
Not Applicable	There were two main unintended outcomes of the CBBD programming including clients becoming bankable and SMEs choosing CBDCs over banks even when charged higher rates of interest.

Analysis of key informant interviews suggests that CBBD programming contributes to two main unintended outcomes which speak to the success of CBDCs.

The first unintended outcome relates to SMEs becoming bankable – as it means that banks view these SMEs as capable of returning a profit and want to do business with them. The survey results show that 31 per cent of clients considered that the CBDC loans they received acted as conduits to other sources of funding. Key informants confirmed that CBDC clients have become bankable and that this should be considered a success story for CBDCs as well as their clients, though some CBDCs are concerned that after nurturing clients, they lose those businesses to banks.

The second unintended outcome is that some SMEs prefer to do business with CBDCs over traditional banking institutions even when lower interest rates could be obtained elsewhere. In particular, key informants suggested that factors such as existing and/or long-term relationships with CBDC staff, CBDC knowledge of the area, and the non-financial services offered provide incentives for some SMEs to return to the community-based lending organizations.

4.4 Barriers to the Achievement of CBBD Outcomes

Judgment Criteria	Key Findings
Evidence that factors impeding success of the CBBD sub- program are known and mitigation strategies exist.	ACOA and the CBDCs are aware of internal and external factors impeding the success of CBBD programming and many are being mitigated. There are three broad areas of barriers: access to funding; awareness, capacity and skills; and contextual factors.

The evaluation identified three main categories of barriers impacting the achievement of CBBD outcomes: access to funding; awareness, capacity and skills; and contextual factors. While most of these may be addressed through the introduction of the CFoT funding model and other mitigation strategies, areas warranting further assessment and discussion relate to the range of interest rates charged by CBDCs and succession planning, an issue which is gaining more attention in recent years, particularly given the aging population in rural Atlantic Canada.

The intent of the CFoT funding model is to have a consistent funding approach across regions, to address the needs of target groups, and to ensure large unused cash balances are put to work. Mechanisms in place to support the CFoT include a suite of loan products targeting the needs of specific groups, the RMF, the Entrepreneurial Training Fund (ETF), and the CFoT Oversight Committee, which helps ensure consistent implementation of the model and mitigates risk. Key informants state that the RMF is an important factor in facilitating CBDCs taking on higher-risk loans as it introduced a shared-risk approach to some lending.

Recognizing the challenges facing the CBDCs, the Atlantic Association of CBDCs commissioned a study in 2013 to identify ways to increase CBDC lending activity. It reported that barriers to improving lending activity included the regional/global economies, depopulation, timing of the implementation of loan products associated with the CFoT funding model, and awareness of programming among potential clients. ACOA and the CBDCs took steps to address these issues, including increasing CBDC loan limits, improving the marketing of CBDC programming, and ongoing CFoT oversight and monitoring of performance.

Access to funding

The most common challenge to business success that surveyed clients report was access to capital, with 71 per cent indicating that it was a challenge to a “great” or to “some” extent. Specific issues related to access to funding include the following:

Interest rates – Just over half of surveyed clients reported that interest rates were a barrier. Key informants also identified high interest rates and the range of rates offered by some CBDCs as a concern. Forty-three per cent of surveyed CBDC recipients indicated that interest rates range from prime +3.2 per cent to prime +6.4 per cent. The remaining respondents did not provide a range, did not base their rates on prime or did not know the interest rates. Some key informants noted that interest rates could not be standardized because of the variation in projects and the levels of risk associated with them.
CBDC funding limit – Almost two thirds (63 per cent) of CBDC funding recipients reported that the maximum funding limit of $150,000 ($170,000 under special circumstances) was an issue impacting their ability to achieve programming objectives. This issue has recently been addressed as the Agency has implemented a change in the terms and conditions of the contribution agreements with ACOA that allows CBDCs to issue loans up to $225,000 under exceptional circumstances.
Risk aversion – Some key informants reported that CBDCs are becoming more risk averse – meaning that they take on less risky loans in order to prevent losses and to maintain a strong return on investment. Since CBBD funding is envisioned as providing access to capital for SMEs that could be viewed as higher risk by traditional lending agencies, risk aversion would reduce CBDC lending activity with target clients. Some key informants noted that the Atlantic Canada Community Business Investment Fund (ACCBIF) allows CBDCs to access additional capital at competitive lending rates and is therefore an important measure to ensure adequate access to capital. The RMF is one mechanism implemented through the CFoT aimed at encouraging CBDCs to take on more risk through its lending activities.
CBDC operational funding – CBDCs reported issues related to accessing operational support for their organizations, including not knowing funding levels at beginning of each fiscal year (61 per cent) and lack of operational funding (53 per cent).

Awareness, Capacity and Business Skills

In order for CBDCs and Ulnooweg to provide loans and other business services to entrepreneurs, they have to be aware of the programming. CBDC funding recipients (66 per cent) and key informants identified that awareness of the programming was a challenge. Furthermore, key informants indicated that creating awareness of the services provided by Ulnooweg continues to be a challenge in Aboriginal communities. The development and implementation of a CBDC communications plan is expected to improve awareness among entrepreneurs of funding and other services available through CBDCs.

Training and capacity of both clients and CBDC boards was also identified as barriers to program success. CBDCs provide business advice and counselling, which acts as a mitigation measure. Training support through the ETF and consulting advisory services addresses the need for capacity and skills development of SMEs. However, key informants reported that limitations of the ETF prevent entrepreneurs from accessing needed technical and other non-eligible training.

In 2012-2013, as part of its due diligence process for providing operational support to CBDCs, ACOA, in partnership with the CBDC network, developed and adopted minimum standards for board governance. The standards focused on accountability, transparency and confidentiality and are intended to further strengthen CBDCs credibility, legitimacy, branding, internal corporate responsibility and industry standards. The establishment of the minimum board standards and associated board training initiatives are seen as addressing board skills and capacity, which were identified by key informants as being uneven across CBDCs.

There are two issues related to capacity and business skills that do not appear to have mitigation measures in place. First, key informants reported that there is inconsistent delivery of non-financial services across CBDCs, with some CBDCs perceived as more proactive in offering these services than others. This is an important consideration as it is expected that SMEs that access such services may perform better than those that do not. Second, key informants reported that there is insufficient succession planning and issues related to attracting and maintaining skilled CBDC staff and board members, impacting the ability to ensure stable organizations into the future.

Contextual Factors

The evaluation also identified contextual factors that have impacted the achievement of project outcomes. A literature review indicated the global economic downturn of 2008-2010 negatively affected SMEs and lending activity. According to surveyed CBBD recipients, the most commonly cited barriers impacting the achievement of results included fluctuating government funding levels (77 per cent), the collapse of industry/downturns in the economy (72 per cent), lack of public awareness of the CFP (66 per cent), lending limits (63 per cent), not knowing funding levels at the start of the fiscal year (61 per cent), an aging population (58 per cent), out-migration (54 per cent), and the lack of operational funds (53 per cent). Key informants also identified succession planning as a barrier faced by SMEs when planning to sell their businesses.

4.5 Adequacy of Performance Measurement

Judgment Criteria	Key Findings
CBBD sub-program performance measurement is adequate and effective in reporting on the achievement of outcomes. Performance measurement information is considered useful and is being used by ACOA and ACOA-funded organizations Performance information is collected and is available for use during the evaluation.	The Evaluation found improvements in performance measurement data since the last evaluation. Data are collected and used to manage the program. These include the CFoT funding formula, information extracted from CBDC quarterly E-Reports and special purpose reports, Statistics Canada analyses as well as performance measurement results from indicators identified through the national Community Futures PMS. Some gaps in performance measurement continue to exist. These include incomplete CBDC client contact information, incomplete data on non-financial services, gaps in the collection of some performance indicators, and targets that have not yet been established.

Judgment Criteria

Key Findings

CBBD sub-program performance measurement is adequate and effective in reporting on the achievement of outcomes.

Performance measurement information is considered useful and is being used by ACOA and ACOA-funded organizations

Performance information is collected and is available for use during the evaluation.

The Evaluation found improvements in performance measurement data since the last evaluation.

Data are collected and used to manage the program. These include the CFoT funding formula, information extracted from CBDC quarterly E-Reports and special purpose reports, Statistics Canada analyses as well as performance measurement results from indicators identified through the national Community Futures PMS.

Some gaps in performance measurement continue to exist. These include incomplete CBDC client contact information, incomplete data on non-financial services, gaps in the collection of some performance indicators, and targets that have not yet been established.

Progress has been made in ensuring the availability and use of performance measurement information which allows for evidence-based decision-making and improved communication of results. However, gaps continue to exist in terms of client contact information, provision of non-financial services and Community Futures PMS data collection.

Following the 2009 CFP evaluation, ACOA, together with the Canada Economic Development for Quebec Regions, Western Economic Diversification Canada, the Federal Economic Development Agency for Southern Ontario and the Federal Economic Development Initiative for Northern Ontario under Industry Canada, led a process to revise the PMS for the national Community Futures Program. Highlights of this revised strategy, approved in August 2010, include the following:

new performance indicators for jobs with regular reporting requirements to the Agency;
standardization of financial statements, implemented in 2011-2012, to provide clarity in terms of loan loss activity;
identification and consistent definition of data elements and indicators to enable CBDCs to systematically define, gather and report on jobs and loan loss and to measure performance;
commitment to enter into a contract with Statistics Canada to obtain a series of data tabulations to assist with measuring the performance of the CFP;
establishment of a permanent inter-departmental committee responsible for oversight of the collection and management of performance information; and
a performance measurement framework that identifies the critical data requirements and performance measures to demonstrate achievement of CFP expected results.

The level of performance information provided by CBDCs to ACOA has improved significantly since the last evaluation. Sources of CBDC data include CFoT funding formula results, quarterly E-Reports, special purpose reports, financial statements, active loans and loan product reports, and the annual reports of the CBDCs. The special purpose report, an annual report issued by the CBDCs and verified by their auditors, is considered the most reliable in terms of reporting the number of loans. The standardization of the CBDC financial statements has improved ACOA's ability to systematically perform trend analysis and compare results across the CBDC network. Key informants highlighted the adoption of standardized financial statements by CBDCs as essential to ensuring the availability of standard indicators of performance, including loan management.

The majority of recipients surveyed reported that they were collecting and monitoring performance data (98 per cent) and using the information for decision making (85 per cent). CBBD performance measurement data is used by ACOA to aid program management. ACOA uses high-level dashboard data to monitor program activity such as the number of projects and funding amounts. Program management uses QAccess queries and CBDC E-Report data to monitor activities and/or outcomes and to inform decision making. CBBD financial data were available and used in the evaluation, particularly from ACOA’s GX and QAccess systems.

Opportunities for improvement include:

Gaps in CBDC client contact information: the original client contact data provided by the CBDCs (12,939 records in total) were missing 50 per cent of e-mail addresses and 6 per cent of telephone numbers. Gaps in the completeness of client contact information varied greatly by CBDC, ranging from 1 to 55 per cent for telephone numbers and 23 to 99 per cent for e-mail addresses. Mechanisms used to store client contact information ranged from TEA software[See Footnote 24] to Excel spreadsheets to hard copy files. The result is that CBDCs do not have a consistent electronic means to capture client contact information in an easily accessible format. This impacted the Agency’s ability to assess client performance through the use of tools such as client surveys. These gaps may be partly due to issues surrounding the TEA reporting system. Key informants noted that the TEA software, the tool used by CBDCs to capture client contact information and project related data, is complex and time consuming. This is consistent with the 2009 CFP evaluation, which stated that due to incomplete data, there was a risk that certain segments of the client population were not included in the survey.
Non-financial CBDC services: data related to non-financial services is not consistently collected and reported. While TEA data did present some information related to non-financial services (e.g. counselling activities), evaluators were advised that it was not consistently reported and was therefore, at best, an estimate of the number and types of services provided. This is consistent with the 2009 CFP evaluation, which acknowledged that data on non-loan services were not typically captured.
Community Futures PMS data collection by ACOA: Although significant progress has been made on the Community Futures PMS, some indicator data was not being collected at the time of the evaluation. For example, indicators such as dollar value of start-up loans and the number and dollar value of other loans are not being collected. In addition, targets have yet to be established for key indicators, which impacts the ability to draw conclusions on the appropriateness of results. For example, targets have not yet been established for loan loss rates.

When asked to identify the most important indicators for measuring success, CBDC recipient survey respondents indicated business survival rate (88 per cent), employment growth (87 per cent), business productivity (59 per cent), growth in sales (53 per cent) and percentage of businesses funded by type of industry (40 per cent).

5. Findings: Performance – Efficiency and Economy

Overall, the evaluation found that the CBBD programming demonstrates efficiency and economy in the utilization of resources.[See Footnote 25] The existing model of delivering the majority of CBBD programming through CBDCs and Ulnooweg allows the Agency to achieve an important part of its mandate in an efficient and economical fashion. CBDC associations add efficiencies through coordination and cost-saving measures, including the CFoT funding model, the ACCBIF, management information systems and group purchasing initiatives. The CFoT funding model appears to have improved governance and investment fund management tools, though it is too early to fully assess its impact. Since the previous evaluation, ACOA and CBDCs have adopted a continuous improvement approach to program management and monitoring. This has allowed them to measure and monitor performance and continue to explore delivery alternatives. CBDCs’ management of loan funds is now supported by more consistent measures.

The evaluation assessed efficiency and economy by examining (1) evidence of efficient utilisation of resources, (2) loan management practices, and (3) consideration of alternative delivery mechanisms, lessons learned and best practices.

5.1 Efficient Utilization of Resources

Judgment Criteria	Key Findings
ACOA has in place structures and mechanisms to ensure the most efficient and economical means are being used to administer the programming.	Mechanisms that support efficient and economical delivery of the CBBD programming include the delivery through CBDCs and Ulnooweg, enhanced planning, governance and delivery through the new CFoT funding model, and collaborative relationships among the CBDCs, the CBDC associations and ACOA.
Delivery costs compare favourably to the delivery costs calculated as part of previous evaluations and to the costs of similar programming offered in other jurisdictions.	Overall, delivery costs appear reasonable and stable. Over the period of this evaluation: It cost $0.13 in total CBBD operations and maintenance expenditures to deliver $1 in grants and contributions (G&Cs). Considering the total operational expenditures (ACOA and CBDC) and the total investments to SMEs, it cost $0.33 for every $1 in loans provided to SMEs. The total average operating cost per CBDC, including all sources of funding, was $422.9 thousand per year. The average CFP contribution to operating costs of CBDCs was $253.4 thousand. This is an increase over the previous evaluation, which reported an average of $220.3 thousand for 2003-2004 to 2007-2008. The average loan investment per job created or maintained was $13,577. Total leveraged funds were $338.6 million, with an average of $0.95 in leveraged funds per CBDC investment dollar.

Judgment Criteria

Key Findings

ACOA has in place structures and mechanisms to ensure the most efficient and economical means are being used to administer the programming.

Mechanisms that support efficient and economical delivery of the CBBD programming include the delivery through CBDCs and Ulnooweg, enhanced planning, governance and delivery through the new CFoT funding model, and collaborative relationships among the CBDCs, the CBDC associations and ACOA.

Delivery costs compare favourably to the delivery costs calculated as part of previous evaluations and to the costs of similar programming offered in other jurisdictions.

Overall, delivery costs appear reasonable and stable. Over the period of this evaluation:

It cost $0.13 in total CBBD operations and maintenance expenditures to deliver $1 in grants and contributions (G&Cs). Considering the total operational expenditures (ACOA and CBDC) and the total investments to SMEs, it cost $0.33 for every $1 in loans provided to SMEs.
The total average operating cost per CBDC, including all sources of funding, was $422.9 thousand per year. The average CFP contribution to operating costs of CBDCs was $253.4 thousand. This is an increase over the previous evaluation, which reported an average of $220.3 thousand for 2003-2004 to 2007-2008.
The average loan investment per job created or maintained was $13,577.
Total leveraged funds were $338.6 million, with an average of $0.95 in leveraged funds per CBDC investment dollar.

5.1.1 Structures and Mechanism that Support Efficiency and Economy

Multiple mechanisms currently exist to support efficient and economical program delivery. These can be described within the categories of planning and budgeting, governance and delivery.

Planning and Budgeting Mechanisms

Strong and coordinated planning and reporting mechanisms support the efficiency and economy of the CBBD programming. Key informants noted that the CFoT funding model includes a revised operational funding formula that allows for better planning and transferability of funds. The funding formula makes clear the annual amount of funding for each CBDC, allowing for better planning. The operational funding agreements also require that a minimum of 70 per cent of funds be in active loans. CBDCs falling short of this target must transfer investment funds into the ACCBIF to reach a minimum of 70 per cent of funds in active loans, meaning that those CBDCs with surplus funds invest in the fund and those with higher demand are able to borrow from it.

Another important change since the last evaluation that supports better planning is the standardization of performance data collection, reporting and tracking through the national Community Futures PMS committee.

Efficiency Mechanisms and Ulnooweg: As it does with the CBDCs, ACOA works with Ulnooweg to ensure mechanisms are in place to support the efficient delivery of programming to Aboriginal entrepreneurs.

Planning: Ulnooweg develops annual work plans, strategic plans and standardized financial statements to support planning and budgeting.

Governance: ACOA’s Head Office coordinates the administration of funding to Ulnooweg on behalf of the Agency. Ulnooweg is governed by a board of directors that includes all Atlantic Canada Chiefs, ensuring that the needs and context surrounding Aboriginal entrepreneurs is reflected in operations.

Delivery: Ulnooweg understands the unique assets and needs of Aboriginal entrepreneurs across Atlantic Canada. Some key informants expressed interest in exploring opportunities for better coordination and/or communication with CBDCs.

Governance Mechanisms

Clear governance promotes improved efficiency and economy through well-defined roles, decision making and procedures. According to the Institute on Governance, “governance determines who has power, who makes decisions, how other players make their voice heard and how account is rendered.”

Each of ACOA’s regional offices work with their respective CBDCs and the CBDC provincial association to administer and monitor operational funding projects. ACOA’s Head Office administers funding projects to the Atlantic Association of CBDCs as well as to Ulnooweg. The Head Office also plays important coordination roles in terms of performance measurement, governance and liaising with the AACBDC. ACOA’s Head Office works closely with the AACBDC to oversee delivery of the CFoT model, particularly the operational funding formula used to determine the operational funding requirements of the CBDCs in order to achieve objectives and expected results. New CBDC board governance minimum standards were introduced in 2012-2013. Fifty-nine per cent of survey recipients reported that their CBDC was already meeting or exceeding board governance standards. A further 31 per cent reported that their CBDC was in the process of implementing new standards and this process would be completed by March 31, 2014. Both surveyed recipients (79 per cent) and key informants reported that the new standards would strengthen CBDC boards. In fact, key informants indicated there were already positive results stemming from the new board governance standards in terms of greater focus on accountability, transparency and sharing of best practices among boards.

Key informants reported that CBDCs have strong governance structures, with over 400 volunteer board members. They also reported that the different associations provide specific coordination, cost-saving and standardization measures that aid efficient operation of the CBDCs. According to CBDC survey recipients, all three levels of CBDC associations contribute to the efficient operation of the CBDCs: provincial associations (88 per cent); the AACBDC (86 per cent); and the Community Futures Network of Canada (68 per cent).

CBDC and ACOA key informants reported that communication and coordination is achieved among CBDCs and associations through several pan-CBDC committees. The CFoT oversight committee was cited as a particularly important committee for coordination between CBDCs and ACOA. It includes representation from ACOA, the AACBDC and the ACCBIF board of directors. Given that the committee was initially conceived as a temporary structure, key informants highlight that the need for a longer term committee should be reviewed. Furthermore, some key informants stated that any long-standing committee should ensure representation from all ACOA regional offices and clarify roles and responsibilities in terms of decision making.

Delivery mechanisms

The delivery of loans through CBDCs is perhaps the greatest mechanism for efficiency and economy of programming. CBDCs are established, experienced community-based organizations located in rural communities throughout Atlantic Canada. They have business expertise and knowledge of economic development and their local nature means they understand their communities’ unique assets and needs.

Early indications point to the introduction of the CFoT funding model in 2011-2012 as contributing to greater efficiencies in delivery. The CFoT model was put in place to maximize the efficiency and effectiveness of the resources available to all CBDCs and to address undercapitalization issues, including the merging of the Seed Capital Investment fund with the general investment fund.

CBDC funding recipient survey data supports the early benefits of the CFoT. According to 73 per cent of surveyed recipients, the introduction of the CFoT enhanced effectiveness and efficiency. Recipients reported that the following specific components of the CFoT improved effectiveness and efficiency: Entrepreneurial Training Funding (84 per cent); Risk Mitigation Fund (83 per cent); having a suite of loan products (78 per cent); removal of silos – merging of Seed Capital and other investment funds (70 per cent); and recapitalization of the CBDCs (62 per cent).

ACOA has made internal administrative changes that also contribute to program efficiencies. Key informants reported that ACOA’s move to using multi-year operational contribution agreements instead of amending projects annually saves time and resources for both funding recipients and ACOA staff.

5.1.2 Delivery Costs

Delivery costs appear reasonable in comparison to the 2009 CFP evaluation and to the achievement of outcomes. The evaluation examined the cost-effectiveness of the CBBD sub-program based upon: ACOA operational expenditures in relation to G&C expenditures; ACOA and CBDC operational expenditures in relation to CBDC investments in SMEs; the operating dollars provided to the CBDCs; and costs in relationship to the expected outcome of jobs being created or maintained.

ACOA and CBDC operational costs and investments

As noted earlier, total ACOA expenditures for CBBD programming over the evaluation period amounted to $120.8 million, with $107.1 million in G&Cs, $11.5 million in salaries and $2.2 million in operational costs. On average, there was $0.13 in operations and maintenance expenditures to deliver $1 in G&Cs. However, since the delivery of CBBD programming is largely through CBDCs, it is important to consider the total of ACOA and CBDC operational costs in relation to the amount of loans provided to SMEs by the CBDCs.

As shown in Table 8, the total operating costs of CBDCs over the period, including CFP and other funding, was approximately $104 million. Considering that over $353.1 million in loans was provided to SMEs by CBDCs over that period, the total operational expenditures (ACOA and CBDC) was less than $0.33 for every $1 in loans provided to SMEs. In other words, more than $3 in loans were delivered per dollar of ACOA and CBDC operational expenses.[See Footnote 26]

The average CBDC operational budget, including ACOA operational funding and all other sources, was $0.423 million over the six-year period of the evaluation. Of this total budget, ACOA’s average CFP contribution per CBDC over this period was $0.253 million, with the 41 CBDCs receiving a total of over $61.3 million from the CFP alone, meaning that ACOA contributed more than 60 per cent of CBDC total operating budgets.[See Footnote 27] The average CFP contribution amount per CBDC has increased by 15 per cent since the last evaluation, which reported an average operating contribution of $0.220 million. According to key informants, the increase in the operational budget over time is related to normal inflation in costs, including wages, rent, utilities, technology evolution and travel.

Table 8: Operating Costs ($M) for CBDCs in Atlantic Canada

	2007-2008	2008-2009	2009-2010	2010-2011	2011-2012†	2012-2013†	Total
Total CBDC operating costs*	16.8	17.1	17.7	17.6	17.6	17.1	104.0
Average CBDC operating cost	0.410	0.418	0.432	0.431	0.430	0.417	Average0.423
CFP contribution to operational costs**	10.2	10.4	10.4	10.8	9.8†	9.7	61.3

* Total CBDC operating costs were taken from E-Report data.

** ACOA CFP contribution to CBDCs comes from QAccess data.

† Two CBDCs did not receive ACOA funding in 2011-2012 and 2012-2013; figures represent 39 CBDCs for these years while figures for other years represent 41 CBDCs.

ACOA provides funding to the four provincial CBDC associations and the AACBDC. According to QAccess data, of the $19.7 million in contributions approved for the associations over the six years (2007-2008 to 2012-2013), $14.2 million was from the CFP, $5.0 million was from Business Development Program and the remaining was from Innovative Communities Fund. This means that the average CFP investment in the associations was $2.3 million annually. ACOA’s investment in CBDC associations appears to have increased following the 2009 CFP evaluation, which reported an average of $1.9 million to associations per year. This increase is consistent with feedback from key informants who noted that the AACBDC has taken on greater roles and responsibilities since the previous evaluation related to: management of the ACCBIF; management of the Risk Mitigation Fund (RMF); administration of the Entrepreneurship Training Fund; coordination of the Community Futures Network of Canada; strengthening corporate governance; developing strategies related to communications, marketing and social media; and the management information systems (MIS) project.

CBDC investment per job maintained or created

With total CBDC investments to SMEs at $353.1 million, the investment was $13,489 per job created or maintained (Table 9). The evaluation examined the CFP investments per job separately in order to make a comparison to the 2009 CFP evaluation results. The amount of CFP loans per job created or maintained from 2007-2008 to 2010-2011 was $14,888 compared to $14,253 during the previous evaluation period, suggesting that amount of loans per job has remained relatively stable.[See Footnote 28] When considering the value of the investment, it is important to also keep in mind that the majority of loans are repaid to the CBDC by the client, making the true investment per job created much smaller.

Table 9: CBDC Investment per Job Created/Maintained

	2007-2008	2008-2009	2009-2010	2010-2011	2011-2012	2012-2013	Total
Total CBDC investment* ($M)	55.1	56.1	62.0	62.1	57.5	60.3	353.1
Number of jobs created	1,841	1,524	1,950	1,994	1,263	1,579	10,151
Number of jobs maintained	2,278	2,320	3,012	2,877	2,586	2,956	16,029
Investment per job created or maintained ($)	13,389	14,607	12,486	12,743	14,930	13,305	13,489

Source: CBDC E-Report data

* Total CBDC investments means the amount of loans to SMEs. They include both Community Futures and Seed Capital investments from 2007-2008 to 2010-2011; in 2011-2012 both funds were merged into one general investment fund.

Leveraging

CBDCs reported extensive leveraging of investment funds provided to clients. Leveraging demonstrates that CBDC investments resulted in SMEs accessing investments from other sources. Over the six years, CBDCs reported investments of $353.1 million to SMEs and SMEs had a further $338.6 million in leveraged funds beyond CBDC investments. On average, there was $0.95 levered per CBDC dollar invested. Leveraging appears to have increased since the previous evaluation of the CFP, which showed $0.84 per dollar invested.

5.2 Management of Loan Funds

Judgment Criteria	Key Findings
Management of CBDC loan funds compares favourably to set targets, to the previous evaluation, and to practices in other similar organizations.	Overall, program data, key informants and survey recipients suggest that loan funds are managed appropriately. Over the evaluation period: The number of loan applications approved decreased and the average loan value increased; this is in keeping with the findings of the 2009 CFP evaluation, suggesting a trend over the past decade of CBDCs providing fewer loans but of higher value. Loan loss rates were relatively similar over the period of the study until implementation of the CFoT in 2011-2012 when rates increased following portfolio clean-up. The loan loss average was 2.85 per cent compared to 2.28 per cent reported in the 2009 CFP evaluation. There was a slight increase in the overall percentage of CBDC investment funds in active loans. While there is variability across the CBDCs, at least 70 per cent of funds are out in the community as active loans or are invested in the ACCBIF. Variability in loan management is being monitored through the CFoT Oversight Committee.

Judgment Criteria

Key Findings

Management of CBDC loan funds compares favourably to set targets, to the previous evaluation, and to practices in other similar organizations.

Overall, program data, key informants and survey recipients suggest that loan funds are managed appropriately. Over the evaluation period:

The number of loan applications approved decreased and the average loan value increased; this is in keeping with the findings of the 2009 CFP evaluation, suggesting a trend over the past decade of CBDCs providing fewer loans but of higher value.
Loan loss rates were relatively similar over the period of the study until implementation of the CFoT in 2011-2012 when rates increased following portfolio clean-up. The loan loss average was 2.85 per cent compared to 2.28 per cent reported in the 2009 CFP evaluation.
There was a slight increase in the overall percentage of CBDC investment funds in active loans. While there is variability across the CBDCs, at least 70 per cent of funds are out in the community as active loans or are invested in the ACCBIF. Variability in loan management is being monitored through the CFoT Oversight Committee.

Evidence from the recipient survey, key informant interviews and performance data suggests that investment funds are well-managed.[See Footnote 29] Only 8 per cent of recipients reported that their CBDC had concerns with the management of their own fund. Fifteen per cent of recipients reported that they had concerns with the management of the ACCBIF, with the most frequent issue related to having to contribute to the ACCBIF as part of the new CFoT. Consistent with the survey results, the majority of key informants had no concerns with CBDC loan management. However, several key informants reported that some CBDCs are risk-averse and do not do enough lending, which is further discussed in section 5.2.2.

5.2.1 CBDC Loan Approval Rate and Average Loan

According to CBDC performance data, the combined CFP and Seed Capital loan approvals over the period decreased from 1,739 in 2007-2008 to 1,425 in 2012-2013. Due to the investment funds being merged in 2011-2012, a decrease in the number of loans was expected over the period. However, analysis of loans approved in the final two years of the study show that the decline in the number of loans approved continued after the merging of funds. The approval rate for loans over the period remained stable, with an average of 77 per cent and a range of only 76 per cent to 78 per cent. The average combined loan value over the same period increased from $31,712 in 2007-2008 to $42,343 in 2012-2013.[See Footnote 30]

These findings are similar to those reported for the 2009 CFP evaluation, which found that the number of loans decreased while loan values increased over the five-year period. Key informants stated that the decrease in loan activity was most likely related to slow recovery from the economic downturn and trends in out-migration.

5.2.2 CBDC Loan Loss Rates

Overall, CBDC loan loss rates appear to be reasonable based on a comparison with the results of the 2009 CFP evaluation.[See Footnote 31] The loans provided to SMEs by CBDCs are fully repayable, so examining the extent that outstanding loans are unrecoverable provides an important indicator of CBDC loan management practices. It is also important to consider that CBDCs are encouraged to take more risk than traditional lenders.

The national CF performance measurement strategy (PMS) identifies loan loss rates based on provisional expenses as an indicator of loan management.[See Footnote 32] However, CBDCs in Atlantic Canada only began reporting provisional expenses in 2011-2012 when they implemented standardized annual financial statements. In 2011-2012, the loan loss rate based on loan loss provisions was 2.73 per cent; in 2012-2013, it was 4.07 per cent.

In order to better identify trends over time and to compare loan loss data to the previous evaluation, which used a different formula, this evaluation also calculated the loan loss rates based on write-offs for the full six years of the study.[See Footnote 33] The provisional loan loss rate, based on write-offs, was similar, fluctuating between 2.34 per cent and 3.78 per cent over the evaluation period with increases in 2011-2012 (3.78 per cent) and 2012-2013 (3.17 per cent).[See Footnote 34] Key informants explained that the increase during these years was the result of a portfolio clean-up that the CBDCs undertook after the implementation of the CFoT and the introduction of standardized financial statements.

The average loan loss rate appears to have increased overall from the previous evaluation, which reported an average 2.28 per cent, to this evaluation, which shows an average 2.85 per cent. However, when examining CFP investments only, the average has decreased, with an average loan loss rate for 2007-2008 to 2010-2011 of 2.19 per cent.[See Footnote 35]

There is variability in loan loss rates among CBDCs in Atlantic Canada. Table 10 shows the number and percentage of CBDCs by range of loan loss rate based on write-offs and including Seed Capital investments. Over 75 per cent of CBDCs had average loan loss rates of less than 4 per cent, and all had average loan loss rates of less than 8 per cent. Loan loss rates appear to have increased since the 2009 CFP evaluation, which reported that more than 75 per cent of CBDCs had average loan loss rates of less than 3 per cent.

Table 10: Loan Loss Rates of CBDCs

Loan loss rate	# CBDCs in Range	Cumulative %
˂1%	7	17.1%
≥1% to ˂2%	13	48.8%
≥2% to ˂3%	7	65.9%
≥3% to ˂4%	4	75.6%
≥4% to ˂5%	5	87.8%
≥5% to ˂6%	0	87.8%
≥6% to ˂7%	4	97.6%
≥7% to ˂8%	1	100.0%
Total	41

Source: CBDC E-Report data.

Departments and agencies delivering the CFP have not established target loan loss rates. Though some key informants reported that appropriate loss rates vary from one location to another depending on differing community economic realities, the lack of a target range of acceptable loan loss rates based on a thorough understanding of important contextual factors facing CBDCs and SMEs in Atlantic Canada is a barrier to identifying trends and loan management issues for mitigation.

The adoption of standard financial statements across CBDCs in Atlantic Canada is a positive step in better monitoring and managing loan portfolios. It enables CBDCs to conduct timely and useful analysis of portfolio quality, determine trends in the portfolio over time, and identify possible issues for mitigation.

5.2.3 Percentage of Investment Funds in Active Loans

The percentage of investment funds in active loans shows the extent that available CBDC loan financing is being put to use in communities versus sitting in financial reserves. While the overall Atlantic percentage of CBDC investment funds in active loans was relatively consistent, showing a slight increase over the past decade, there was variability among CBDCs.

According to the national Community Futures PMS definition, which takes into account CBDC investments in the ACCBIF, the overall percentage of CBDC funds in active loans surpassed the Agency target of 70 per cent each year from 2007-2008 to 2012-2013.[See Footnote 36] The percentage of funds in active loans ranged from 82.8 per cent in 2008-2009 to 84.6 per cent in both 2009-2010 and 2010-2011.[See Footnote 37]

The evaluation also examined variability in the percentage of CBDC investment funds in active loans based on the calculation used by ACOA to determine individual CBDC operational funding levels. This calculation is different from that used to report in the Departmental Performance Report as it shows the percentage of funds in active loans prior to adjustments made to reach a 70 per cent operational target. As noted previously, as of April 2010, all CBDC operational funding agreements included a clause ensuring that each recipient has a minimum of 70 per cent of its investment funds in active loans; CBDCs must invest any shortfall in the ACCBIF to reach the 70 per cent operational target.[See Footnote 38]

Based on this operational formula, Table 11 presents the percentage of funds in active loans by region and Atlantic-wide. Cape Breton and Nova Scotia regional data were combined to protect the anonymity of the sole CBDC located in Cape Breton.[See Footnote 39] The Atlantic percentages ranged from a low of 81.5 per cent in 2008-2009 to 83.6 per cent in 2012-2013.[See Footnote 40] However, variability exists among the regions. CBDCs in New Brunswick have the highest percentages of investments in active loans and Newfoundland and Labrador CBDCs have the lowest. The percentage of funds in active loans has increased since the previous evaluation, with the current six-year average at 82.6 per cent compared to the five-year average of 77.8 per cent reported in the 2009 evaluation.

Table 11: Percentage of Funds in Active Loans by ACOA Region

Region	2007- 2008	2008- 2009	2009- 2010	2010- 2011	2011- 2012	2012- 2013	6 Year Average
NB	93.0%	94.4%	93.0%	93.1%	87.4%	87.7%	91.4%
NL	71.3%	68.1%	71.4%	71.1%	71.0%	68.8%	70.3%
NS*	78.6%	83.2%	88.1%	86.4%	88.5%	87.9%	85.5%
PEI	89.1%	80.2%	81.1%	78.2%	79.3%	89.8%	83.0%
Atlantic	83.0%	81.5%	83.4%	82.2%	81.6%	83.6%	82.6%

Source: CBBD program calculations to determine operational funding levels using CBDC E-Report data.

* Data for the one CBDC in Cape Breton was included with the Nova Scotia region calculations.

Key informants stated that the 70 per cent operational target motivates CBDCs not only to make more loans but to take on riskier loans, fulfilling the raison d’être of their organizations. However, several key informants reported that some CBDCs continue to be risk-averse and do not do enough lending, as evidenced by the variability in the percentage of investment funds in active loans among CBDCs. Informants noted that economic factors such as population shifts and industry closures could be at play in many locations with limited lending.

5.3 Alternative Modes of Delivery, Lessons Learned and Best Practices

Judgment Criteria	Key Findings
Program management has considered and continues to explore alternative modes of delivery.	The implementation of the CFoT funding model during the period reflects management’s consideration of alternative modes of delivery. Program management has monitored progress and unintended outcomes through the CFoT Oversight Committee. Also, the Agency is considering increasing the maximum CBDC loan limit, on an exception basis, along with new research related to the economic development of rural communities.
Lessons learned and best practices are identified regularly both by ACOA and funding recipients.	The key lesson learned was the need to monitor and manage any unexpected behaviours that result from the CFoT model. Best practices include: strong collaboration and coordination among CBDCs and associations as well as between ACOA and the CBDC network; the collection and reporting of performance information through standardized financial statements and E-Reports; and the use of Statistic Canada data to contrast the results of CFP-assisted firms with comparable firms. Promising practices include implementation of governance standards (accountability, transparency, confidentiality) and board and staff professional development and skills training.

Judgment Criteria

Key Findings

Program management has considered and continues to explore alternative modes of delivery.

The implementation of the CFoT funding model during the period reflects management’s consideration of alternative modes of delivery. Program management has monitored progress and unintended outcomes through the CFoT Oversight Committee. Also, the Agency is considering increasing the maximum CBDC loan limit, on an exception basis, along with new research related to the economic development of rural communities.

Lessons learned and best practices are identified regularly both by ACOA and funding recipients.

The key lesson learned was the need to monitor and manage any unexpected behaviours that result from the CFoT model.

Best practices include: strong collaboration and coordination among CBDCs and associations as well as between ACOA and the CBDC network; the collection and reporting of performance information through standardized financial statements and E-Reports; and the use of Statistic Canada data to contrast the results of CFP-assisted firms with comparable firms.

Promising practices include implementation of governance standards (accountability, transparency, confidentiality) and board and staff professional development and skills training.

5.3.1 Alternative Modes of Delivery

The evaluation does not identify a more effective or efficient approach to the delivery of the CBBD programming. Almost all of the surveyed CBDC funding recipients (97 per cent) and key informants agreed that using the CFP to support the network of CBDCs to deliver on the CFP objectives was both an appropriate and a cost-effective way to support SMEs. Many interviewees stated that the strength of the program resides in the local involvement and decision making through volunteer boards of directors that know the communities in which they are based.

As discussed previously, key informants spoke of important interrelated contextual issues affecting CBBD programming, particularly lending activity, including aging population, out-migration, skilled labour shortages, access to capital, struggling local economies and business succession planning opportunities. They confirmed that ACOA program management is aware of shifting contextual factors and has considered implications on programming.

ACOA program management has worked closely with CBDCs to enhance lending activity since the 2009 evaluation of the CFP. Most notably, key informants stated that the introduction of the CFoT funding model is expected to add to efficiency and effectiveness of the programming with an operational funding formula based on performance and the RMF, introducing a shared risk approach to increase lending activity. While it is too early to form a conclusion on the extent of improvements, the CFoT Oversight Committee is monitoring the implementation and intended and unintended outcomes of the model.

ACOA has increased the loan limit for CBDCs, allowing CBDCs to provide financial assistance to SMEs (under exceptional circumstances) up to $225,000 from the previous level of $170,000. This is expected to decrease the amount of surplus investment and increase lending activity. A 2013 study commissioned by the AACBDC identified that in light of the decreasing number of loans and increased CBDC liquidity, increasing the loan limit under exceptional circumstances would increase lending activity.¹ The decision to increase loan limits was supported by many CBDC and ACOA key informants.

Through its policy function, the Agency is considering new research related to the economic development of rural communities in recognition of trends related to out-migration,[See Footnote 41] aging populations and changes to traditional industries. Key informants also identified the need for rural business succession planning and decreasing access to traditional financial institutions in rural communities, both requiring further consideration of implications on program delivery moving forward.

5.3.2 Lessons Learned and Best Practices.

As discussed previously, the evaluation identified possible unintended CBDC financial reporting practices resulting from the new CFoT model. According to several key informants, some CBDCs were not fully satisfied with the operational funding model based on performance and need, believing that a strong return on investment should not diminish operational funding from ACOA. It was possible with the original CFoT formula to adjust non-cash expenses in order to decrease return on investment, thereby benefiting operational allowances. The CFoT Oversight Committee was considering actions in 2013-2014 to adjust the funding model to ensure that non-cash expenses (i.e. provisional expenses) would not be used as a factor in determining future operational budgets.

The evaluation identified a number of best practices associated with the CBBD programming, including the strong collaboration and coordination among CBDCs and associations as well as between ACOA and the network of CBDCs. The establishment of a national PMS for the Community Futures Program for the identification, collection and reporting of performance information is an important improvement over the last CFP evaluation and a best practice. Other best practices relate to the use of standardized financial statements, special purpose reports and E-Reports, all of which contribute to more consistent performance reporting. The use of Statistics Canada data to report on the impact of CBDC-assisted SMEs with those of comparable firms is another best practice that provides significant evidence of the impact of CBDCs on SMEs.

There are several promising practices that are in the early stages of implementation or that are region-specific. The Nova Scotia region’s pilot project of providing the Nova Scotia Association of CBDCs with funding to undertake a consulting advisory services initiative puts decision making closer to the local level and decreases ACOA’s administration. Consulting advisory services provide new and expanding businesses with financial assistance to hire consultants or other experts to conduct business diagnostic assessments, develop management skills, analyze market readiness and export potential, and improve productivity or efficiency. According to key informants, other regions have recently implemented a similar approach in using the CBDCs as a means of delivering such services to SMEs.

New governance standards and board governance training indicate potential for greater CBDC board capacity, including enhanced skills, accountability, transparency and confidentiality. Some regions reported positive results from board and staff professional development and skills training activities. According to key informants, these initiatives are being monitored by ACOA and the CFoT Oversight Committee to ensure they are achieving intended outcomes, including the need for adjustments.

6. Conclusions and Recommendations

The CBBD evaluation conclusions and recommendations were identified from analysis of the key findings presented in this report. They were validated through discussion and consultation with ACOA and other stakeholders. Table 12 shows the linkages between key findings, conclusions and recommendations.

6.1 Conclusions

The following nine conclusions were identified by the evaluation team based on the evidence presented in this report:

Access to capital remains an important need of rural entrepreneurs in Atlantic Canada. ACOA is aware of existing and emerging needs and is adjusting its programming in response, largely through the implementation and monitoring of the CFoT model.
Activities undertaken as part of the CBBD sub-program are relevant for ACOA and the federal government and to the needs of local, rural communities. CBBD activities continue to be aligned with ACOA and broader federal priorities related to economic development. CBDC activities align with the needs of local, rural communities. Financial as well as non-financial services are important to SMEs.
CBBD programming is consistent with the roles, responsibilities and mandate of similar programming delivered by other federal departments and agencies across Canada. Within Atlantic Canada, CBBD programming complements rather than duplicates similar programming delivered by other organizations. The programming has unique qualities: local community-based governance and delivery; a suite of support services in addition to financing; and the ability to support higher risk SMEs.
ACOA provides important financial support to community economic development projects in Atlantic Canada through its CBBD sub-program. Its funding is incremental to the activities of the CBDCs and to the associations that support them as well as to Ulnooweg. Program funding positively impacts the creation and the expansion or modernization of SMEs in rural Atlantic Canada and many would not have proceeded without the Agency’s support.
The CBBD programming is achieving expected outcomes by providing SMEs with access to capital to start, modernize and expand businesses. The sub-program has contributed to the creation and the expansion or modernization of SMEs in rural Atlantic Canada.
ACOA and the CBDCs are aware of the internal and external contextual factors that impede the success of CBBD programming and it is evident that many are being mitigated. Some factors would benefit from further investigation: opportunities for consistency in service delivery; SME succession planning; gaps in skills development.
Early evidence suggests that the CFoT is on track to meet its stated objectives. Further time will be required in order to fully assess the impact on program effectiveness. There are questions related to the risk of unintended behaviours resulting from the CFoT as well as the ongoing need for and composition of a governance committee.
ACOA and funding recipients have made significant progress in ensuring the availability and use of performance measurement information, allowing for better and timelier programming decisions and improved communication of results. Gaps include client contact information, data on non-financial services and certain Community Futures PMS collection data.
The existing model of delivery of the CBBD programming through CBDCs and Ulnooweg allows the Agency to achieve an important part of its mandate in an efficient and economical manner. CBDC investment funds appear to be well managed. Mechanisms have been established to better support the ongoing monitoring and management of investment funds going forward. Better collaboration between Ulnooweg and the CBDCs holds potential for additional efficiencies.

6.2 Recommendations

The three evaluation recommendations reflect discussion and advice from the members of the Evaluation Advisory Committee. Each of the recommendations aim to build upon progress made to program delivery, performance and efficiency since the previous evaluation, while ensuring that ACOA identifies and considers emerging programming needs on an ongoing basis.

Recognizing the progress that has been made since the previous evaluation, continue to improve the performance of CBBD programming. There has been significant improvement to the delivery of the programming, particularly through the recent implementation of the CFoT model. In the spirit of continuous improvement, it is recommended that ACOA program management:
1. Work in co-operation with the CBDCs and their associations to increase understanding of the CFoT model among CBDC and ACOA staff and to foster the achievement of objectives and expected behaviours while minimizing risks. In particular, it is recommended that governance mechanisms be reviewed to better understand and reflect the complex relationships between ACOA and the CBDC network and the ongoing work of the CFoT Oversight Committee.
2. Support CBDCs and their associations in exploring opportunities for greater consistency of service delivery in areas that hold the greatest potential for enhancing effectiveness and efficiency of the programming.
3. Engage in dialogue with the CBDCs, their associations and Ulnooweg to identify areas for enhanced collaboration with the aim of improving performance and efficiency of programming for Aboriginal entrepreneurs across Atlantic Canada.
Further advance policy research and knowledge of economic development in rural regions to ensure emerging programming needs are identified and addressed and that programming remains relevant and effective. It is recommended that ACOA program management, in co-operation with the CBDCs and their associations, consider ongoing research on economic development in rural regions to further support the strategic direction of the CBDCs and Ulnooweg. Particular areas of knowledge development could include the geographical evolution of economic activities, changing demographics, access to capital and the extent and impacts of the loss of banking institutions in rural communities, business succession planning and other supports for SMEs in rural communities.
Continue to strengthen the availability and use of performance measurement information in support of results-based management and decision-making. It is recommended that ACOA program management work in collaboration with CBDC associations (in support of their members) to improve information technology and tools as well as to address remaining gaps in performance information by:
1. Supporting the standardized collection by all CBDCs of client contact information necessary for program management, performance measurement and evaluation purposes. This should include contact information for clients receiving financial and/or non-financial services.
2. Examining the non-financial services offered by the CBDCs (e.g. provision of business information, counselling and skills development) in order to identify data that could be used to describe, track and better understand the need for and utilization of these services. Data collection should be focused on value-added information for the management of CBDCs and should be integrated into the existing reporting platform.
3. Supporting ongoing work through the National Community Futures Performance Measurement Strategy Committee to review the implementation of the strategy, including the validation of indicators, collection of data and setting of targets.

Table 12: Alignment of CBBD Evaluation Findings, Conclusions and Recommendations

(D)

Appendix A – CBBD Evaluation Management Action Plan

Recommendations	Management Responses	Planned Actions	Responsibility	Target Date
1. Recognizing the progress that has been made since the previous evaluation, continue to improve the performance of the CBBD programming. There have been significant improvements to the delivery of the programming, particularly through the recent implementation of the Community Futures of Tomorrow ( CFoT) model. In the spirit of continuous improvement, it is recommended that ACOA program management:
a) Work in co-operation with the CBDCs and their associations to increase understanding of the CFoT model among CBDCs and ACOA staff and to foster the achievement of objectives and expected behaviours while minimizing risks. In particular, it is recommended that governance mechanisms be reviewed to better understand and reflect the complex relationships between ACOA and the CBDC network and the ongoing work of the CFoT oversight committee.	Agreed	i. In consultation with the AACBDC, ACOA will define the roles and responsibilities of ACOA Head Office, ACOA regional offices and the CFoT monitoring functions/Oversight Committee.	HO CD DG Regional DGs	Summer 2014
		ii. ACOA, in consultation with the AACBDC, will develop and provide tools to ACOA staff across the regions to increase awareness and common understanding of the CFoT model, roles and responsibilities of ACOA Head Office, ACOA regional offices and the CFoT monitoring functions/Oversight Committee.	HO CD DG Regional DGs	Fall 2014
		iii. ACOA will meet with individual CBDC boards and use tools developed to ensure common understanding of the CFoT monitoring functions, overall objectives, roles and responsibilities.	HO DG Regional DGs Members of the OC	January 2015
		iv. ACOA, in collaboration with the AACBDC, will undertake a review of the CFoT model to ensure model parameters are encouraging behaviours that support the CFoT objectives. Where possible, the model parameters will be adjusted to provide clarity and simplicity.	HO CD DG Regional DGs Members of the OC	Spring 2015
b) Support CBDCs and their associations in exploring opportunities for greater consistency of service delivery in areas that hold the greatest potential for enhancing effectiveness and efficiency of the programming.	Agreed	i. ACOA will undertake a consultative process with the AACBDC and the provincial associations of the CBDCs to discuss and explore opportunities for greater consistency of service delivery.	HO CD DG Regional DGs	Summer 2015
	Agreed	ii. ACOA, in collaboration with the AACBDC and the provincial associations, will help develop and support the implementation of an action plan that identifies areas that hold the greatest potential for enhancing effectiveness and efficiency of the programming.	HO CD DG Regional DGs	Summer 2015
c) Engage in dialogue with CBDCs, their associations, and Ulnooweg to identify areas for enhanced collaboration with the aim of improving performance and efficiency of programming for Aboriginal entrepreneurs across Atlantic Canada.	Agreed	i. ACOA will engage in a consultative process with the AACBDC and Aboriginal business development organizations to discuss and explore opportunities for improving collaboration, with the aim of developing an action plan that enhances performance and efficiency of programming to Aboriginal entrepreneurs.	HO CD DG	Fall 2014
2. Further advance policy research and knowledge of economic development in rural regions to ensure emerging programming needs are identified and addressed and that programming remains relevant and effective. It is recommended that ACOA program management, in co-operation with the CBDCs and their associations, consider ongoing research on economic development in rural regions to further support the strategic direction of the CBDCs and Ulnooweg. Particular areas of knowledge development could include the geographical evolution of economic activities, changing demographics, access to capital, and the extent and impacts of the loss of banking institutions in rural communities, business succession planning and other supports SMEs in rural communities.	Agreed	i. ACOA’s CD senior management will collaborate with the Policy unit to ensure the policy research agenda links with ACOA’s strategic priorities that support the efforts of the AACBDC and CBDCs to address issues such as development of key sectors, succession planning for rural SMEs, access to traditional banking institutions in rural communities, etc.	HO CD DG HO DG Policy	Spring 2015
		ii. ACOA will use Functional Regions research and other relevant research related to rural economic development to better inform the economic context in which CBDCs operate to ensure activities and supports are aligned with regional needs.	HO CD DG	Spring 2015
		iii. ACOA will ensure that support provided to the Atlantic Aboriginal Economic Development Integrated Research Project (AAEDIRP) through the Aboriginal Policy Congress (APC) aligns with the needs of Aboriginal business development organizations.	HO CD DG	Fall 2015
3. Continue to strengthen the availability and use of performance measurement information in support of results-based management and decision making. It is recommended that ACOA program management work in collaboration with the CBDC associations (in support of their members) to improve information technology and tools as well as to address remaining gaps in performance information by:
a) Supporting the standardized collection by all CBDCs of client contact information necessary for program management, performance measurement and evaluation purposes. This should include contact information for clients receiving financial and/or non-financial services.	Agreed	ACOA will support the AACBDC in the standardization of data collection needs and the implementation of the new CBDC loan platform, which includes key client profile information for clients receiving financial and non-financial services. This data will be collected on a go-forward basis. Historical data will be maintained in its current form.	HO CD DG	April 2015
b) Examining the non-financial services offered by CBDCs (e.g. provision of business information, counselling and skills development) in order to identify data that could be used to describe, track and better understand the need for and utilization of these services. Data collection should be focused on value-added information for the management of the CBDCs and should be integrated into the existing reporting platform.	Agreed	i. ACOA will consult other RDAs on what non-financial data they collect and how.	HO CD DG	Fall 2014
		ii. ACOA will work with the AACBDC and the CBDCs to determine what non-financial activity indicators would provide all parties with better performance information needed for management decision making.	HO CD DG	Spring 2015
		iii. ACOA will develop an action plan, in consultation with the AACBDC, to ensure non-financial data is being collected appropriately.	HO CD DG	April 2015
c) Supporting ongoing work through the National Community Futures Performance Measurement Strategy ( CF PMS) Committee to review the implementation of the strategy, including the validation of indicators, collection of data and setting of targets.	Agreed	i. ACOA will support FedNor as the chair of the PMS committee for 2014 in their work to validate indicators and target setting in the CF PMS and making necessary adjustments to reflect current programming needs.	HO CD DG	Winter 2015
		ii. ACOA will provide a leadership role in managing the Business Number Analysis Agreement with StatsCan.	HO CD DG	Fall 2014
		iii. ACOA will complete the CBBD PMS for the Agency and will ensure that both performance frameworks are aligned.	HO CD DG	Summer 2014

Appendix B – Community Economic Development Conceptual/Analytical Framework

(D)

Appendix C – Evaluation Questions, Judgment Criteria and Methods

When judgment criteria in the table below refer to the results of previous evaluations as a benchmark for assessing current program success, it should be noted that in some cases, baseline information is not available. In those cases, this evaluation will be used to gather baseline data for future evaluations.

Evaluation Questions	Judgment Criteria	Doc/ Literature Review	Interviews	Client Survey [See Footnote 42]	Administrative Data	Statistics Canada Data [See Footnote 43] Analysis
Relevance – extent to which the CBBD sub-program addresses a demonstrable need and is relevant to ACOA’s mandate, strategic objectives as well as government-wide priorities and strategies
Issue 1: Continued Need for the Program
1.1 To what extent does the CBBD sub-program continue to address a demonstrable need? ( Q1.1 [See Footnote 44])	The needs that the programming is expected to meet are still present to at least the same degree as they were five years ago.	x	x	x		x
1.2 To what extent is the CBBD sub-program responsive to existing and emerging needs of Atlantic Canadians? (Q1.1)	Business loans and counselling activities are yielding results and emerging needs were responded to. Urban/rural/sectoral needs still exist to at least the same degree as they did five years ago. The role that CBBD funding recipients play in communities is consistent with their mandate, and with community needs and expectations.	x	x	x		x
Issue 2: Alignment with Government Priorities
2.1 To what extent is the CBBD sub-program aligned with federal government priorities and expectations, and to ACOA’s strategic outcome? (Q1.2)	There is logical alignment between the programming, federal government priorities and ACOA’s strategic outcome. The alignment is recognized and made explicit.	x	x
Issue 3: Alignment with Federal Roles and Responsibilities
3.1 To what extent does the CBBD sub-program align with federal roles and responsibilities? What other mechanisms exist to address these needs? (Q1.3; Q1.4 )	ACOA is mandated by law to fulfill the role. Other jurisdictions administer such programming through the federal government. Needs are met by alternative services (extent of duplication, overlap or complementarity).	x	x
Performance
Issue 4: Effectiveness - the extent to which CBBD objectives have been achieved within the context of expected results and outcomes.
4.1. Incrementality: What impact would the absence of the CBBD funding/assistance have on the start-up, survival and growth of the SME? (Q2.1)	The impact of the absence of the programming is similar or greater to that observed during the previous evaluation. Survival rates of SMEs supported through CBBD funded activity are at least as high as survival rates of unsupported SMEs. (Note: This judgment criterion reflects the assumption that CBBD programming supports firms that traditional lenders would consider to be higher risk).	x	x	x		x
4.2 To what extent have the CBBD outcomes been achieved? (Q2.2) [See Footnote 45] Immediate Atlantic Canadian businesses have access to capital to start, modernize and/or expand their businesses CBDCs have access to the tools and knowledge needed to operate efficiently Creation of SMEs Expansion, modernization and stabilization of existing businesses Intermediate: Atlantic Canadian businesses have access to capital Atlantic Canadian businesses have access to business information and counselling Ultimate: Dynamic and sustainable communities for Atlantic Canada	The performance of CBDC-supported SMEs is at least as good as that of similar, unsupported SMEs in terms of supporting employment growth and maintaining businesses. According to the CF PMS, targets for the following CFP indicators (applicable to CBBD) will be developed during 2012 and will be used to establish baseline data for future evaluations of CBBD. For non- CFP projects within CBBD, the following indicators will be used (where possible) or alternative indicators will be developed: PMS 1: Employment growth PMS 2: % of businesses funded by industry PMS 3: % of employed persons working in various industry sectors PMS 4: Businesses’ productivity PMS 5: Business survival rate PMS 6: Clients indicating improved business practices as a result of CFOs support (CFO Survey) PMS 7: # of new business start-ups (through lending only) PMS 8: Growth in sales PMS 10: % of active loans PMS 11: Number and value of start-up loans PMS 12: Number and value of other loans PMS 13: Amount leveraged per dollar invested by CFO PMS 14: Level of client satisfaction with CFO business development services (CFO Survey)	x	x	x	x	x
4.2.1 To what extent do the results achieved for the CBBD sub-program align with the immediate, intermediate and ultimate outcomes of the CFP? ( Q2.2)	Evidence that a strong alignment exists between the CBBD and the CFP logic models and associated PM strategies. Ability to address the outcomes of the CFP (note: outcomes in italics do not apply to CBBD [See Footnote 46]): Immediate Outcomes: Improved access to capital and leveraged capital through loans, loan guarantees and equity investment to businesses and social enterprises Enhanced/maintained business development services, information, counselling, referrals, training Strengthened community strategic plans CED Projects and Partnerships Intermediate Outcomes: Improved business practices and increased entrepreneurship Strengthened and expanded businesses Strengthened capacity for socio-economic development [See Footnote 47] Ultimate Outcomes: Economic stability, growth and job creation Diversified and competitive local rural economies Economically sustainable communities.	x	x	x	x	x
4.3 What are the barriers to achieving CBBD immediate, intermediate and ultimate outcomes, and to what extent are these being mitigated? ( Q 2.2)	Evidence that factors that are impeding success of the CBBD sub-program are known and mitigation strategies exist.	x	x	x		x
4.4 To what extent have unintended impacts been achieved from CBBD programming? (Q2.3)	Not applicable.	x	x	x		x
4.4.1 Has the CFoT funding model had a positive impact on the effectiveness and efficiency of CF programming under the CBBD? ( Q 2.2; Q3.1-3.2)	Quantitative and qualitative evidence that the CFoT funding model has increased the effectiveness and efficiency of the delivery of CF programming. (Applicable to CBDCs only)	x	x	x	x
4.5 To what extent have the CBBD performance measurement and reporting structures been implemented and effectively contribute to reporting CBBD outcomes? How is the performance information being used by ACOA and ACOA-funded organizations to support decisions making? ( Q 2.4)	Qualitative evidence that the CBBD sub-program performance measurement data is adequate and is effective in reporting on the achievement of outcomes. Qualitative evidence that the CBBD sub-program performance measurement data is considered useful and is being used by ACOA and ACOA-funded organizations. Performance data is being collected and is available for use during the evaluation.	x	x	x	x
Issue 5: Efficiency and Economy – the extent to which CBBD activities are undertaken in an affordable manner, taking into consideration the relationship between outputs and the resources to produce them and the extent to which resources allocated to the CBBD sub-program are well-utilized, taking into consideration alternative delivery mechanisms.
5.1 To what extent is the CBBD sub-program efficient in the context of the results being achieved? ( Q3.1)	Qualitative evidence that ACOA has in place structure/mechanisms to ensure that the most efficient and economical means are being used to administer the programming. For example: Budget/forecasts Expenditures (e.g. salary and operations) Governance processes Priority setting/planning Timeliness of delivery Nature of work with network of CBDCs Delivery costs compare favourably to the delivery costs as calculated during the previous evaluation and to the delivery costs of similar programming offered in other jurisdictions.	x	x	x	x
5.1.1 To what extent do CBBD funding recipients have access to the tools and knowledge needed to operate efficiently? ( Q3.1)	Qualitative evidence that CBBD funding recipients have access to the tools and knowledge needed to operate efficiently.	x	x	x	x
5.2 Is there a more cost-effective way of achieving expected results, taking into consideration alternative delivery mechanisms, best practices and lessons learned? ( Q3.2)	Lessons learned and best practices are identified regularly. CBBD funding recipients take into consideration their own best practices/lessons learned in improving alternative approaches (efficiency). Program management has considered and continues to explore alternative modes of delivery (note: efficiencies resulting from the implementation of the CFoT model will be explored here – applicable to CBDCs only).	x	x	x
5.3 Are CBDC investment funds well managed? ( Q3.3)	Management of loan funds compares favourably to set targets, to the previous evaluation and to practices in other similar organizations (e.g. Ulnooweg Development Group Inc.). CBDCs board governance standards align with Board Governance Minimum Standards adopted by ACOA in 2012-2013. [See Footnote 48]	x	x	x	x
5.3.1 Are CBDC loan loss rates acceptable? Do the CBDCs carry an acceptable level of risk? ( Q3.3)	Same as above. (Applicable to CBDCs only) Common definition of loan loss rate is the following: Loan loss rate defined as: Annual loan loss expense / Gross loan receivable	x			x

Limitations	Mitigating Strategies
Attribution vs. Contribution: due to its complexity, programming results can be influenced by a multitude of factors	CD impact model accounts for contextual variables that can influence outcome achievement. Statistics Canada data used to compare clients and non-client performance.
Weaknesses in 2009 CF Survey – representativeness of responses is unclear.	2013 survey questions revised to reflect clearer, more concise wording. Sampling strategy to ensure representative sample; 90% confidence interval.
New PWGSC requirements for contractor approval, including site inspections, meant that timelines for fielding both surveys was delayed until December 2013.	Consider alternate models of administering the surveys (e.g. in-house using SharePoint) if contract could not proceed in a timely manner due to contracting delays. Use of telephone surveys as a backup for e-mail surveys to ensure representativeness and to meet confidence levels. Extended fielding of surveys into mid-January to account for the holiday period, with multiple reminder notices from both survey firm and ACOA to recipients.
Many CBDCs did not have an e-mail address (52 per cent), or telephone (7 per cent) number or other client contact information available in a consistent electronic format. Data gaps varied by CBDC, ranging from 23 to 99 per cent of e-mail addresses missing and 0 to 55 per cent of telephone numbers incomplete. Limited information was available on non-loan clients (i.e. those that accessed counselling, advice or referral services).	Use of a customized standard report template to obtain data from the TEA system in an electronic format. Follow up with individual CBDCs to fill in data gaps. Use of telephone surveys as a backup for e-mail surveys to ensure representativeness and to meet confidence levels. Focus client survey on financial activities.
Implementation of the CFoT – A new delivery model was implemented in 2011-2012, resulting in only one full year of data related to 2012-2013.	Scope of evaluation amended to include 2012-2013 fiscal year. Incorporate questions related to the CFoT and its progress in survey and key informant interviews. Consider Seed Capital (Rural) E-Report data as part of data set prior to 2011-2012.

Appendix E – Community Futures Program Logic Model

(D)

Footnotes

[1]Partnering departments and agencies include: Canada Economic Development for Quebec Regions; Western Economic Diversification Canada; Federal Economic Development Agency for Southern Ontario; and Federal Economic Development Initiative for Northern Ontario, under Industry Canada.

[2]SCI expenditures are reflected in CBBD totals but were not a focus of the evaluation due to the SCI review undertaken by ACOA senior management in 2011-2012 and 2012-2013. The study, conducted by an external consulting firm, found that the SCI continues to be relevant in supporting youth entrepreneurship and identified a number of areas for increasing efficiencies related to program delivery. The report’s conclusions and recommendations were considered as part of the CBBD evaluation document review.

[3]Projects initiated prior to 2007-2008 but amended within the study’s five-year time frame were included in the evaluation.

[4]Prairie Research Associates Inc provided support to plan, administer and analyze the findings for two surveys, one to CBDC Executive Directors and board members (recipients) and another to clients of the CBDCs.

[5]All recommendations resulting from the 2009 study were fully addressed. Recommendation 7, related to identifying reasons for the variances across regions in investment dollars disbursed, was cancelled as it was addressed during the development the Community Futures of Tomorrow model.

[6]Appendix E, National Community Futures PMS, 2012.

[7]The survey methodology was supported by an external consultant. In order to maintain a uniform sample of clients (given the nature of operations and funding arrangements) and to enable comparison to the previous evaluation, only CBDCs were included in the recipient survey. Other CBBD recipients such as the Atlantic and provincial associations and the Ulnooweg Development Group were covered through interviews.

[8]There are different types of Community Future organizations across the country. They are called Community Futures Development Corporations in Quebec and Ontario, and Community Futures Organizations (CFOs) in Western Canada. Linking the national network of 259 CF organizations are 19 provincial, regional and national CF associations collaborating directly with federal, provincial and municipal governments.

[9]These activities are not within the scope of the CBBD evaluation but were reflected in the Community Mobilization Community Investment Evaluation (2014).

[10]As of 2011-2012, 39 of 41 CBDCs had accepted all aspects of the CFoT model, including operational funding. Two CBDCs in Cape Breton (INRICH CBDC and CBDC Northside Victoria) decided to continue providing services using their existing resources to support their operations; however, they remain eligible recipients under the Community Futures Program.

[11]The 2009 CFP evaluation found that the variability in percentage of funds in active loans suggested that some CBDCs were underperforming and recommended that ACOA work with the CBDCs to establish appropriate targets for active loans, with 70 per cent being a reasonable target based on analysis of performance data.

[12]A comparison of the CBBD and CFP logic models was undertaken by the evaluation team to ensure alignment of activities, outcomes and outputs. No significant gaps were identified.

[13]Source: GX financial systems; ACOA Departmental Performance Reports (2007-2008 to 2012-2013).

[14]Expenditures related to Cape Breton Region are overstated in ACOA's GX system and have been reduced by approximately $481,000 to reflect actual expenditure data provided by ECBC.

[15]The ACOA assistance amounts include projects amended within the scope of the evaluation.

[16]Corresponds to “sub-element” field in QAccess.

[17]Provision of Capital represents funding for: core training, Seed Capital, strategic initiatives, recapitalization fund and the entrepreneurial training fund.

[18]Consulting advisory services is a vehicle to secure external consultant support for SMEs. CBDCs will engage consultants to work with identified clients on an individual basis to develop customized solutions and plans for any one of the following activities: (a) diagnostic assessment, (b) business management skills development, (c) assessing export potential, (d) enhancing competitiveness, and (e) access to capital.

[19]According to Statistics Canada, the labour force participation rate is determined by the percentage of the population aged 15 years and over that is either employed or unemployed.

[20]Based on Statistics Canada analysis, data include overtime and are based on gross taxable payroll before source deductions.

[21]The ACOA PAA changed in 2014-2015 to reflect two sub-programs under the CD program: http://www.acoa-apeca.gc.ca/eng/publications/ParliamentaryReports/Pages/RPP_2014-15_SecI.aspx#sopaa.

[22]Seed Capital and general CFP investment funds were merged into one investment fund in 2011-2012. During the first four years covered by the evaluation, Seed Capital loans, also issued by CBDCs, were captured by the Agency separately.

[23]The business survival rate is calculated using the cumulative method, defined as the ratio of firms that have entered the market and that are still operating over the total number of firms that entered the market over a specific period of time.

[24]TEA software: the tool used by CBDCs to capture client contact information and project related data.

[25]According to the Treasure Board Secretariat Directive on the Evaluation Function (Canada, 2009a), demonstration of efficiency and economy is an “assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.”

[26]Operational cost per investment = total operating costs (ACOA operational expenditures plus CBDC operational expenditures) / total CBDC investment or (14,093,000 + $104,031,611)/ $353,130,434. There are two issues that suggest that the actual operational expense per investment is likely lower: 1) not all ACOA operating costs are associated with CBDCs and 2) While this analysis assumes that all the operational expenses were for loan activities, there are non-financial services provided to entrepreneurs who do not receive loans.

[27]Though CBDCs also received BDP and a very small amount of ICF funding from ACOA over the period, totalling $3.6 million, the evaluation focused on CFP for this calculation as CFP provided over 94 per cent of total G&Cs to CBDCs.

[28]The 2009 Evaluation of CFP included investment per job created or maintained considering CFP only; calculations did not include Seed Capital funding.

[29]The assessment of loan management was challenging due to a lack of consistent data across the evaluation period (i.e. merging of Seed Capital and CFP investment funds in 2011-2012, standardized financial statements introduced in 2011-2012) and difficulty comparing results to the 2009 CFP evaluation.

[30]These loan values represent both CF and Seed Capital. The average CF loan actually increased from $43,352 in 2007-2008 to $47,193 in 2009-2010 and $46,639 in 2010-2011, and then decreased to $38,828 in 2011-2012 and went back up slightly to $42,343 in 2012-2013.

[31]A number of factors prevented a full assessment of loan loss rates: the recent adoption of loan loss calculations based on provisional expenses; inconsistent financial data for the entire period; and the lack of a target range of acceptable rates in Atlantic Canada.

[32]The formula used to calculate loan loss rates based on provision is: provision of bad debt expense/gross loans receivable.

[33]The formula used to calculate loan loss rates based on write-offs is: write-offs/total value of outstanding investment.

[34]Because Seed Capital investment funds were merged into the general CFP investment funds beginning in 2011-2012, the current study included Seed Capital investment funds in all six fiscal years.

[35]The 2009 CFP evaluation considered only CFP funds for loan loss rate calculations; calculations of the CFP-only loan loss rates for 2011-2012 and 2012-2013 were not possible due to the merging of the CFP and Seed Capital investments during those years.

[36]ACOA began reporting the percentage of investment funds in active loans as part of its Departmental Performance Report in 2012-2013. These calculations are based on financial data from the previous year due to the timing of the DPR and availability of data. There was an error in the calculations used for the figure included in the 2012-2013 DRP; therefore, the evaluation used the revised figure as per program calculations. The Agency target increased to 80 per cent in 2013-2014.

[37]The National Community Futures PMS definition of percentage of total CBDC funds in active business loans was established as total active loans value ÷ total investment fund balance. For ACOA, the definition takes into account the CBDC funds invested into the ACCBIF by subtracting that amount from the denominator. The two Cape Breton CBDCs that did not sign on to the CFoT are not part of the calculations for 2010-2011 or 2011-2012 because their financial data was not readily available.

[38]The change in the funding formula was made based on a recommendation of the 2009 CFP evaluation. The report found that the variability in percentage of funds in active loans suggested that some CBDCs were under-performing and recommended that ACOA work with CBDCs to establish appropriate targets for active loans, with 70 per cent being a reasonable target based on analysis of performance data.

[39]The calculations provided by the program do not include the two CBDCs that did not sign on to the CFoT for any of the years presented.

[40]The percentage is established as: Total Loans ÷ (Cash + Amount loaned to ACCBIF − 37k + Total Loans). The $37,000 adjustment is required to offset a one-time investment by all CBDCs in the ACCBIF.

[41]Out-migration in this sense includes population shifts from rural to urban centres as well as from Atlantic Canada to western parts of the country.

[42]Depending on the issue to be addressed, reference to client survey under methods refers to a survey of CBDCs and/or to a survey of the clients of CBDCs (the ultimate client).

[43]Statistics Canada data refers to both performance measurement and census data.

[44]Questions in italics refer to the alignment with the CFP PMS Evaluation Framework Questions (Appendix B) as per July 30, 2012 version.

[45]In order to meeting CFP reporting requirements as outlined in the National CFP PMS, the achievement of the CFP outcomes will be assessed as part of addressing the achievement of CBBD outcomes.

[46]In Atlantic Canada, CFP activities related to CED and strategic planning are not undertaken by CBDCs, but by other not-for-profit organizations who are not constrained by the urban/rural limitations imposed upon CBDCs.

[47]This outcome links to the Community Mobilization/Community Investment (CM/CI) evaluation currently underway. Collaboration with CM/CI evaluation team will occur during conduct of CBBD evaluation to ensure linkages related to this outcome are noted.

[48]ACOA began implementing minimum board governance standards with CBDCs in 2012-2013. The purpose of this judgment criterion is to assess the degree to which minimum board standards already exist across the CBDCs.

Page details

Date modified:: 2021-12-07

Evaluation of the Atlantic Canada Opportunities Agency Community-Based Business Development Sub-Program

Glossary of Acronyms

Table of Contents

List of Figures

List of Tables

Acknowledgements

Executive Summary

1. Introduction

1.1 Evaluation Overview

1.2 Evaluation Design and Methodology

1.3 Evaluation Strengths, Limitations and Mitigating Measures

2. Profile of the Community-Based Business Development Sub-program

2.1 Context

2.1.1 Community Futures Program

2.1.2 Other CBBD programming

2.2 Program Theory

2.2.1 Program Logic Model

2.3 Program Accountability and Governance

2.4 Expenditure Profile

2.4.1 CBBD Expenditures

2.4.2 Project Profile

3. Findings: Relevance

3.1 Continued Need for the Programming

3.2 Alignment with Government Priorities

3.3 Alignment with Federal Roles and Responsibilities

4. Findings: Performance – Effectiveness

4.1 Incrementality

4.2 Achievement of Expected Results

4.2.1 Immediate and Intermediate Outcomes

4.2.2 Ultimate Outcomes

4.3 Unintended Outcomes of CBBD Programming

4.4 Barriers to the Achievement of CBBD Outcomes

4.5 Adequacy of Performance Measurement

5. Findings: Performance – Efficiency and Economy

5.1 Efficient Utilization of Resources

5.1.1 Structures and Mechanism that Support Efficiency and Economy

5.1.2 Delivery Costs

5.2 Management of Loan Funds

5.2.1 CBDC Loan Approval Rate and Average Loan

5.2.2 CBDC Loan Loss Rates

5.2.3 Percentage of Investment Funds in Active Loans

5.3 Alternative Modes of Delivery, Lessons Learned and Best Practices

5.3.1 Alternative Modes of Delivery

5.3.2 Lessons Learned and Best Practices.

6. Conclusions and Recommendations

6.1 Conclusions

6.2 Recommendations

Appendix A – CBBD Evaluation Management Action Plan

Appendix B – Community Economic Development Conceptual/Analytical Framework

Appendix C – Evaluation Questions, Judgment Criteria and Methods

Appendix E – Community Futures Program Logic Model

Footnotes

Page details