Audit of the Implementation of the Aboriginal Skills and Employment Training Strategy, May 2014
Executive summary
The Aboriginal Skills and Employment Training Strategy (ASETS) is a contribution program designed to help increase Aboriginal participation in the labour market, ensuring that First Nations, Inuit and Métis people are engaged in sustainable, meaningful employment.ASETS provides funding to Aboriginal organizations to support the costs of labour market programs, services and other activities undertaken by these organizations.
ASETS is the successor strategy to the Aboriginal Human Resources Development Strategy (AHRDS). Three strategic pillars were introduced to support ASETS in :
- Demand-driven skills development;
- Partnerships; and
- Accountability for improved results.
Audit objective
The objective of this audit was to review the management framework of ASETS to ensure that appropriate controls were designed and are operating as intended.
Summary of key findings
The audit noted that appropriate controls have been designed for the management of ASETS. An adequate governance framework is in place. Further, requirements for the implementation and management of ASETS were well communicated to regional officers and recipients. However, the audit identified areas where the management framework and the application of controls could be strengthened. These include:
- Program risks were assessed but the risk assessment did not address pre-identified risks and did not include input from the regions.
- Controls used to Protected were not functioning as intended.
- Mechanisms for ensuring parties outside the Department are safeguarding personal information were not yet in place at the conclusion of the audit fieldwork.
Audit conclusion
The audit concluded that appropriate controls were designed to manage ASETS. However, these controls did not always operate as intended. Specifically, Protected controls Protected need to be consistently applied. As this program continues until , there are still opportunities to address these issues.
Recommendations
- Assistant Deputy Minister (ADM), Skills and Employment Branch (SEB), should update the Program Risk Assessment to include the issues raised in this report and other risks that may be identified by the regions and branches involved.
- Executive Heads, Service Management (EHSMs) should ensure that controls for Protected are applied consistently.
- EHSMs should ensure that controls for monitoring privacy requirements in contribution agreements are applied consistently.
1.0 Background
1.1 Context
ASETS is a contribution program designed to help increase Aboriginal participation in the labour market, ensuring that First Nations, Inuit and Métis people are engaged in sustainable, meaningful employment. ASETS provides funding to Aboriginal organizations to support the costs of labour market programs, services and other activities undertaken by these organizations.
ASETS succeeded the AHRDS in . The total program cost for ASETS is forecasted at $1.68B for the fiscal years – through – . Three strategic pillars were introduced to support ASETS:
- Demand-driven skills development to link intervention activities with local and regional employer demand;
- Partnerships with the private sector and the different levels of government to improve employment outcomes and use of resources; and
- Accountability for improved results to ensure that recipients can assess performance against targets and address accountability requirements.
While AHRDS and ASETS share similar aims, the transition to the new Strategy represented a fundamental change in how program activities would be planned, executed, reported and assessed. Under ASETS, the activities undertaken by recipient organizations were required to be implemented according to approved multi-year business plans that were to address the strategic pillars.
The Aboriginal Affairs Directorate, within SEB, has a lead role in the design of the delivery model for overall program and policy management. SEB also monitors performance and results, supports evaluation activities and ensures financial accountability.
The Aboriginal Program Operations Directorate, within Program Operations Branch (POB), provides functional direction and guidance to Service Canada regional offices regarding the operation of ASETS including financial monitoring, project risk assessment and results reporting. Regional responsibilities include the management of the approval process, claims and payments management, as well as activity and expenditure monitoring and reporting.
As of , there were 82 recipients Footnote 1 across the country. In many cases, services are delivered by third parties called “sub-agreement holders”. Recipients are responsible for monitoring sub-agreement holders to ensure they observe the provisions of the agreements. While the Department’s focus is on the activities of the recipients, the Department is expected to visit on-site with all sub-agreement holders at least once Department is expected to visit on-site with all sub-agreement holders at least once during the life of the master agreements.
1.2 Audit objective
The objective of this audit was to review the management framework of ASETS to ensure that appropriate controls were designed and are operating as intended.
1.3 Scope
The audit included the frameworks, control processes and systems designed to support the implementation and administration of ASETS.
Fieldwork took place in the Western-Territories Region and in the Ontario Region during the second year of the program. The sites selected were Vancouver, Edmonton and Toronto. The conduct phase, including work in National Headquarters (NHQ), took place from February through . The audit considered the controls that were in place between the beginning of the implementation and the conclusion of the site visits in .
1.4 Methodology
The following audit techniques were used:
- Interviews with SEB and POB management, as well as Aboriginal Programs management and staff in Vancouver, Edmonton and Toronto;
- Document review and analysis of Treasury Board and departmental policies, guidelines and processes; and
- Audit tests of recipient and sub-agreement holder documentation, from the sites visited, to assess the administration of the agreements with respect to the eligibility of expenses, monitoring and protection of personal information.
A random sample of 23 recipients was taken, representing half of the recipients managed at each of the three sites visited. These sampled agreements totalled $471 million, representing 31% of the total agreement value of $1.5B. Sampling was stratified so that recipients with larger agreements had a greater chance of being selected.
2.0 Audit findings
2.1 Program risk management practices can be strengthened
Sound risk management can drive an oversight framework relating to performance, third party management, financial monitoring, privacy and contribution agreement terms and conditions. Strong program risk management practices are particularly beneficial in complex delivery situations such as that of ASETS. This $1.5B program is undertaken by two branches in NHQ, all four regions, 82 recipients and hundreds of third parties.
Risk management for the ASETS program could have been more comprehensive. The regions were not involved in risk assessments and some risks which were identified at early stages of implementation were not considered when program risks were assessed Protected. For example, a risk relating to recipient performance data systems was noted in the – SEB Business Plan but was not included in the original assessment. The performance information issues raised later in this report (see section 2.2) may have been mitigated earlier had this risk been considered. In addition, risk assessments did not include Protected issues raised in previous internal audits of contribution programs which are similar to those raised in this report Protected.
Project level risks are assessed by regional officers who numerically assess ten predetermined risk factors for each project in the Risk Assessment, Management and Mitigation tool in order to establish the quantity of project monitoring. Standardized monitoring checklists have been developed for ASETS which are used when visiting onsite with recipients. The audit team found that these checklists address several areas of risk associated with the program, though they do not address certain program risks such as recipient privacy controls. These checklists could be more useful if they could be adapted to address key risks identified with a particular recipient. This could allow, for example, for specific recipient Protected issues to be specifically addressed during onsite visits. We believe that projects could be more effectively managed if the application of these checklists focused on mitigating risks particular to a project. By placing an emphasis on areas of greater risk to the program and particular recipients, the Department would be able to target scarce resources during regional monitoring activities.
Overall, risk management practices around ASETS can be strengthened. As the program does not expire until and planning for the ASETS renewal will begin in – , there remain opportunities to improve risk management.
Recommendation
ADM, SEB, should update the Program Risk Assessment to include the issues raised in this report and other risks that may be identified by the regions and branches involved.
Management response
Agree. SEB, in collaboration with the POB, the Chief Audit Executive, and other Branches, will update the program risk assessment, which will include considerations of risks identified in this audit as well as a review of risks identified in the original program risk assessment completed in . The updated Program Risk Assessment will be used to inform policy development and program delivery for all ESDC Aboriginal labour market programs. The estimated completion date is .
2.2 Governance framework is adequate
Direction
Overall, the departmental direction to regions and recipients for the implementation and management of ASETS was well communicated and sufficient. However, throughout testing during regional site visits, it was noted that several requirements outlined in the guidance documents were not consistently applied.
Departmental requirements included the Contribution Agreement, the Companion Manual, the ASETS Implementation Manual and Mid-Year Dialogue Guide. The key elements of these documents are noted below:
Contribution agreement
- The agreement outlines the expectations of how recipients are to manage the delivery of the program.
- Includes the ASETS Recipient Financial and Activity Monitoring Guide which clearly and concisely provides minimum requirements on how expense claims are to be validated, on-site reviews of documents are to be monitored, and how activity performance is to be analyzed. These procedures were to be applied by the Department towards recipients and by recipients towards their projects and third parties.
Companion manual
- Disseminated to both the regions and recipients, this manual clarifies the provisions of the ASETS agreements.
- Emphasizes the requirement that the Department approves sub-agreements between recipients and third parties.
- Emphasizes that the Department must be in a position to verify expenditures of funds until it reaches an individual.
ASETS implementation manual
- Advised regional officers on the information to be analyzed and discussed with recipients prior to the approval of recipient applications for ASETS.
- The information to analyze and discuss included past recipient audit findings and compliance issues, as well as clients’ past results versus targets.
Mid-year dialogue guide
- The objective of the mid-year dialogue process was to obtain performance information regarding the progress recipients have made in reaching their targets.
- Mid-year dialogue guides were developed for both regions and recipients on how to use the analysis of recipient annual reports demand driven results in discussions around performance information.
The audit team considered the direction sufficient for the implementation and management of ASETS. However, as previously noted, it was found that the direction provided in the above-mentioned documents was not consistently applied. Protected While regions and recipients indicated they would have appreciated that these requirements be communicated earlier in the implementation, it is the audit team’s opinion that the principal factor causing the inconsistency Protected
Performance
As “Accountability for improved results” is one of the three pillars of ASETS, the systems to report the performance results of clients against targets must be effective. Under the ASETS Performance Measurement Strategy, a “Standard Data File” (SDF) is used by recipients to report detailed information about their clients. The SDF is uploaded to the Department where it is processed to validate client and recipient results.
In an effort to address data collection and quality challenges, which were noted in an AHRDS evaluation, the SDF for ASETS was redesigned. The number of fields in the SDF was reduced from 88 to 36, resulting in a reduced reporting burden on recipients and fields were made mandatory to complete for each individual. However, during the ASETS audit period, the data collection and reporting challenges continued. Recipient and departmental software were not properly reconfigured, as a result, recipients were not able to transmit complete records of their client activity. It is important to note that management is aware of these issues and the audit team was informed that remedial action is being taken to rectify the situation.
In addition to the SDF, annual performance reports are submitted by recipients to demonstrate progress against annual operational plans. These reports are designed to enhance accountability and help senior management monitor the extent to which program objectives were being achieved. The audit team found that the design of the recipient annual performance reports allowed for a comparison of intervention activity results against annual operating plans. Protected
Protected However, in – , the Protected Region began requiring that these activities be reconcilable. The audit team considers this as a good practice.
As previously noted, a mid-year dialogue process was put in place. This process was designed to capture contextual planning and performance information in order for the Department to gain a comprehensive picture of recipient progress and challenges against the three ASETS pillars. The audit team considers the design of the mid-year dialogue process to be a best practice. Audit testing revealed that regional officers had used the mid-year dialogue “Highlight Notes” to successfully report meaningful recipient risk and administration information. The review of the Highlight Notes from the 23 sampled recipient files reported that 21 recipients were making progress and achieving their goals. Protected
At the conclusion of the fieldwork, the audit team was informed that the Department is working to resolve issues with software configuration and is developing tools to assist with the evaluation of recipient performance. We encourage the Department to continue its efforts in these areas in order to be able to report reliably on performance.
2.3 Protected
Protected
Recipient applications In order to participate in the Strategy, recipients were required to submit applications to the Department outlining the activities they would undertake in pursuing the three ASETS pillars.
New application requirements for recipients were introduced as part of ASETS. These included:
- Strategic Business Plans, describing recipients’ intentions over five years; and
- Annual Operating Plans, describing recipients’ intentions for the first 18 months.
The audit team found that the design of these two plans was appropriate as they compelled recipients to provide the Department with important information about how they would deliver the Strategy and improve results over the five years. Specifically, the Strategic Business Plans required recipients to outline not only their current capacity to deliver demand-driven interventions, but to also describe the enhancements they would be making to this capacity over the five year period.
Our audit tests showed that Annual Operating Plans addressed recipients’ goals for enhancing internal capacity and service delivery for the first 18 months. However, most of the Strategic Business Plans reviewed did not identify tangible service delivery or internal capacity enhancements which were to take place after the first 18 months. Protected
Protected
Protected
Monitoring of third parties
The audit team found that the Contribution Agreement terms and conditions, as well as the Recipient Financial and Activity Monitoring Guide, provide the foundation to allow for effective monitoring of the organizations that ultimately deliver the program. The Agreement and Guide require that recipients engaging third parties to deliver ASETS services are to:
- Provide formal “sub-agreements” between recipients and third parties to the Department for approval;
- Properly monitor the third parties; and
- Make available transaction level information about the expenses incurred by third parties.
In addition, the agreements contained provisions where on-site monitoring visits by departmental officers would occur at least once for each “sub-agreement holder” (third party) during the life of the master agreement with the recipient.
This design provides a sufficient basis for the Department to ensure:
- It is aware of the third parties delivering services as well as the terms and expectations under which they are operating;
- Recipients are detecting and correcting problems with their third parties; and
- Expenses incurred by recipients and their third parties are eligible for reimbursement.
Audit testing revealed that these requirements were not always being adhered to. For example, of the 23 recipients sampled, six were using third parties which had not been approved by the Department to deliver ASETS services, nor were sub-agreements available. The ASETS Funding Agreement Companion Guide, provided to recipients and regional officers, states that “if any funds do flow to a non-approved sub-agreement, these funds are considered non-eligible expenses”. Protected
The audit also found that monitoring visits by regional officers to third parties had not begun for 13 of the 15 recipients which use third parties. It is important to note these monitoring visits may be scheduled throughout the life of the agreement. However, the audit team observed Protected situations where recipients use Protected more third parties. In these situations, there is a risk that there may not be enough time to visit all third parties before the agreements expire on .
Protected
Based on the audit results, the team concluded that Protected controls in place at the time of the ASETS implementation were properly designed.
Protected
Recommendation
EHSMs should ensure that controls for Protected are applied consistently.
Management response
Agree. Regional management will undertake regular verification and provide oversight to ensure that existing internal controls are applied and will enhance Protected expertise assigned to ASETS agreements. The estimated completion date is .
2.4 Controls over privacy of information can be strengthened
Mechanisms for verifying that parties outside the Department are safeguarding personal information have not been fully implemented. As this multi-year program continues into , there are still opportunities to address these issues.
As part of ASETS program delivery, recipients and third parties collect personal information from individuals seeking to access ASETS services. Recipients and third parties also access personal information provided by Canada in order to administer the program and determine individuals’ eligibility to receive Employment Insurance (EI) benefits while participating in labour market programs.
The Department has signed contribution agreements with its recipients, which contain requirements for the exchange, collection and use of personal information. However, the Department does not have records that recipients have signed sub-agreements with third parties which contain the required privacy provisions. As noted previously under monitoring of third parties, six of the 23 recipients sampled used third parties which had not been approved by the Department to deliver ASETS services. Therefore, in these cases, the Department is not aware of the expectations placed on third parties for the protection of personal information. Furthermore, at the time of the audit, site visits to third parties had not yet been conducted. As a result, the Department was not aware if third parties have implemented adequate practices to protect personal information. Administration of the program requires recipient and third party provider employees to access EI information and transmit personal information about individuals.
ASETS contribution agreements require that a separate Service Level Agreement (SLA) between the Department and recipients formally govern their employees’ access to Canada’s data systems. The absence of these SLAs means that recipients are not bound to ensure that individuals employed by the recipient and third parties are abiding by the system and security requirements.
A Privacy Impact Assessment (PIA) had been drafted at the time ASETS was implemented. Since the conclusion of audit fieldwork, a draft ASETS PIA and SLA Action Plan has been prepared which, when implemented, should address the concerns noted above. The audit team has been informed that the draft PIA and SLA templates are being finalized and are expected to be presented to the Privacy and Information Technology Security Committee for consideration in .
Recommendation
EHSMs should ensure that controls for monitoring privacy requirements in contribution agreements are applied consistently.
Management response
Agree. Service Level Agreements will be implemented for all agreement holders and controls for monitoring privacy will be consistently applied and monitored. The estimated completion date is .
3.0 Conclusion
The audit concluded that appropriate controls were designed to manage ASETS. However, these controls did not always operate as intended. Specifically, Protected controls Protected need to be consistently applied. As this program continues until , there are still opportunities to address these issues.
4.0 Statement of assurance
In our professional judgement, sufficient and appropriate audit procedures were performed and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on observations and analyses at the time of our audit. The conclusions are applicable only for the assessment of integrated planning and risk management processes. The evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.
Appendix A: Audit criteria assessment
It is expected that:
1. A risk management strategy has been developed, at both the program level and the project level, to assess key risks and identify mitigating controls, actions and that the risk management strategy was communicated and acted upon for those key risks identified. (Section 2.1)
Conclusion: Can be strengthened
2.1. Policies, procedures and directives were developed for the management of ASETS and were communicated to managers, project officers, recipients and sub-agreement holders involved in the Strategy. (Section 2.2)
Conclusion: Adequate
2.2. Reporting systems are in place to provide management with timely, accurate and useful information for decision making and performance measurement. (Section 2.2)
a. Sub Criteria: Designed
Conclusion: Adequate
b. Sub Criteria: Operating as intended
Conclusion: Can be strengthened
3.1. Applications have been assessed to ensure that the proposals are complete and address the requirements of the Strategy. (Section 2.3)
a. Sub Criteria: Designed
Conclusion: Adequate
b. Sub Criteria: Operating as intended
Conclusion: Controls exist but not operating as intended
Protected
a. Sub Criteria: Protected
Conclusion: Protected
b. Sub Criteria: Protected
Conclusion: Protected
Protected
a. Sub Criteria: Protected
Conclusion: Protected
b. Sub Criteria: Protected
Conclusion: Protected
3.4. Risk based monitoring of payments, performance and mitigation strategies have been applied proactively. (Sections 2.1 and 2.3)
a. Sub Criteria: Designed
Conclusion: Can be strengthened
b. Sub Criteria: Operating as intended
Conclusion: Controls exist but not operating as intended
4. Mechanisms are in place to safeguard personal information as it is managed by the Department, recipients and sub-agreement holders. (Section 2.4)
a. Sub Criteria: Designed
Conclusion: Can be strengthened Footnote 2
b. Sub Criteria: Operating as intended
Conclusion: Controls exist but not fully implemented
Appendix B: Glossary
- ADM
- Assistant Deputy Minister
- AHRDS
- Aboriginal Human Resources Development Strategy
- ASETS
- Aboriginal Skills and Employment Training Strategy
- EHSM
- Executive Heads Service Management
- EI
- Employment Insurance
- NHQ
- National Headquarters
- PIA
- Privacy Impact Assessment
- POB
- Program Operations Branch
- SDF
- Standard Data File
- SEB
- Skills and Employment Branch
- SLA
- Service Level Agreement
Page details
- Date modified: