Audit of Information Supporting Governance Committees
Table of contents
Executive summary
In response to both the Government’s Digital Operations and Strategic Plan 2018 to 2022, and the Data Strategy Roadmap, Shared Services Canada (SSC) developed and published its own Data Strategy 2019 to 2021. This strategy identifies critical issues pertaining to data within the organisation and aligns them to their related Government of Canada reporting requirement. To support key strategic oversight committees, a sound decision-making process requires complete, reliable, and timely information.
This audit aimed to determine whether control processes, governance, and risk management are in place, efficient and effective to provide information in a complete, accurate, and timely manner to the senior management committees.
The following areas were examined:
- Governance: Does SSC have oversight bodies related to the management of data; are authority, responsibilities, and accountability with regards to data stewardship and ownership defined; and, is information presented at senior executive committees provided in a timely manner?
- Risk Management: Has management identified and documented its risks related to data and information to ensure accurate and complete strategic reporting?
- Internal Controls: Do SSC’s oversight bodies request and receive sufficient, complete, and accurate information; is information reporting reviewed and approved by Branches prior to submission to senior management committees?
The audit found the following:
- In-scope governance committees (that is EOB, PMB, SPPRB) did not have a defined role in providing data oversight. The Data and Business Analytics Council (DBAC) is not a recognized body within SSC’s governance structure
- The organisational risk profile includes a mitigation action plan that identifies information and data related risks, internal control measures, and corresponding corrective actions; it does not, however, identify an accountable “Office of Primary Interest”
- SSC does not have policies that address information quality assurance and related challenge functions to ensure the integrity and accuracy of the data presented to governance committees
- There is a varying degree of quality assurance and data accuracy at the Branches and Business Lines level and the burden is placed on data stewards, rather than data owners, to ensure data accuracy
- In-scope centralized quality assurance activities, such as the Project Management Centre of Excellence (PMCoE) and the Quality Secretariat for the Finance, Investment Management Board (FIMB), served to ensure their respective oversight committees – Project Management Board (PMB) and FIMB – that presentations are of a sound quality
- There are many key data sources used to build presentations with limited access controls. Structured databases, workbooks and other alternative/discovery data, provide a significant challenge to track information included in presentations to their data sources. Errors were often “fixed on the fly, but not at the source”, an effective process at a time and point, but not efficient over time
Begonia Lojk
Chief Audit and Evaluation Executive
A. Introduction
1. Background
In January 2018, a team from Statistics Canada, Treasury Board Secretariat and the Privy Council Office produced a Report for the Clerk of the Privy Council entitled “A Data Strategy Roadmap for the Federal Public Service”. This roadmap was intended to set a foundation for government to create more value for Canadians from the data it holds while ensuring the privacy and protection of personal information.
In March, 2019, Treasury Board Secretariat released the (third iteration) Digital Operations Strategic Plan 2018-2022, which details the Government of Canada’s digital vision, standards and strategic planning process for how technology and technological change are managed.
In response to both the Digital Operations and Strategic Plan, and the Data Strategy Roadmap, SSC has developed and published its own Data Strategy 2019 to 2021. This strategy identifies critical issues pertaining to data within the organisation and aligns them to related Government of Canada reporting requirements, and to strategies intended to achieve specific key results and larger, overall strategic outcomes. The SSC Data Strategy will guide the implementation of concrete actions on how data is created, protected, used, managed, and shared, in support of decision-making and day-to-day operations.
2. Rationale for the audit
During the development of the risk based audit plan (2019 to 2022) concerns were noted regarding the complex origins of data from untrusted sources, roles and responsibilities around the stewardship of information, and process inefficiencies.
Having complete, reliable, and timely information is critical to a sound decision-making process. Key strategic management committees require accurate and timely information to support decision making. Inadequate information for decision making can lead to suboptimal planning, resource allocation, and performance management decisions.
3. Audit authority
The audit of Information Supporting Governance Committees was approved by the President, following recommendation by the Departmental Audit Committee of the 2019 to 2022 Risk-Based Audit Plan.
4. Objective of the audit
The objective of the audit was to determine whether control processes, governance, and risk management were in place, efficient and effective to provide information in a complete, accurate, and timely manner to the senior management committees.
5. Scope
Given the impact and importance of the decisions made by the Department's strategic management committees, the process to support decision making at these committees was identified as a high audit priority. The scope included the Senior Management Board (SMB) (now called Executive Oversight Board, EOB), the SPPRB.Footnote 1, Project Management Board (PMB), and the Data Business Analytics Council (DBAC).
The scope includes all processes, procedures, systems and tools used to control the management of information provided to governance committees for decision-making, and related quality assurance activities. It covered the period of September 2018 to November 2019, inclusive.
6. Methodology
To conduct the examination, a variety of testing methods were used. This included multiple interviews, and the review of 18 selected presentations provided to the in-scope oversight committees. Each presentation deck was the subject of tests to determine the number of key data sources, the accuracy of data, the quality assurance and challenge performed, and their completeness and timeliness.
7. Statement of conformance
In my professional judgment as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The opinion is applicable only to the entity examined. The engagement was conducted in conformance to the requirements of the Policy on Internal Audit, its associated directive, Internal Audit Standards of the International Professional Practices Framework and Code of Ethics. The evidence was gathered in compliance with the procedures and practices that meet the auditing standards, as corroborated by the results of the quality assurance and improvement program. The evidence gathered was sufficient to provide senior management with proof of the opinion derived from the internal audit.
B. Findings, recommendations and management response
1. Governance, Roles, and Responsibilities
1.1 Oversight bodies
Audit criteria: Oversight bodies related to the management of data have been established and communicated.
It was expected that Terms of Reference (ToR) documents for the selected committees – that is SMB (now EOB), PMB, SPPRB, and DBAC would clearly articulate their role with regards to data oversight and stewardship and that this role would have been communicated across the department.
A detailed analysis of the ToRs in relation to the roles of data oversight and data stewardship showed that none of the governance committees reviewed had a direct role in providing data oversight or stewardship acknowledgement and direction. Specific linkages between the information to the owners or stewards of the content presented were generally unclear.
In January 2020, SSC adopted a new governance model in an effort to better guide decision-making and oversight efforts over key projects and activities. In response to these new developments, some boards included in the scope of the audit were changed or split, resulting in revisions to their ToR to better steer the Department towards the goals of the new governance model. The updated TORs do not have anything related to committees’ role with regards to data oversight and stewardship.
It should be noted that DBAC was formed in 2019 to establish data management, business analytics expertise and culture at SSC. One of its priorities is to ensure implementation of SSC’s data strategy and better communicate the roles of data steward and data owner across the Department. While the roles of the oversight committees are communicated to the department on the Intranet, the role(s) of DBAC is not formally identified in SSC’s current governance structure. It was also noted that the ToR for DBAC outlined oversight over data management as a Branch responsibility and did not identify reporting relationships to any other SSC governance committee.
Overall, none of the in-scope governance committees TORs identify a clear committee role in providing data oversight or stewardship, and DBAC’s oversight role over data management is not identified as part of SSC’s governance structure.
Recommendation 1
Medium priority
The Senior Assistant Deputy Minister (SADM), Strategy and Engagement Branch should ensure that:
- The EOB, PMB, SPPRB Terms of References clearly identify their roles in providing data oversight or stewardship.
- The role of the Data Business Analytics Council is recognized within SSC’s corporate governance structure and communicated on the intranet or as appropriate.
Management response
Management agrees with the recommendation to include data oversight and stewardship in the Terms of Reference and include the Data Business Analytics Council in the formal governance structure.
1.2 Authorities, responsibilities and accountabilities
Audit criteria: Responsibilities and performance expectations related to data management are known, clearly defined, and communicated.
It was expected that responsibilities, and performance expectations pertaining to data management were clearly defined and communicated across the Department.
The audit team interviewed multiple stakeholders and reviewed pertinent relevant policies, directives, and document guidance. Multiple interviewees expressed their concerns vis-a-vis the significant challenges of data management and quality assurance. They also mentioned that roles and responsibilities related to data management, data stewardship and/or data ownership.Footnote 2 were not clearly defined.
While SSC’s data strategy is available across the Department, interviewees conceded that the responsibilities and performance expectations related to data stewards, data owners and the inherent data quality assurance were not well defined within the Department; this places the burden on data stewards to validate the accuracy of data and ensure data integrity. While data validation is being done effectively by data stewards on a case by case presentation, the approach is inefficient, subject to error, and depends solely on the data steward time and ability to review the presentation.
The audit found some information on who has ownership over some data through publications such as:
- the SSC Project Management and Delivery Operating Guide indicates that Project Managers are data stewards for the Enterprise Portfolio System
- the SSC Project Management Directive also indicates that projects are required to provide up-to-date data related to project progress, risks, issues and changes to approve baseline requirements
Nonetheless, The audit team was unable to find any departmental documentation - for example a departmental policy, directive, or guideline, which defines roles and responsibilities pertaining to data management; including the definition of “data stewardship and ownership”.
Overall, the audit found that responsibilities related to data stewardship and ownership were not clearly defined, communicated and fully understood, and that there were no available policies, procedures or guidelines which specifically define “data stewardship and ownership”.
Recommendation 2
Medium priority
The ADM Corporate Services, through the CIO should ensure that data management responsibilities and performance expectations outlining key definitions of data stewardship, and data ownership are clearly defined and communicated appropriately.
Management response
Management agrees with the recommendation. SSC has implemented a governance for data management, business analytics and is fostering a culture of strong data stewardship. Ensuring that SSC’s governance is clear and well understood is essential to achieving the principles of the consistent application of data within the department and the need to use authoritative data sources in evidence-based decision making. The governance for data management is led by the Data Business Analytics Council (DBAC). It is co-chaired by the Corporate Services Branch and the Operations Management Branch which helps to ensure there is a strong integration of data management practices across the department. There is an approved ToR that clearly outlines its mandate, roles and responsibilities. Supporting the DBAC, there is the Data and Analytics Centre of Excellence (DACoE) which is responsible for implementing SSC’s Data Management Strategy and providing business analytics expertise. Within this governance, Data Management Teams work with a network of Data Stewards, who represent key stakeholders across the department, with a view to exchanging information and leveraging best practices. A ToR for the Data Stewards Community of Practice (DSCoP) was approved by DBAC in May 2020 and is reviewed annually. The mandate of the DSCoP is to provide a forum at the working level within the department to exchange ideas, provide insight on issues, provide a challenge function, and leverage best practices across the department.
1.3 Information to Executive Committees is timely
Audit criteria: Information is provided to the members of the oversight bodies in advance of the scheduled meeting date to allow sufficient time for review.
The audit expected to find presentation decks, reports and/or other attachments would be provided to in-scope oversight boards in advance of meetings to allow sufficient time for review.
To assess whether information presented at senior executive committees was timely, the auditors examined a sample of 1 months' presentations from each in-scope committees managed by the ECS. In total 35 presentations and 51 associated documents were examined. The test consisted of determining if information was provided to the ECS on time and subsequently distributed by the secretariat to committee members on time.
Based on the evidence gathered and results of the tests conducted, the audit found that ECS did not receive the information early enough to allow the QA process to be performed before submitting the presentations to the committee members. Notwithstanding this, information was being provided to senior executive committees sufficiently in advance of meetings to allow for committee members to review its content. For example:
- it was found that 31 out of 51 (61%) associated presentation documents were submitted to the ECS by the required due date
- it was found that 31 out of 35 (89%) presentations were mailed out to committee members for review two days prior to the committee meeting date
- of the 20 late documents, a due date extension was requested for only one (1) document
Furthermore, DBAC’s Co-Chairs confirmed that information was being provided to DBAC in a timely manner, allowing sufficient time to review information, documents and reports provided. The PMB Chair also received documents in advance of meetings, and relied on an in-depth review of documents, and artefacts, supported by quality assurance conducted by the PMCoE.
Based on the evidence gathered and the results of the tests conducted, the audit team concludes that overall information is being presented to senior executive committees in a timely manner.
2. Risk management
2.1 Risk and data management
Audit criteria: Management has identified, assessed, mitigated and documented its risks related to data sources and information to ensure accurate and complete strategic reporting.
The audit expected to find that management identified and documented its risks related to data and information to ensure accurate and complete reporting. Specifically, the risk management process for the management of data should include critical elements such as: SSC’s objectives and outcomes, risks identification and assessment, management responses, and risks monitoring.
The audit found that SSC has an established Organisational Risk Profile (ORP). The ORP is provided to SSC’s senior management via SMB (now EOB), where potential risks and issues are reported, and discussed. The ORP reported the quality of information as a high risk. In addition, the ORP identified the risk related to data and quality of information, its corresponding internal controls, and mitigation action plan. However, the mitigation action plan did not identify an “Office of Primary Interest” accountable to ensure it is fully implemented, reported and monitored.
Recommendation 3
Medium priority
The SADM, Strategy and Engagement Branch should ensure that the Corporate Risk Profile designates an “Office of Primary Interest” accountable for managing risk related data and information, which will also be accountable for the risk response strategies and for periodically reporting to EOB.
Management response
Management agrees with the recommendation. Clarifying accountabilities and appropriate senior management oversight is an important aspect of an integrated Corporate Risk Profile (CRP).
The next iteration of the CRP is currently in development and will seek to align operational and corporate risk management data information and corresponding roles. This includes identifying accountabilities associated with managing risk related data and information, as well as risk response strategies.
3. Internal controls
3.1 Information quality
Audit criteria: Oversight Bodies Receive Complete, and Accurate Information
The audit expected to find that quality assurance (QA) processes are in place to ensure that information presented via decks, reports, and other presentation aids is sufficient, complete, accurate, and supported with key data sources and subject to sound QA processes.
3.1.1 Quality assurance
To assess data accuracy and QA, the audit team reverse-engineered 18 presentation decks back to their original key data sources, and reconstructed them. The audit results highlight that SSC has a two tier system related to QA:
First tier
The first level of QA is conducted at the Branch level. The audit results indicate that Branch level QAs vary considerably, and inconsistencies exist between SSC branches and Business Lines in terms of the depth of the QA and the validation conducted before the presentations are sent to centralized functions. There is minimum challenge offered since the QA mostly consists of an exchange of e-mails or telephone calls to confirm or correct data and the interpretation made of it - the story line.
Second tier
The second level of quality assurance is provided by centralized quality assurance teams and the ECS. While ECS conduct a formatting check on presentations provided to senior executive to ensure it meets the corporate standards, it does not perform a data accuracy and interpretation of data - the story line - prior to presentations being made available to executive committees, unless there is a glaring discrepancy identified.
On the other hand, the interview results, the analysis of RoDs, and action trackers indicate that presentations made at senior executive committees were reviewed by the PMCoE which serves as a centralized quality assurance function for PMB; a last check point to make presentations available to PMB. The function ensures the presentation decks use the correct tool kits available, and goes into further quality assurance processes such as validating data, identifying discrepancies, ensuring tombstone data is communicated correctly and all artifacts are complete, current, and timely, according to the appropriate gate and approved. For example, for project related presentations, all draft reports are cleared via stakeholder sessions which provide all participants with an opportunity to re-check their own input and approve the overall data that has been compiled. To mitigate this risk, the PMCoE, project managers, Branches, and Business Lines developed a strategy which consist of using the Task and Financial Authorizations to estimate as close as possible capacity to support IT projects, whether SSC or partner-led.
There were, however, discrepancies found between data sources and final presentations which can be attributed to corrections that were made “on the fly”, but not corrected at the source. This business practice increases the risk of introducing perpetual errors in data sets, but more importantly, an error may find its way into a revolving, or single, presentation deck, and may lead to an incorrect decision.
Although, SSC does not have policies that address information quality assurance and related challenge functions to ensure the integrity and accuracy of the data presented to governance committees, committee secretariat and oversight bodies have put in place mechanisms to ensure presentations made to executive committees, include appropriate and complete information.
Overall, for the information provided to the in-scope committees, there is minimum challenge function provided at the branch and business line levels. It is left to PMCoE and ECS to provide challenges, with the caveat that ECS does not perform a data accuracy or interpretation of data, rather they ensure formatting and structure meet SSC standards. Managers are often left with fixing data “on the fly” rather than fixing at the source, with the risk of carrying a data gap, or discrepancy. The lack of policies to address data accuracy and quality assurance from an organisational perspective compounds the institutional risk related to quality of information; as included in the ORP.
Recommendation 4
Medium priority
The ADM, Strategy and Engagement Branch should develop and promulgate a standardized quality assurance approach, including formal approvals across Branches and Business Lines when submitting information to governance committees.
Management response
Management agrees with the need to develop and implement a standardized quality assurance approach across Branches and Business Lines when submitting information to governance committees. This approach will identify the appropriate data sources for various types of data. It will continue to be the responsibility of the presenting ADM to ensure that the data is timely and accurate, using the approved source.
3.1.2 Data sources
The responsibility to develop decks and reports often remains with first line executives who will gather data from multiple sources of information. The key data sources most often used to build presentation decks includes People Soft, SIGMA, Salary Forecast Tool, Tasks and Financial Authorizations, other structured data bases, and personal workbooks with fragile access controls, reports, other alternative ledgers and discovery data from Branches and Business Lines.
Most managers tasked with developing decks and reports for senior executive committees develop and maintain workbooks for their own use, and to have readily available historical and current data. The reason cited is focused on official systems lagging on current data. For example:
- the latest invoices from suppliers may not be entered into the official systems, but are available to project managers who will capture the information in a workbook
- the most significant challenge for senior managers and managers was to determine organisational capacity. PeopleSoft and the Salary Forecast Tool are not compatible, the Public Service tracks employee, but it does not have the most up-to-date information, leading Branches to develop workbooks and unofficial organisational charts to track employee and determine capacity available for SSC led and/or partner projects
The testing of decks demonstrates that depending on the subject of the presentation, data sources are not always available. Presentation decks are mostly void of direct links to key data sources, leaving the recipient of the information without the ability to link the key data sources with the information presented, to ensure quality, and build a high level of trust into the process. For example, planning reports and presentations to a strategic committee such as SPPRB were often in draft form, with statements of acknowledgement of facts but without sources of information. Even the more data-dependent presentations on specific projects or planned results lacked data source information and could not be tracked to the original data sources. It was also difficult to track from supporting drafts (produced during the challenge process) to the final draft placed before the committee for consideration despite the abundant challenge; there is no way to completely validate presentation statements or data.
By contrast, project specific presentations to PMB often contained sources of data, and were easier to track. For instance, the Task and Financial Authorizations included capacity estimates, were approved and signed and could be followed when reverse engineering the presentation decks to key sources of data.
Multiple key data sources, including workbooks with fragile access controls, and discovery data from Branches and Business Lines, are used to build presentation decks. These workbooks have one password and may be accessed by numerous users. Access to workbooks by multiple users increases risks of data errors, while promoting an environment void of a single source of truth.
Recommendation 5
High priority
The SADM, Corporate Services in collaboration with the Data Business Analytics Council should streamline the number and volume of workbooks to reduce alternative data sources.
Management response
Management agrees with the recommendation and recognizes the importance of reducing the reliance on workbooks as well as the use of alternative data sources, and replacing these individual practices with established data management principles and authoritative data sources. SSC is making significant progress in these areas. The department has an approved Data Management Strategy and has a sound governance structure, supported by a network of data stewards and analysts which aim to foster a stronger data and analytics culture. Within this governance framework, a number of initiatives have been developed, or are underway, which contribute towards maturing SSC’s data management practices. Some of these activities include:
- The establishment of an Enterprise Data Repository (EDR) in January 2020, providing a validated source of enterprise data for users across the department
- The development of a catalogue of data issues
- The development of a standardized integrated reporting system (where applicable) to eliminate individual workbooks and to breakdown organizational silos; and
- The development of a business glossary.
- The development of a Data Quality Framework which assesses known data issues and provides guidance and strategies to address them in the future.
3.2 Information reporting is approved
Audit criteria: Information reporting is reviewed for completeness, accuracy, relevance, timeliness, appropriateness, reasonableness, and approved.
The audit expected to find that information reporting approval is evidenced, and control breakdowns are reported to management.
In general, prior to the presentation date, Branch level managers and ADM offices will request data sources from data owners to build the presentation deck along with accompanying reports when necessary. Branch managers will review data to ensure its accuracy, and that the story line is representative of the data sets. Any apparent deviation, or outliers, will be further examined with data owners to ensure its accuracy. The presentation will then be forwarded to the ECS, who will review it to ensure it meets formatting and structure guidelines, however it does not challenge the deck owner, or Branch senior manager, about the content unless it’s an obvious error or the information is not logical. After the review is conducted, the presentation is included in the agenda of the executive committee and distributed to its members.
Multiple evidence such as signoffs is available to support that Branch managers approved the presentation and the information within, and sign off is obtained via e-mail through the different parties such as ADM Offices.
Overall, approvals are conducted at the Branch level prior to presentation at executive committees. At a minimum, completeness of presentation is managed over time, and data accuracy issues and incorrect data interpretation are managed centrally through QA functions such as, the PMCoE, the QA secretariat for FIMB and ECS.
C. Conclusion
In my opinion, as the Chief Audit Executive, SSC generally has effective data and information controls and risk management practices to support oversight bodies, but has inefficient and onerous quality assurance mechanisms.
Control weaknesses were identified, particularly in ensuring data accuracy. Also, risk management mechanisms to ensure corrective action plans are fully implemented, were found to be lacking clear accountabilities.
We found:
- In-scope governance committees (that is EOB, PMB, SPPRB) did not have a defined role in providing data oversight. The Data and Business Analytics Council (DBAC) is not a recognized body within SSC’s governance structure as published in SSC’s intranet site
- The organisational risk profile includes a mitigation action plan that identifies information and data related risks, internal control measures, and corresponding corrective actions, but it does not identify an accountable “Office of Primary Interest”
- SSC does not have policies that address information quality assurance and related challenge functions to ensure the integrity and accuracy of the data presented to governance committees
- There is a varying degree of quality assurance and data accuracy at the Branch and Business Line levels and the burden is placed on data stewards, rather than data owners, to ensure data accuracy
- In-scope centralized quality assurance activities, such as the Project Management Centre of Excellence (PMCoE) and the Quality Secretariat for the Finance, Investment Management Board (FIMB), served to ensure their respective oversight committees – Project Management Board (PMB) and FIMB – received presentations that were of sound quality
- There are many key data sources used to build presentations with limited access controls. Structured databases, workbooks and other alternative/discovery data, provide a significant challenge to track information included in presentations to their data sources. Errors were often fixed “on the fly”, but not at the “source”, which is effective but not efficient over time
These findings are important because they will pose challenges to a successful and efficient organisational data systems and present a burden to provide quality information to SSC’s oversight committees.
Annex A – Specific Lines of Enquiry and Audit Criteria
| Audit Criteria | Criteria Description |
|---|---|
| Line of Inquiry 1: Governance Structure | |
| 1.1 Governance of data management | Oversight bodies related to the management of data have been established and communicated |
| 1.2 Data stewardship and ownership | Authority, Responsibilities, and accountability with regards to data stewardship and ownership are well defined, communicated |
| 1.3 Timeliness | Information presented at senior executive committees is provided in a timely manner |
| Line of Inquiry 2: Risk Management | |
| 2.1 Risk framework | Management has identified and documented its risks related to data and information to ensure accurate and complete strategic reporting. |
| Line of Inquiry 3: Internal Controls | |
| 3.1 Quality of information | Oversight bodies request and receive sufficient, complete, timely and accurate information |
| 3.2 Approval of information reporting | Information reporting is reviewed and approved by Branches prior to submission to senior management committees. |
Annex B – Audit recommendations prioritization
Internal engagement recommendations are assigned a rating by OAE in terms of recommended priority for management to address. The rating reflects the risk exposure attributed to the audit observation(s) and underlying condition(s) covered by the recommendation along with organisational context.
| Rating | Explanation |
|---|---|
| High priority |
|
| Medium priority |
|
| Low priority |
|
Page details
- Date modified: