Internal Audit – Oversight, Use, and Continuous Improvement of Business Intelligence

Final Report
Audit, Evaluation, and Risk Branch
Audit Committee
June 18, 2024

Executive Summary

The Canada Revenue Agency (CRA) creates and collects taxpayer information and has millions of taxpayer data records at its disposal. These records are used for several purposes, including developing business intelligence to support the CRA’s compliance, collections and verification activities. Business intelligence refers to information derived from the data available to an organization to provide knowledge and insights to support decision-making and program administration.

The audit objective was to provide the Commissioner, CRA management, and the Board of Management (Board) with assurance that business intelligence is being overseen and used, and continuously improved by the CRA to support its compliance, collections and verification programs.

Overall, the audit concluded that important components of the business intelligence framework are in place at the CRA; however, some improvements are needed to strengthen it. Below are the findings of the audit:

• The CRA has a documented governance framework for the management of data and acquisition of tools for the development of business intelligence; however, the audit found the following actions are required: establish CRA-wide objectives for business intelligence; clarify roles and responsibilities; and determine a common definition of what constitutes business intelligence.
• The CRA Business Intelligence Strategy has evolved to focus on the data governance, and acquisition of tools, but it is missing strategies to leverage and coordinate business intelligence horizontally.
• Working groups are in place to foster and advance continuous improvement in CRA business intelligence work. However limited horizontal collaborations were observed.
• There are no CRA expectations for developing business intelligence, and inconsistent continuous improvement processes were identified. This has led to instances where potentially outdated business intelligence was not reviewed and updated for multiple years.
• A process including an oversight body is in place to ensure an enterprise view has been considered when acquiring new business intelligence tools. However, there is no formal process to support the effective and timely deployment of the tools to the numerous business intelligence teams throughout the Agency.

These findings suggest that improvements are needed to strengthen the CRA’s ability to leverage its business intelligence capabilities to better support informed programs and operational decision-making.

Summary of recommendations

The Service, Innovation and Integration Branch (SIIB), in collaboration with the Information Technology Branch (ITB) and other stakeholders, should strengthen the governance framework for business intelligence by addressing the following:

• Establish CRA-wide objectives for business intelligence, including clear roles and responsibilities and a common definition of what constitutes business intelligence
• Develop a strategy to coordinate, leverage and monitor business intelligence efforts horizontally to ensure business intelligence meets program decision-making needs and supports the CRA Strategic Priorities
• Establish expectations for developing business intelligence, including continuous improvement requirements, to ensure the CRA leverages its business intelligence and continues to improve it
• Develop an effective deployment process for the timely delivery of business intelligence tools

Management response

The SIIB agrees with the recommendations in this report and has developed related action plans. The Audit, Evaluation, and Risk Branch (AERB) has determined the action plans appear reasonable to address the recommendations.

Introduction

The CRA’s mandate is to administer tax, benefits, and related programs, and to ensure compliance on behalf of governments across Canada, thereby contributing to the ongoing economic and social well-being of Canadians. Daily, the CRA creates and collects taxpayer information, and has millions of taxpayer data records at its disposal. These records are used for several purposes, including to support the CRA’s compliance, collections and verification activities.

Business intelligence refers to information derived from the data available to an organization to provide knowledge and insights to support decision-making and program administration. Business intelligence can include a wide variety of practices, such as reporting, analytics, and research to obtain actionable information. For instance, data can be converted into business intelligence to detect possible tax havens or to inform the CRA about taxpayer debt and possible fraud related to benefit payments. Business intelligence at the CRA can be seen as project-based work, with project themes being determined based on branch or regional needs. It can also be used by the CRA to inform the day-to-day branch and regional workloads in compliance, collections and verification programs.

Business intelligence activities have been under the responsibility of the SIIB since 2011, with service delivery provided by the ITB. Through its core mandate, the SIIB aims to:

• maximize the CRA’s ability to use its information assets, including both structured data and unstructured information, to the benefit of Canadians
• lead cross-cutting CRA projects in the interest of an integrated enterprise approach and results
• support informed internal decision-making on the key questions the CRA faces in its operations and strategic priorities

The Chief Data Officer (CDO) is also the Assistant Commissioner of the SIIB. The mandate of the Chief Data Officer is to provide leadership and strategic oversight to leverage data as an Agency-wide asset. Effective data management and governance will ensure the delivery of quality and reliable data to business intelligence users and Canadians.

The ITB, on the other hand, operates, maintains, develops, and evolves the CRA’s information technology at the core of Canada’s tax and benefit administration. It also provides technical support to the business intelligence self-service systems for users to access the data required to produce the relevant business intelligence. The Branch provides data management services, such as data modelling, data delivery, data mining and data infrastructure services, as well as business intelligence services, including promoting the business intelligence agenda within the CRA.

Although the CRA has been developing business intelligence for a long time, the demand for it has grown since 2011. To meet this increased demand, CRA branches and regions have developed their own business intelligence capacities. This means business intelligence has been undertaken and delivered in a decentralized manner throughout the CRA.

In 2020, the CRA introduced the CRA Information and Data Strategy, which supports the management of data and information assets through four pillars: governance, people and culture, environment and infrastructure, and supporting innovation and transformation. This strategy complements the Government of Canada’s priorities set out in the 2023-2026 Data Strategy for the Federal Public Service and its vision to strengthen data-driven results and outcomes, drive the federal public service to consider data by design, and effectively use data in decision-making.

Focus of the audit

This internal audit was included in the 2022-2023 Risk-Based Assurance and Advisory Plan (Mid-Year Update), which was approved by the Board of Management (Board) on March 30, 2022. The Assignment Planning Memorandum was approved by the Commissioner on September 11, 2023.

Importance

This audit is important because the CRA’s business intelligence capacity has expanded rapidly in recent years. The CRA has committed to optimizing the use of data, to facilitating data sharing and collaboration, to increase its usability and reusability, and to supporting evidence-based decision-making.

Objective

The audit objective was to provide the Commissioner, CRA management, and the Board with assurance that business intelligence is being overseen and used, and continuously improved by the CRA to support its compliance, collections and verification programs.

Scope

Consultations with management were also held to help determine the scope of the audit. As a result, it was determined this audit would focus on the governance structures, uses, and continuous improvement of business intelligence at the CRA.

The scope of this audit did not include an examination of data security, human resource challenges, applications, tools, and infrastructure, nor their lifecycle used to develop business intelligence.^{Footnote 1}

The period covered in this audit was from April 1, 2020, to March 31, 2023.

Audit criteria and methodology

The audit criteria and methodology can be found in Appendix A.

The examination phase of the audit took place from June 2023 to November 2023.

This audit was conducted in accordance with the International Standards for the Professional Practice of Internal Auditing, as supported by the results of the quality assurance and improvement program.

Findings, recommendations, and action plans

The SIIB agrees with the recommendations in this report and has developed related action plans. The AERB determined the action plans appear reasonable to address the recommendations.

Business intelligence oversight, structure and strategy

Governance frameworks are necessary to ensure effective processes are in place to properly administer programs. The audit expected the governance framework for business intelligence to include the following components:

• data governance
• business areas responsible for business intelligence
• enterprise and branch/regional business intelligence strategy
• roles, responsibilities, and accountability
• levels of approval required for decision-making
• monitoring and reporting
• leadership to support the advancement of business intelligence across the Agency

The internal audit team reviewed the documented roles and responsibilities, observed governance committee meetings, and conducted interviews with the various business intelligence teams and representatives. The team found that the SIIB generally acts as a facilitator while the other branches are individually responsible for their business intelligence.

The governance committee tasked with the governance of business intelligence is called the Business Intelligence Director General Steering Committee (BIDGSC) and is co-chaired by the SIIB and the ITB. This committee has been providing a platform for director generals to communicate and endorse business intelligence tools requirements and priorities, and discuss data governance. However, the BIDGSC does not have the authority to provide oversight and direction on the development and leveraging of business intelligence in the programs as that falls under the authority of each individual branch within the CRA. As such, the audit found that branches had developed varying definitions of “business intelligence” and, therefore, different understandings of what information should be considered “business intelligence.”

The absence of a clear responsible lead over business intelligence in the Agency and the lack of a consistent understanding of what business intelligence is increasing the risk that business intelligence may not be appropriately managed in terms of ensuring consistent quality and overseeing access to the information.

A formal documented strategy is important to set the direction for business intelligence, as well as its priorities. The CRA’s Information and Data Strategy, which was in place during the audit, was an evolution from the Business Intelligence Strategy of 2014. But unlike the 2014 strategy, Information and Data Strategy did not include dedicated strategies for business intelligence development, such as projects aimed at advancing the integration of business intelligence into CRA decision-making.

In the absence of a dedicated strategy, the internal audit reviewed and observed the activities of the BIDGSC. As per its terms of reference, the BIDGSC aims to ensure the strategic alignment of the business intelligence program with the Agency strategic plans, priorities and business drivers. However, these activities were not observed during the audit. The BIDGSC largely focused on components required to develop Business Intelligence, such as data and tools.

Furthermore, although there are research and artificial intelligence repositories available, the team did not observe an established repository that captures CRA-wide business intelligence to help track and monitor ongoing business intelligence efforts. Without a method to document business intelligence efforts, there is an elevated risk of duplication of efforts in developing business intelligence and a limited ability to leverage business intelligence horizontally. 

Without a strategy to help coordinate and leverage business intelligence horizontally, the Agency may be omitting important untapped business intelligence potential that could lead to better informed decisions that can help the CRA achieve its strategic objectives, especially combatting aggressive tax planning.

Working groups

Working groups brings together business intelligence developers from all teams in the CRA to enhance horizontal collaboration and networking with peers. This collaborative work is important for developers as they share knowledge and expertise and work together toward common objectives, to ultimately help improve business intelligence in the CRA.

The internal audit reviewed the objectives of working groups related to business intelligence. The audit also observed working group sessions and interviewed a sample of business intelligence teams to review if the working groups are adding value to CRA business intelligence.

Developers have the opportunity to share and communicate information, by participating in the working groups and their online platforms. The majority of the working group meetings observed consisted of presentations about work completed by the different teams and of expertise sharing regarding the use of certain tools. While these collaborative efforts are helpful, the audit noted that the business intelligence developers are still mostly working in silos, as there are very limited joint business intelligence projects occurring or emerging in the working groups.

The majority of the sample of business intelligence teams developing workloads did not participate in CRA-wide working groups. Instead, they took part in working groups that were limited to their program or branch, therefore, collaborating only within their program. The inconsistent definition of “business intelligence” across the CRA (as discussed on 3.1.1) led some teams to perceive their work as not fitting into the mandates of the business intelligence working groups. This demonstrates that, while there were collaborations within certain programs, opportunities for horizontal collaborations were not fully leveraged.

Continuous improvement

Continuous improvement processes are important to ensure fast changing areas, such as business intelligence programs, are adapting to changing environments.

The internal audit consulted a sample of business intelligence teams working in workload development to determine if there was a periodic revision or reassessment of the business intelligence processes to ensure they were working as intended, and to identify opportunities for improvement.

The internal audit found that there were generally no requirements for the programs to continuously assess and draw lessons from their business intelligence to ensure continuous improvement. The majority of workload teams consulted do not have a formal continuous improvement process and were relying on informal processes that varied from consistent cyclical reviews to ad hoc reviews. The majority of workload business intelligence teams consulted were relying on informal processes to provide peer reviews on the quality of the business intelligence produced.

Given the important role business intelligence plays in supporting the CRA’s workload development, inconsistent approaches to continuous improvement increase the risk of insufficient quality of business intelligence to support sound decision-making.

As business intelligence advances and evolves, it is important to ensure that the CRA staff have the tools required to support their business intelligence needs. The audit expected that processes would be in place to identify and prioritize the varying needs of developers and users, and that there is an established process for both the acquisition and deployment of tools. An acquisition process includes ensuring the tool requested is the most suitable one and that it is not already available. The process also requires purchasing the most suitable number of licenses and versions after collecting the requirements from the business intelligence developers. A deployment process, on the other hand, includes the distribution of the tool’s license, prioritizing the group of users piloting it and those who mostly need it, and ensuring the Information Technology security requirements are met. This process is important to ensure that the CRA is able to provide CRA staff with new business intelligence tools in a timely manner to support the development of quality business intelligence.

The SIIB leads both the acquisition and deployment of business intelligence tools throughout the Agency. This process is separate from other IT-related deployment processes led by the Information Technology Branch. The audit found there was an acquisition process in place, which included controls to help coordinate the acquisition of tools, with the oversight of the BIDGSC.

The audit included a review of the deployment of a business intelligence tool to be used across the CRA. The audit team found that the business intelligence tool deployment experienced some delays as there was no established process in place. The situation required unforeseen interventions, which could have been addressed earlier had there been an established deployment process. As a result, some business intelligence teams waited beyond the original delivery date to receive the tool.

Restricted or delayed access to innovative business intelligence tools can have a constraining effect on the quality and timeliness of business intelligence that needs to be generated and analyzed.

Recommendation #1

The SIIB, in collaboration with the ITB and other stakeholders, should strengthen the governance framework for business intelligence by addressing the following:

1. Establish CRA-wide objectives for Business Intelligence, including clear role responsibilities and a common definition of what constitutes Business Intelligence
2. Develop a strategy to coordinate, leverage and monitor business Intelligence efforts horizontally to ensure business intelligence meets program decision-making needs and supports the CRA Strategic Priorities
3. Establish expectations for developing Business Intelligence including continuous improvement requirements to ensure the CRA learns and continues to improve its business intelligence
4. Develop an effective deployment process for the timely delivery of business intelligence tools

Management Action Plan #1

The SIIB and the ITB agree with this recommendation and have developed an action plan to address it. The SIIB recognizes the benefit to strengthen the governance framework for business intelligence. Along with key stakeholders, the SIIB will lead the following actions.

1. By March 2025, SIIB will lead discussions and collaboration at the Assistant Commissioner (AC) level, including involvement of both corporate and program branches as well as regions, to establish enterprise-wide objectives and direction for business intelligence. This work will include the development of a document that identifies key definitions, roles and responsibilities for business intelligence areas in the CRA, and that illustrates how activities in this space are linked to broad strategies and plans such as the Artificial Intelligence Enablement Plan and the Information and Data Strategy.
2. By June 2025, SIIB will use existing strategies and/or strategies in development to enhance, coordinate, leverage and monitor Business Intelligence efforts horizontally. For example, in collaboration with the ITB, continue the process of developing a strategy that will leverage common capabilities that the cloud environment can offer to enhance business intelligence. In addition, continue to consult stakeholders, finalize, and publish this new strategy.
3. By June 2025, SIIB will undertake a scan of Business Intelligence areas, including a review of peer international organizations, and draft a document that will identify the gaps and opportunities to promote horizontal collaboration. SIIB will also consult and develop a plan to address these gaps and opportunities to strengthen the CRA Business Intelligence community of practice.
4. By December 2024, SIIB will collaborate with the ITB and other branches and regions to perform a review of the last business intelligence tool deployed; the review will consider prioritization process, resource allocations, governance options and identify lessons learned and the steps required for a new formal process.

Conclusion

Overall, the audit concluded that important components of the business intelligence framework are in place at the CRA; however, some improvements are needed to strengthen it. Below are the findings of the audit:

• The CRA has a documented governance framework for the management of data and acquisition of tools for the development of business intelligence, however, the audit found the following actions are required: establish CRA-wide objectives for business intelligence; clarify roles and responsibilities and determine a common definition of what constitutes ’business intelligence.
• The CRA Business Intelligence Strategy has evolved to focus on the data governance, and acquisition of tools, but it is missing strategies to leverage and coordinate business intelligence horizontally.
• Working groups are in place to foster and advance continuous improvement in CRA business intelligence work. However limited horizontal collaborations were observed.
• There are no CRA expectations for developing business intelligence, and inconsistent continuous improvement processes were identified. This has led to instances where business intelligence used to inform operational decisions were not reviewed and updated for multiple years.
• A process including an oversight body is in place to ensure an enterprise view has been considered when acquiring new business intelligence tools. However, there is no formal process to support the effective and timely deployment of the tools to the numerous business intelligence teams throughout the Agency.

Acknowledgement

In closing, we would like to acknowledge and thank the Service, Innovation and Integration Branch, the Information and Technology Branch, the Collections and Verification Branch (CVB), the Compliance Programs Branch (CPB), the Digital Transformation Program Branch (DTPB), and Business Intelligence and Quality Assurance from the Western, Ontario, Quebec, and Atlantic regions for the time dedicated and the information provided during the course of this engagement.

Appendices

Appendix A: Audit criteria and methodology

Audit criteria

Based on the Audit, Evaluation, and Risk Branch’s risk assessment, the following line of enquiry was identified:

Audit Methodology

The methodology used in the examination included the following:

• Document review of roles and responsibilities, strategies, and policies
  • Reviewed the documents that state the roles and responsibilities of branches, governance committee members, and working groups. The reviewed documents include terms of references, directives, meeting minutes, summaries of discussion, policies, and strategies.
  • Reviewed the documented business intelligence requirements and IT architecture governance records of decisions, standards, and enterprise solution considerations.
• Process and methodology review
  • Identified and selected teams from the CPB and the CVB who use business intelligence and documented what business intelligence they use, and what information, if any, is communicated back to the business intelligence teams as part of their continuous improvement processes.
• Interviews
  • Conducted interviews with key stakeholders (committee members, decision makers, and business intelligence producers/developers) in the SIIB, the ITB, the CVB, the CPB, the DTPB, and Business Intelligence and Quality Assurance from the Western, Ontario, Quebec and Atlantic Regions to understand their roles and responsibilities and to gather information on the line of enquiry.
• Observation of governance committee meetings and working group meetings
  • Identified and attended both governance committee and working group meetings with the intent to observe and compare against a checklist that included factors such as key stakeholders, inclusivity, agenda items, partnerships and collaboration already in place and potential opportunities. The checklist also included the observation of the processes for sharing knowledge and best practices, the processes and mechanisms for sharing information that results from meetings and for feedback opportunities within the committees and working groups, within each branch and across the CRA.

Appendix B: Glossary