Evaluation Study – Horizontal Compliance Management: GST/HST Audit Program

Final Report

Audit, Evaluation, and Risk Branch

May 2018

Executive summary

The Agency Management Committee of the Canada Revenue Agency (CRA) approved the evaluation framework of the Horizontal Compliance Management Evaluation Study on July 18, 2012. Horizontal compliance management refers to factors which are seen to support an organization’s ability to operate in a coordinated way. These factors include governance structures which oversee strategic and operational decisions, information systems, and performance measurement. Greater coordination can be achieved by effectively managing these factors and their downstream impacts on each other or on other stakeholders.

In order to assess the effectiveness of internal management practices, the evaluation examined organizational change introduced to the Goods and Services Tax/Harmonized Sales Tax (GST/HST) Audit Program over the period of April 2012 to March 2017. This evaluation looks at the governance structures providing oversight, the ability of compliance workers to access and use compliance information, and the monitoring and reporting framework in place. The evaluation assesses the effectiveness of practices at the organizational level rather than through the lens of a particular program or activity in order to examine interrelated and interdependent parts.

Findings

Overall, a series of organizational changes impacted the GST/HST Audit Program in expected and unexpected ways. Lessons can be learned from the post-implementation monitoring that occurred, as the impacts were not all detected.

In examining program results, the following key findings emerged with respect to a robust performance measurement and reporting system which can shed light on both desired and undesired results:

• the GST/HST Audit Program performance results suggest productivity declined after the introduction of organizational changes;
• the emphasis on audit quality is competing with traditional targets and performance indicators; and
• performance indicators reinforce individual program mandates and do not provide insight on multiple obligations of the same taxpayer (filing, reporting, and payment).

With respect to the availability of sufficient and timely information which can impact the CRA’s ability to achieve its results and compliance objectives, the following emerged:

• GST/HST auditors had access to timelier and a greater amount of information with the introduction of a new case management system; and
• the inconsistent performance and reliability of the new case management system limited GST/HST auditors’ ability to access and use the information.

In examining what contributed to the observations noted above, the following key findings emerged with respect to the design and functioning of governance structures:

• oversight focuses on project management metrics while aspects of quality receive less attention;
• the adoption of Agile strengthened the collaboration between business and Information Technology (IT); and
• there are post-implementation performance gaps based on the monitoring and reporting approach for major projects.

Recommendations

A series of four recommendations are provided. Management responses have been developed for the recommendations.

One recommendation has been developed for GST/HST program-specific reporting to support comprehensive measurement of results and increased collaboration.

1. The CRA should articulate a plan to supplement the Taxes Earned By Audit performance indicator with other measures and consider their impact on horizontal compliance.

One recommendation has been developed to support integrated corporate reporting.

2. The CRA should adopt a GST/HST horizontal performance indicator and also examine and implement ways to strengthen the organization’s ability to introduce more horizontally integrated performance indicators.

Two recommendations have been developed to reduce pre- and post-implementation risks of Strategic Investment Plan projects within CRA governance structures.

3. The CRA should re-examine and modify as required its approach to the planning and subsequent post-implementation monitoring and reporting of Strategic Investment Plan projects and performance.

4. The CRA should strengthen its governance to ensure product quality, system functionality and user experiences are considered.

1. Introduction

Evaluation foundation and framework

A feasibility study completed in 2011 found that an evaluation of horizontality was possible from two perspectives: first, in an external taxpayer behaviour sense, captured by the term “full compliance” and second, in the sense of internal administration captured by the term “horizontal compliance management”. In July 2012, the Agency Management Committee of the Canada Revenue Agency (CRA) approved the evaluation framework of the Horizontal Compliance Management Evaluation Study.

The framework identified two evaluation issues to align with the concept of horizontality in both the external, taxpayer-centric sense and the internal, management sense. The framework noted that Issue 1 findings (external focus) would play an important role in supporting how Issue 2 (internal focus) would be addressed and that they would need to be completed sequentially.

In February 2014, findings on Issue 1 were delivered in the Horizontal Compliance Management T1 Self-Employed Interim Report to the CRA’s Management Audit and Evaluation Committee. This report was exploratory in nature and did not contain recommendations. It provided the known compliance outcomes being achieved in the T1 self-employed segment, using the full compliance perspective (filing, payment, and reporting)^{Footnote 1} .

Establishment of current evaluation

This evaluation provides findings on Issue 2 by assessing the effectiveness of internal horizontal management practices within the organization. The following four research questions^{Footnote 2}  capture the factors which are seen to support an organization’s ability to operate in a coordinated way: 

1. Do those involved in co-managed accounts (i.e., accounts that require intervention by multiple programs) have sufficient information?
2. Are governance structures used for sharing information and coordinating program changes that are undertaken?
3. How are downstream impacts managed?
4. What alternative indicators may shed new insight into past performance or support early warning systems?

To assess management practices, the evaluation scoped in the Goods and Services Tax/Harmonized Sales Tax (GST/HST) Audit Program; this was approved by the Management Audit and Evaluation Committee in May 2014. As a compliance organization, CRA employees require tools and materials which facilitate a consistent, efficient, and accurate approach to work. In recent years, the CRA introduced a fully integrated technology solution to compliance workers. It was designed to improve the ability to share, collaborate, and move work across geographic and business lines while ensuring integrity.

In the context of the GST/HST Audit Program, this evaluation looks at the governance structures providing oversight, the ability of compliance workers to access and use compliance information, and the monitoring and reporting framework in place. The evaluation assesses the effectiveness of practices at the organizational level rather than through the lens of a particular program or activity in order to examine interrelated and interdependent parts.

Scope exclusion

The evaluation had two initial research purposes: to merge the T1 self-employed population with its GST/HST reporting requirements in order to achieve a more fulsome taxpayer view related to Issue 1 and to examine the effectiveness of internal horizontal management practices (Issue 2). The evaluation originally pursued the commitments in the context of both T1 self-employed and GST/HST compliance activities. However, data gathering activities led to the evaluation reducing scope to focus on Issue 2, the effectiveness of internal management practices, and in respect of the GST/HST Audit Program only. There was significant complexity merging data across the T1 and GST/HST business lines and early fieldwork did not find an adequate sample of T1 self-employed governance decisions to seek evidence of coordinated actions based on the decisions taken.

2. Background

GST/HST Audit Program

The CRA’s GST/HST Audit Program is designed to make sure GST/HST registrants comply with reporting requirements and to identify and address non‐compliance in order to protect Canada’s revenue base. Within the CRA, the GST/HST Directorate of the Domestic Compliance Programs Branch is responsible for the functional management of three Divisions delivering a suite of five GST/HST audit programs:

• Large Business Audit Program;
• Refund Integrity Program;
• Small and Medium Business Audit Program;
• Specialty Audit Program; and
• Aggressive GST/HST Planning.

Between fiscal years^{Footnote 3} 2012 to 2013 through 2015 to 2016, the GST/HST Audit Program had an average budget of $165 million, representing 2,100 full-time equivalents. The CRA completes approximately 70,000 GST/HST audits annually, generating a reported fiscal impact^{Footnote 4} between $1.5 and $2.4 billion from fiscal years 2012 to 2013 and 2015 to 2016.

The CRA recognizes that enhancements and innovations are important to be a world class tax and benefit administration. It periodically implements organizational changes of varying degrees and scopes. Between 2010 and 2014, the GST/HST Audit Program experienced a series of organizational changes (Figure 1). The organizational changes did not impact all GST/HST audit programs to the same extent, as certain changes were targeted to specific programs. Select changes are further described given their operational impacts on the day-to-day routines and performance of auditors.  

Figure 1: Organizational changes introduced  to the GST/HST Audit Program from 2010 to 2014

Harmonization

In July 2010, Ontario and British Columbia implemented a harmonized sales tax. British Columbia subsequently voted to exit harmonization which took effect in April 2013. The workload effects of harmonization began to operationally impact GST/HST auditors in fiscal year 2012 to 2013 as the harmonized audits entered the workload. As part of the transfer of responsibilities, employees from the province of Ontario joined the CRA in 2010 and again in 2012. Similar employee transfers occurred on a smaller scale with British Columbia and Prince Edward Island.

Pre-Assessment National Inventory

In 2012, the Refund Integrity Program shifted from a regional to a national inventory workload called Pre-Assessment National Inventory. It is the primary risk assessment tool for all credit returns filed by registrants across the country. The national inventory allowed auditors to identify and examine high risk files no matter where they happened to be located.

Business Intelligence and Audit Quality Review

In September 2013, the Business Intelligence Program and the Audit Quality Review Program were implemented. The two programs are based on the pillars of enhanced risk assessment and an increased focus on audit quality.

The Business Intelligence Program consists of workload development. A new business intelligence role was created to segregate duties so that auditors could no longer both select files and audit them. The business intelligence teams are responsible for gathering, analyzing, integrating and applying business intelligence in determining file selection. This approach improves the organization’s ability to select the highest risk files and develop more focused strategies. Choosing the highest risk files is the first step to detecting and addressing non-compliance.

The Audit Quality Review Program provides the Business Intelligence Program with better information which can then be mined to enhance the subsequent file selection. It is a national program designed to improve the quality of audits and provide management with a degree of assurance that audit activities are performed with integrity and meet national pre-determined standards.

Case management system (Integras)

From October 2013 to April 2014, a new case management system^{Footnote 5} (Integras) was deployed to the GST/HST Audit community across the country. GST/HST was part of the first wave to receive full deployment of the system, adding 2,500 employees to the user base. Integras was intended for use by all audit groups and was eventually deployed to the T1 and T2 business lines bringing the total number of users to 7,000 employees.

The development and deployment of Integras was a major investment project for the CRA. The project consolidated and modernized the tool set used by auditors and addressed sustainability issues related to legacy compliance systems. Integras provided an automated workload management, auditing, and reporting system, including an electronic record of decisions, approvals, and accesses. It is a common workload tool across audit functions helping ensure horizontality across audit programs.

Integras was built on the whole of client approach to compliance, incorporating audit controls, supporting business transformation initiatives including enhanced business intelligence capacity, and improving the efficiency and effectiveness of auditors by housing all information in an integrated work environment. Auditors rely on Integras to, among other things, view and access assigned cases, communicate with team leaders, and store evidence and documentation. It transformed the way audit cases were processed, managed, stored and retrieved.

3. Evaluation methodologies

The evaluation study covers the period from April 2012 to March 2017 to ensure a broad timeframe and allow for trend analysis. The period of time studied provided an opportunity to examine GST/HST Audit Program performance prior to and after the changes introduced in 2013 and 2014 that impacted the daily routines of the GST/HST audit community, namely Business Intelligence, Audit Quality Review, and Integras. Five main methodologies were used in support of this evaluation:

Data analysis of GST/HST audit results from a variety of sources and systems within the Agency Data Warehouse such as Integras, Audit Information Management System, GST “Line of Business” Case System, GST/HST Return and Adjustment Processing System, Standardized Accounting, Automated Collections and Source Deductions Enforcement System, Revenue Enforcement Management Information and Tracking System, and CRA administrative data.

Document reviews of program information, policies, and procedures; project documentation and communications; corporate and performance reporting instruments; governance structures; executive performance agreements; and third party research and reports.

Case study of the Compliance Systems Redesign project which deployed Integras to the GST/HST Audit Program and the existing governance and oversight processes from the time period.

Benchmarking of performance reporting from other tax administrations – United States of America, Province of Quebec, United Kingdom, and Australia, as well as an audit from the Office of the Auditor General of Ontario on the deployment of an information technology (IT) project supporting the administration and delivery of social assistance in Ontario.

In-depth interviews with a horizontal cross-section of CRA managers and staff from five branches and four regions. A total of 40 officials from Headquarters (HQ) and 60 from the regions were interviewed. Representatives were included from the Assessment, Benefit, and Service Branch, Collections and Verification Branch, Compliance Programs Branch^{Footnote 6}, Information Technology Branch, and Strategy and Integration Branch.

4. Findings

A series of organizational changes impacted the GST/HST Audit Program in expected and unexpected ways. Findings suggest there is unmitigated risk in the period prior to a major IT deployment and immediately after which may not be detected by monitoring and reporting.

4.1 Performance measurement, monitoring, and reporting

Robust performance measurement and reporting provide the ability to shed light on desired and undesired results and initiate corrective measures when needed. For the GST/HST Audit Program the key internal performance measure is “Tax Earned by Audit” and the key measure reported publicly is “fiscal impact.” Tax Earned by Audit and fiscal impact are similar measures that both provide a global dollar value of detected reporting non-compliance. For GST/HST, Tax Earned by Audit represents the sum of federal and provincial taxes^{Footnote 7} while fiscal impact includes Tax Earned by Audit, interest and penalties, and other audit related adjustments.

The GST/HST Tax Earned by Audit and fiscal impact performance indicators represent the breadth of audit activity. They are calculated based on audits which vary in their taxes at risk and therefore, revenues due (payable) to the government. Generally, GST/HST audits classify taxes at risk in one of three ways:

1. Audits which prevent unwarranted refunds from being paid: these audits contribute to Tax Earned by Audit by preventing refund payments to non-compliant credit filers. The Refund Integrity Program is the primary contributor. The Program’s verification work prior to assessment prevents the issuance of large refunds that may, if issued, pose a collections risk. For example, over the eight-year period from 2009 to 2017, 19 cases totalling $1.1 billion in refunds claimed, had the refunds disallowed before any payments were made.
2. Audits which do not result in a loss of tax revenues at all and require no subsequent payment or collections efforts: these audits contribute to Tax Earned by Audit but do not result in taxes payable. A coding change introduced in fiscal year 2014 to 2015 newly recognized “wash” transactions as part of GST/HST Tax Earned by Audit calculations. Wash transactions involve situations where the GST/HST that was not charged by one taxpayer would have resulted in an equal and offsetting input tax credit for another taxpayer. In other words, the transaction resulted in no permanent gain or loss of tax revenues^{Footnote 8}.
3. Audits which result in a loss of tax revenues and require subsequent payment or collections efforts: these audits contribute to Tax Earned by Audit by detecting reporting non-compliance. It comes with the risk that the taxes re-assessed need to be paid, but may be disputed by the taxpayer through the objections process, may require collections actions, and may not be fully collected.

Overall, understanding how different audit work contributes to Tax Earned by Audit highlights that not all Tax Earned by Audit is created equal in terms of the potential taxes payable. It is a measure of legitimate audit activity that does not reflect results or outcomes once subsequent CRA interventions on the same taxpayer file are taken into account.

With a better understanding of what Tax Earned by Audit represents, the evaluation reviewed program results to examine trends in performance.

4.1.1 GST/HST Audit Program performance results suggest productivity declined, as to be expected when implementing organizational change

GST/HST auditors stated in interviews that they perceived tasks to take longer to complete after the introduction of organizational changes that impacted their day-to-day activities and processes. In the years following Ontario harmonization, the complexity of files and associated dollars at risk gradually increased over time as the harmonized audits entered the workload and were selected for audit. With the emphasis placed on audit quality and the introduction of the business intelligence function during fiscal year 2013 to 2014, it would be expected to take longer to close an audit as more complex files were selected and decisions were documented to a consistent standard.

According to Program statistics, the number of cases closed by the GST/HST Audit Program declined from almost 89,000 in fiscal year 2012 to 2013 to 68,000 during fiscal year 2015 to 2016, for a total decline of 23% over the four year period. During this same period, the number of annual full-time equivalents increased by 4%.

Further exploring these results found the median time (days) to close a file increased in the Small and Medium Business Audit Program and, to a smaller extent, the Refund Integrity Program. For the Small and Medium Business Audit Program, over the four-year period the overall change in the median time (days) to close a file increased by 35% or 67 days (from 189 days to 256 days). The largest year-over-year increase (20% or 38 days) occurred in the year business intelligence and audit quality review were introduced. The evaluation also observed a 15% year-over-year increase (or 35 days) the next year when the new case management system, Integras, was fully deployed, followed by a 2% year-over-year decrease (or 6 days) the final year.

Figure 2 shows nearly $2.5 billion in fiscal impact identified in fiscal year 2015 to 2016 compared to $745 million in fiscal year 2010 to 2011. Year-over-year increases in total fiscal impact were unexpected given the metrics indicating a decline in productivity. Growth in fiscal impact provides an indication the CRA is selecting and auditing files with greater tax at risk.

In fiscal year 2014 to 2015, the GST/HST Audit Program introduced a coding change to its audit result methodology following an extensive review of policies and practices. The change aimed to ensure the CRA reported on the full spectrum of its audit efforts and resulted in additional fiscal impact due to the inclusion of additional items such as uncollectable amounts, wash transaction amounts, and account correction amounts. This resulted in fiscal impact increasing by over $450 million, or about one-quarter of the total reported compared to earlier years.

Isolating the impact of the coding change provides a more accurate comparison of results year-over-year. All things being held equal, after fiscal impact increased by 33% in fiscal year 2013 to 2014, it declined 16% in fiscal year 2014 to 2015 and remained below its fiscal year 2013 to 2014 levels in the subsequent year. Aggregate reporting shows an increase of 10% in fiscal year 2014 to 2015 from the previous year.

Notably, the evaluation determined that a strategic shift in the workforce composition positively impacted results. The workforce shift over the five year period resulted in increased auditors eligible to take on more complex, higher revenue files that would not have been achieved had the workforce composition remained largely the same. This accounted for a 6% increase in results for fiscal year 2015 to 2016 compared to fiscal year 2010 to 2011.

Figure 2: GST/HST fiscal impact by fiscal year

This section analysed GST/HST audit results largely in terms of outputs and activities to identify trends in program performance not detected by the monitoring and reporting systems in place. These efficiency measures offer insight on the day-to-day experience of auditors which is not provided by reporting total fiscal impact, as Figure 2 shows an overall increase in year-over-year results.

With a better understanding of results, the evaluation looked at the processes that support the tracking and reporting of Tax Earned by Audit. The approach to set annual targets is of particular interest, given its potential to influence horizontal compliance management behaviours and create downstream impacts.

4.1.2 Setting GST/HST program targets has challenges and may be working at cross-purposes with the importance of audit quality

The CRA sets targets in part to support revenue commitments to external stakeholders. In order to track progress on commitments, a complex process is put in place to allocate annual targets at the program and regional levels and align resources accordingly.

To set targets, HQ and regional officials together assess risk in the GST/HST environment and discuss strategy for each program. HQ officials noted that the target-setting approach is largely quantitative and does not take into account contextual changes that could be expected to temporarily impact performance. Meanwhile, regional officials noted that context-dependent factors can make achieving targets a challenge, citing turnover and in-year staffing challenges, potential impacts from the deployment of a new system, and changes in policies and procedures. 

A challenge associated with target setting is that feedback on their accuracy can only be received 18 months after the fact. Targets for an upcoming year are largely finalized in February but results for are not available until June of the following year. The evaluation examined four years of data related to target achievement at the national and regional level. The analysis found that across the four years for all national GST/HST audit programs, Tax Earned by Audit results achieved were within +/- 15% of the target only 7 out of 20 total instances (35%). The variance was greater at the regional level, with results achieved within +/- 15% of the target only 20 out of 80 total instances (25%). Regional variance may be partially explained by the portability of certain workloads among regions to address resource gaps and emerging compliance risks.

In interviews, regional team leaders and auditors expressed mixed opinions about Tax Earned by Audit targets. When targets are reasonable, they can have a motivating effect on everyone. When they are unattainable, frustration and stress can set in, and when they are achieved early in the year, motivation and urgency decline.

In recent years, the Domestic Compliance Programs Branch has emphasized audit quality as a priority. The implementation of Audit Quality Review and a series of related performance indicators was a positive step in reinforcing the message. At the same time, the Branch continued to provide annual Tax Earned by Audit targets at the regional level. While nearly all of the auditors in the regions understood the reasons for the focus on audit quality, interviews found that despite the introduction of Audit Quality Review and its new measures, the field still considers achieving Tax Earned by Audit targets the priority of senior management. In fact, within-region cooperation is common in order to increase the chances of achieving the jointly held target.

Overall, the GST/HST Audit Program is in the midst of a process and cultural shift that accentuates the importance of audit quality and de-emphasizes Tax Earned by Audit targets. Higher quality audits support both the collections and appeals functions. Full realization may be a challenge as long as Tax Earned by Audit and its targets remain in place as the cornerstone of existing performance measurement and reporting.

4.1.3 There are no performance measures in place to enhance coordination and facilitate the horizontal compliance management of multiple obligations of the same taxpayer

Contents of corporate performance reports tend to be vertically aligned to the mandates of branches. Reported outputs from filing, reporting, and payment compliance programs are not connected to determine the extent of horizontal management: the compliance function reports on Tax Earned by Audit, the collections function reports on tax debt resolved, and there is no link made between the two nor to the results of disputed tax assessments reported by the appeals function.

By measuring performance aligned to the mandates of branches or programs, behaviours and choices are reinforced to also occur in silos so that one’s own performance reporting is optimized against the metric. GST/HST auditors interviewed noted this approach provides them with little incentive to act collaboratively with their colleagues in collections or appeals as each role is accountable for its own output. For example, GST/HST auditors have access to T1 and T2 information and notes that most collections agents find helpful but do not have access themselves. When this information is not shared, collection agents may spend time researching a taxpayer’s information which decreases the overall compliance-related productivity. There are no legislative requirements that would prohibit this information sharing from occurring.

An integrated, results-based performance indicator is not measured or reported to provide insight on multiple obligations of the same taxpayer or on performance that implicates multiple branches. Trend analysis of CRA data found that, in recent years, approximately 80% of the cases where the amount payable created by a GST/HST re-assessment exceeded $100,000 have been fully resolved within one year of the assessment being raised. Over time, there is potential to increase the target for this measure which could in turn enhance collaboration and coordination at the Tax Services Office level if implemented as a shared target to be achieved across functions.

Benchmarking suggests that other tax administrations report on variations of an integrated, results-based performance indicator that measures dollars collected from Tax Earned by Audit:

• In its Annual Report 2015-2016, Revenu Quebec reported that 64.7% of the amounts assessed during the year were collected, exceeding its target of a 55% collection rate related to tax audit activities. Such information could support other performance monitoring, as Revenu Quebec reports a return on investment measure based on its collected Tax Earned by Audit measure. In 2014-2015, Revenu Quebec’s overall return on investment was 6.91 dollars of Tax Earned by Audit for every dollar spent.
• The Australian Taxation Office reports the total liabilities raised and the estimated cash collected.
• Her Majesty’s Revenue and Customs of the United Kingdom reports compliance yield, comprised of four distinct and different yield types: cash collected; revenue loss prevention; future revenue benefit; and product and process yield which estimates the impact of legislative changes to close tax loopholes and changes to other processes that reduce opportunities to avoid or evade tax.

4.2 GST/HST audit community experience

The CRA relies on a combination of technology and human efforts to deliver high quality, accurate, and efficient service and compliance. Appropriate and high-performing IT systems are essential to ensure the right information is in the right hands at the right time. To be productive, compliance staff must access, interpret, and manage a vast amount of data and information to make decisions on a taxpayer’s account and support horizontal compliance.

The evaluation reviewed the availability and timeliness of information to GST/HST auditors provided by the new case management. The system eliminated the need for auditors to access three separate information systems and allowed auditors to access and view historical information.

4.2.1 As early adopters of a new case management system, GST/HST auditors had access to timelier and a greater amount of information

Interviews found that the breadth and timeliness of information available to GST/HST auditors improved in various ways during the first two years of the operation of Integras. Auditors noted more relevant information in one easy-to-access location, a more efficient review and monitoring process, and improved facilitation of horizontal information sharing between auditors and the areas of business intelligence and audit quality review. In the case of the Refund Integrity Program, they noted more real-time file selection and viewing in combination with the Pre-Assessment National Inventory in place since 2012.

4.2.2 The new case management system demonstrated inconsistent system performance when implemented

While the vast majority of GST/HST auditors understood the potential and benefits from the new system, they expressed frustration in interviews that Integras was deployed in a state that had undesired impacts on their day-to-day activities, particularly in light of other changes introduced to ensure adequate and accurate documentation of audit decisions. They recognized that there was a learning curve associated with Integras but they also noted that some functions were less reliable, while other functions simply did not work, which they felt reduced their productivity. They noted issues of logging on to the system, synchronization, an unreliable checking-out feature, slow response times, and difficulty finding and organizing files.

Available data supports the perception that Integras did not function reliably upon deployment to the GST/HST Audit community. By November 2015, 18 months after deployment, there were 150 documented workarounds in use by GST/HST auditors for Integras. In fiscal year 2014 to 2015, GST/HST auditors opened over 2,700 bug-related Integras tickets, as tracked by the project office. While the total number of tickets opened in fiscal year 2015 to 2016 declined to 2,200, the average time to resolve tickets increased year over year^{Footnote 9}.

Auditors also experienced frequent system outages, which are periods where Integras was unavailable for a portion of the workday. The nature, scope, and duration of outages varied. Over the high-demand six month period of January 2016 to June 2016, Integras experienced periods of outages that impacted operations an average of 3.33 days each month. Improvement was noted for the period of July to September 2016 when the system experienced periods of outages that impacted operations an average of only 0.66 days per month.

When Integras is combined with the Business Intelligence and Audit Quality Review programs, auditors felt they had to spend more time inputting data into an imperfect system in order to close a file. Closing a file finalizes the CRA’s assessment of a taxpayer’s reporting requirements, allows a notice of re-assessment to be issued, and enables the results to be counted and reported in key performance results.

4.2.3 Sick leave usage increased significantly in the GST/HST Audit Program after the introduction of organizational changes and has since stabilized

The available literature suggests a relationship between organizational changes, work-related stress, and employee absenteeism. After computer usage became a part of the daily routine of many workers, a substantial body of research began to emerge establishing a linkage between slow or poorly designed software and user stress, frustration, anger and even helplessness^{Footnote 10}. Furthermore, the causal link between stress and poor health has been widely accepted by researchers and the general public since the late 1990s^{Footnote 11}. Triggers of stress include having little control over the terms and conditions of the work being done, occupying a job that does not match one’s skills and abilities, and having insufficient support from supervisors or colleagues^{Footnote 12}.

By the mid-2000s, it was generally accepted that untreated workplace stress was associated with increased levels of employee absenteeism and turnover, decreased levels of productivity, as well as lost workdays due to disability or sick leave^{Footnote 13}. More recently, a Morneau Shepell survey conducted in October 2016 of 1,018 employers and employees across Canada found that job re-design, which included technological changes, was found to have the strongest correlation to sick leave usage for both physical and mental health^{Footnote 14}.

As outlined in Figure 1, GST/HST auditors experienced a series of changes in their work environment which changed the ways in which GST/HST auditors routinely conducted their activities. Auditors stated in interviews that as a result of no longer being able to select files for audit, they felt less control over their work. This is to be expected as the segregation of duties was deliberately implemented to foster a culture of integrity. Further, when closing a file, auditors saw their work as becoming increasingly clerical, given the emphasis on documenting decisions in support of audit quality. While there were a number of contributing changes over the period covered by this evaluation, auditors stated in interviews that Integras was the key change causing frustration due in large part to the inconsistent system performance during the first two years of operation.

Two studies^{Footnote 15} that looked at a large-scale IT system deployment to the Ontario Ministry of Community and Social Services showed the unintended impacts on users of deploying a system that did not function optimally. This suggested that it would be worthwhile to examine trends in sick leave usage amongst GST/HST auditors as an indicator of user frustration and stress.

Trends in sick leave usage were examined amongst three mutually exclusive groups: GST/HST auditors, other auditors, and the population of CRA employees. These three groups were compared to control for the effects of proposed changes to sick leave benefits by the Government of Canada^{Footnote 16} since all CRA employees would have been impacted.

Overall, median hours of sick leave usage increased across all three groups (Figure 3) while GST/HST auditors used more sick leave at a faster rate. Fiscal year 2013 to 2014 represents the year business intelligence and audit quality review were introduced and fiscal year 2014 to 2015 represents the year Integras was fully deployed and available for use by GST/HST auditors.

Figure 3 can be summarized as follows: GST/HST auditor sick leave usage increased from fiscal years 2011 to 2012 through 2014 to 2015 before levelling off in fiscal year 2015 to 2016, although at higher levels when compared to fiscal year 2011 to 2012. Overall, there was more time away from the office in the five-year period, and in particular following the introduction of the business intelligence program, audit quality review, and Integras. This may partially explain the decline in productivity and fiscal impact amounts noted in section 4.1.1.

Figure 3: Median Hours of Sick Leave Usage by GST/HST Auditors, Other Auditors and the Rest of CRA, Over Five Fiscal Years

Within the GST/HST audit programs, the timing and frequency of a GST/HST auditor’s exposure to Integras would have been unique to each individual. Comparing auditors based on the number and timing of cases closed in the new system yielded two important findings. First, GST/HST auditors with a higher frequency of exposure took more sick leave. Those who closed a higher number of cases had a higher median use of sick leave than those who closed a smaller number of cases. Second, increases in GST/HST auditor sick leave usage occurred relative to when they were first exposed to the system. Those who closed a file in Integras in fiscal year 2014 to 2015 tended to use more sick leave during that year while those who first closed a file in fiscal year 2015 to 2016 tended to increase their sick leave usage in the second year of operation.

A post-implementation metric such as sick leave usage is not currently monitored to provide an indication of success or user satisfaction with organizational change. Timely post-deployment user surveys that measure the level of comfort in using system features, whether functions are working as intended, and a user’s overall well-being, can help to monitor any impact on employees when introducing change.

4.3 Governance

The ability to take into account potential downstream impacts relies on a combination of information sharing and the effective coordination of decisions within governance structures. The evaluation reviewed the governance around the development and deployment of Integras as a case study to illustrate the design of governance structures and how multiple stakeholders coordinate across functions and responsibilities.

The deployment of Integras was a priority for the CRA. Several years and millions of dollars were invested in developing the case management system recognizing the need for a more sophisticated and integrated solution than the existing one used by auditors.

4.3.1 Project oversight in the CRA focuses on project management metrics such as milestones, dates, and budget

The project to deliver Integras was an extremely large and complex undertaking, implemented over many years. During that time, the project saw its scope, plan, budget and expected benefits reviewed and modified a number of times.

In 2012, an independent third party review acknowledged the need to move forward with a deployment plan. It stated the solution was widely anticipated and the value was apparent while also noting the deployment approach was high risk given the unknowns and uncertainties at the time. The review made a number of recommendations to address key risks and issues. The CRA developed an action plan in response.

Among the over 40 detailed actions were commitments to include deployment in the performance agreements of executives in the Compliance Programs Branch, Information Technology Branch, and for regional Assistant Commissioners. Performance agreements provide insight into what is expected of CRA employees. Their content is guided by the mandates, roles and responsibilities, and changing priorities associated with various positions across the organization. As a result, they influence operational plans and priorities, including decisions on the allocation of resources in support of objectives.

The evaluation reviewed annual performance agreements over a six-year period for executives responsible for the deployment of Integras within the Information Technology Branch and the former Compliance Programs Branch. The analysis found common references to project management metrics such as milestones, dates, and managing budgets. There was little emphasis on monitoring the impact of system functionality on the user experience or productivity and reacting as needed, nor did they integrate and prioritize accountabilities for quality of the solution. Many interviewees stated that expectation for product quality was implied.

The action plan also included the development of deployment criteria used to identify the GST/HST business line as the second of four business lines to adopt the technology. One of the reasons was the level of risk associated with the sustainability of the existing legacy system on which the business line heavily relied. In 2008, a vendor declared, that as of October 2014, the underlying programming language in the existing system would no longer be supported. The programming language was critical to the functioning of GST/HST operations and no backup was available. If it ceased to function, over $1 billion in annual estimated recovery would be at risk. In 2012, the vendor declared that support would be extended until 2024 instead of 2014. No evidence was found of the change being reported in the context of the timing of the Integras deployment.

Considering these factors, senior management faced pressure with the Integras deployment in order to respect time and budget commitments and address sustainability risks. At the same time, there were indications the new system required further development in order to function optimally for users as the April 2014 GST/HST full deployment deadline approached.

In reviewing project documents, the concerns that auditors raised in interviews about the functioning of Integras aligned with testing results; system testing prior to deployment was predictive of impacts on the user experience. The Product Assurance Division of the Information Technology Branch and the Acceptance Testing Division of the Assessment, Benefit and Service Branch^{Footnote 17} tested and reported on the IT backbone and functionality. The Product Assurance Division was not able to fully verify that system stability and performance would meet production standards. The Acceptance Testing Division reported that the majority (71%) of elements tested had no defects however, several other key features, such as the synchronization, checking-out cases, and finding and organizing files functions had a failure rate of about 5%. The failure rates are within acceptable ranges for deployment. The system went into production with the expectation that improvements would be required.

Project documentation suggests that senior decision-makers knew that technical issues existed, as is the case with many system deployments, but were not fully aware of the extent of challenges faced by early adopters. Reporting occurred at regular intervals to various audiences given the importance of the project. Information reported to oversight bodies tended to be aggregated and focused on progress and readiness. Taking a user-centric perspective, the information did not specify the functionalities that more frequently failed during testing, nor did it describe how their failure could impact the user environment.

As the deployment deadline approached, the project management office did not request a change in timeframe and stakeholders did not voice concerns or issues. Integras was deployed nationally to the GST/HST Audit Program in April 2014 meeting the time and budget commitments. While sustainability of the legacy system was a continued risk, the opportunity to defer launch may have existed to allow more time to resolve technical issues and reduce negative impacts on the end users and performance results.

4.3.2 The adoption of Agile strengthened the collaboration between business and IT

The “Agile” methodology represents an iterative and incremental approach to software development, performed in a highly collaborative and flexible manner, within a governance framework that includes both the project sponsor representatives and the IT development team. Through repetition of short work cycles or “sprints”, teams can continually assess the direction of a project, providing time to correct and steer it in another direction. The Agile development methodology was adopted for the Integras development initiative as early as 2007 and was the first CRA project to use Agile.

Within the context of Integras, the project management office situated in the branch of the project sponsor was the hub of the planning, development, and deployment process. It regularly liaised with the Headquarters GST/HST Directorate and collaborated with the Compliance Portfolio Application Support^{Footnote 18} group of the Information Technology Branch to ensure that user needs were appropriately communicated and implemented in the development process.

During the early years of Integras development, the Information Technology Branch had a greater understanding of business mapping and was put in a position to support the business architecture function. The project management office relied on the Information Technology Branch to bridge the gap due to its experience and struggled to offer a robust challenge function. As the Agile process took hold, the level of collaboration between the business and IT sides increased substantively when employees were co-located in the same building. All of this was seen to increase the business’ authority over the project and drive the project management office to invest in its business architecture function. Full benefits were not realized, as difficulties remained involving all stakeholders in user acceptance testing^{Footnote 19} and release activities.

The Canada Revenue Agency – Test Strategy Assessment: Target State Recommendations (Gartner Consulting; August 2016) identified the need for the Product Assurance and Acceptance Testing divisions to be consulted and included earlier in the development process. The study also identified that user acceptance testing performed was closer to being functional testing^{Footnote 20} and that end user acceptance testing was not being done at the CRA.

In support of delivering a quality end product for users, the CRA needs to allocate sufficient time and funding for end user acceptance testing and the consideration of testing results prior to decision-making. This needs to be balanced against other reasons which may drive the need to proceed to production and ultimately deal with issues that arise. For Integras, 61 auditors participated in pilot-testing which started in October 2013 and was intended to last 8 weeks. The decision to fully deploy the system nationally to all of GST/HST occurred three weeks after the start of pilot testing. Two weeks later, a gradual deployment began, enabling the April 2014 deadline where all new cases were to be opened in Integras.

4.3.3 Post-implementation performance gaps emerged based on the approach to monitoring and reporting

The CRA Resource Management Committee^{Footnote 21} is responsible for overseeing the management and progress of major project investments, including establishing budget priorities and requirements in accordance with CRA priorities, overseeing the allocation and control of the CRA's financial resources, and providing oversight of financial and control risks. As part of overseeing major investment projects, the Resource Management Committee and its secretariat review and request details on the future anticipated benefits associated with projects. During the planning phase of a project, a project sponsor is expected to complete a benefits measurement plan that describes the benefits anticipated with the entire investment. After the project is executed and deployed, the project sponsor completes a benefits realization report providing feedback on the extent to which the anticipated benefits were achieved, as assessed by the project sponsor.

In recent years, the Finance and Administration Branch introduced a number of changes to the benefit management process:

• migrated the leadership of the benefit management practice from the Audit, Evaluation, and Risk Branch to the Resource Management Secretariat in the finance and Administration Branch in January 2016 to highlight the importance of this practice and increase information about benefits to decision makers;
• benefit mapping for all major projects to illustrate how the scope is linked to the business outcomes and ultimately to the strategic outcomes of the CRA; and
• a revised guide to benefit management providing an overview of how the intended benefits of a project are expected to be planned, substantiated, tracked, and reported on through each stage of the Major Project Investment Oversight Process.

The process changes show the CRA’s desire to continuously evolve its management practices based on lessons and observations. The impact of the changes are not assessed as part of the evaluation as the requirements were not in place at the time of the Integras deployment for GST/HST. Taking into consideration the steps already taken to provide additional information to decision-makers, the evaluation identified a risk that information on post-implementation performance may not be timely or complete.  

In terms of completeness, when a project sponsor’s planning and reporting responsibilities are considered within the “benefits” paradigm, there is incentive to choose and report primarily on performance indicators that are seen as positive changes. The full range of potential impacts may not be taken into account. In most cases, without an expected improvement against the status quo, the rationale for an investment does not exist. A complete and realistic benefits plan would include undesirable changes or impacts. Revisions were made to the benefits measurement guide in 2017 in an effort to keep benefits evergreen and allow “disbenefits” to emerge.

In terms of timeliness, it is common practice to give users time to familiarize themselves with a new IT system before measuring the benefits derived. The Integras project sponsor, in its project close-out report of April 2017, coped with this issue which can be summarized as follows: measuring too early in the system’s evolution may provide an incomplete view of performance, yet as more time passes it becomes a greater challenge to establish a causal relationship between an IT system and performance given other changes that may occur in the work environment.

The evaluation observations and findings highlight the value and feasibility of immediate post-implementation measurement. Committing to monitor and report during the early days of a deployment might capture important feedback to provide early detection of challenges or unintended impacts such as those noted in sections 4.2.1 and 4.2.2. There is emerging literature suggesting that organizations make insufficient effort to monitor the human asset perspective such as the emotional state of employees and its associated impacts on productivity, corporate performance, and mental health.

In the transition from an implemented project to business as usual, it is unclear where the resources and responsibility would lie to monitor and report on early performance. Once a project is implemented, the project management office begins to wind down and transfer the ongoing maintenance to a functional owner. Reporting on post-implementation metrics such as system performance, tickets raised, or user satisfaction requires resources to gather and analyze the data.

When the Integras project submitted its close-out report in April 2017, three of the four intended benefits were reported as realized. The fourth benefit is measurable and will be reported in 2019. It relates to increased revenue, across several business lines (GST/HST, T1 and T2), resulting from the automation of certain manual processes. The bulk of increased revenue is expected to come from the large file business line onboarded in 2015 through 2016. As noted in section 4.1.1, GST/HST program results are available to provide an indication of partial performance. It may be worthwhile to include more measurable, timely indicators in future project close-out reports.   

5. Conclusion

The CRA has mature structures and processes that work quite well. Within the organization, there are many actors and forces doing the job intended. At times, they are at cross-purposes with each other or result in unintended impacts which may go unmeasured or undetected. Knowledge of these various impacts can be enhanced through better integration and collaboration along the compliance continuum and within governance structures.

The horizontal perspective of this evaluation and its report puts the GST/HST Audit Program performance in context by seeking a greater understanding of performance results. This perspective generates important insights and lessons which can be applied against the governance of future projects and organizational changes.

In examining program results, the following key findings emerged with respect to a robust performance measurement and reporting system which can shed light on both desired and undesired results:

• the GST/HST Audit Program results suggest productivity declined after the introduction of organizational changes;
• the emphasis on audit quality is competing with traditional targets and performance indicators; and
• performance indicators reinforce individual program mandates and do not provide insight on multiple obligations of the same taxpayer (filing, reporting, and payment).

With respect to the availability of sufficient and timely information which can impact the CRA’s ability to achieve its results and compliance objectives, the following key findings emerged:

• GST/HST auditors had access to timelier and a greater amount of information with the introduction of the new case management system; and
• the inconsistent performance of the new case management system limited GST/HST auditors’ ability to access and use the information.

In examining what contributed to the observations noted above, the following key findings emerged with respect to the design and functioning of governance structures:

• oversight focuses on project management metrics while aspects of product quality receive less attention;
• the adoption of Agile strengthened the collaboration between business and IT; and
• there are post-implementation performance gaps based on the monitoring and reporting approach for major projects.

6. Recommendations and management response

One of the benefits to a horizontal perspective is that it can highlight the need for targeted changes rather than transformational ones. If implemented in a coordinated manner, these changes could have fairly large effects across the system. 

The GST/HST case study suggests that unmitigated risk may exist in the period prior to a major IT deployment and immediately after. This can largely be addressed through modifications to governance, roles and responsibilities, communications, and reporting.

One recommendation has been developed for GST/HST program-specific reporting to support comprehensive measurement of results and increased collaboration.

1. The CRA should articulate a plan to supplement the Taxes Earned By Audit performance indicator with other measures and consider their impact on horizontal compliance.

Management response

Domestic Compliance Programs Branch will develop a number of internal performance measures for consideration to supplement existing measures to provide a more comprehensive measure of program outcomes. The proposal will detail potential indicators, risks and costs associated with their adoption, along with proposed implementation strategies and timelines. Following the development of a proposal, suggested program outcomes will be piloted across various GST/HST program areas, prior to full implementation across GST/HST and Small and Medium Enterprises Directorate compliance programs.

Proposed timelines:

March 31, 2019: Development of plan including proposed indicators and detailed roll-out of pilot projects

March 31, 2020: Completion of pilot projects including close-out report

March 31, 2021: Implementation across GST/HST and Income Tax (Small and Medium Enterprise Directorate) compliance programs

One recommendation has been developed to support integrated corporate reporting.

2. The CRA should adopt a GST/HST horizontal performance indicator and also examine and implement ways to strengthen the organization’s ability to introduce more horizontally integrated performance indicators.

Management response

The Strategy and Integration Branch will develop and make recommendations on performance indicator(s) that could be used to track the percentage of GST/HST reassessed through audit compliance actions that is paid/collected, taking into account objections. The recommendation will also take into account data/system limitations.

The Strategy and Integration Branch will undertake this endeavor, which will include proof of concept and testing, in collaboration with impacted program branches: the Domestic Compliance Programs Branch, the Appeals Branch, the Assessment, Benefit, and Service Branch, and the Collections and Verification Branch. This assessment will highlight different options that could ensure a horizontal approach to tracking the performance of GST/HST audit outcomes and which could also be considered for a similar application across other business lines. Consideration will be given to relevant international best practices.

Proposed implementation date: the Strategy and Integration Branch will provide results and options at the Operations and Policy Committee by April 2019.

Two recommendations have been developed to reduce pre- and post-implementation risks of Strategic Investment Plan projects within CRA governance structures. Given the horizontal nature of governance and oversight, the action plans for recommendations 3 and 4 are mutually supportive.

3. The CRA should re-examine and modify as required its approach to the planning and subsequent post-implementation monitoring and reporting of Strategic Investment Plan projects and performance.

Management response

The CRA strives to continuously improve its approach to planning, oversight, post-implementation monitoring, and reporting of major projects. In addition to the changes described in section 4.3.3 which were not considered due to the timing of this evaluation, the following are examples of other improvements recently made:

• Introduced a framework for the analysis of the type of benefits achieved in the Strategic Investment Plan;
• Added a benefits indicator to the CRA’s portfolio overview to report on the continued achievability of the benefits of a project; and
• Added the User Experience Testing group to the Corporate Stakeholders to be consulted as part of the Major Project Investment Oversight Process.

Going forward, the plan for 2018-2019 for the Secretariat in the Finance and Administration Branch supporting the Resource Management Committee responsible to exercise oversight on the portfolio of projects includes:  

• Adjusting the governance of the Major Project Investment Oversight Process so that all discussion for projects with lifecycle costs >$20M take place at the Agency Management Committee rather than the Resource Management Committee. 
• Continuing to use Independent Third Party Reviews to help identify areas requiring additional attention from projects and oversight.
• Continuing to support the Project Management Community of Practice in the CRA by sharing best practices and promoting lessons learned.
• Ensuring resources in the Resource Management Secretariat get certified on change management.
• Completing the CRA self-assessment of the recommendations from the Goss Gilroy report on the lessons learned on the management and implementation of the Transformation of Pay Administration Initiative (Phoenix).
• Using the results of the above-noted Goss Gilroy self-assessment as well as this evaluation to explore and articulate for the Resource Management Committee’s consideration how the CRA’s oversight processes can be further enhanced to inform decision making and better recognize the role of users in the success of projects and the realization of benefits. Areas to be explored for potential improvements include user-centric design, project governance, change management, communication, and post-implementation monitoring.

4. The CRA should strengthen its governance to ensure product quality, system functionality and user experiences are considered.

Management response

With regard to quality and the user experience (UX), processes have evolved significantly. Agile testing practices piloted during Integras have been adopted as best practices and the CRA continues to mature its UX design practice as more and more applications become UX design-enabled. Examples of changes and innovations in testing include:

• Increased reliance on automated testing, as well as earlier and more frequent integration testing.
• Greater coordination between the Information Technology Branch and the Acceptance Testing Division, with user acceptance testers being embedded in development teams when feasible. As an example, Acceptance Testing Division user acceptance testers are currently working in collaboration with IT developers as part of the T1 Systems Redesign Project in a shared Integrated Unit Testing environment.
• Introduced Governance of the CRA UX design as a shared responsibility between the Public Affairs Branch, the Assessment, Benefit, and Service Branch, and the Information Technology Branch.
• Established a UX Design Board with membership from across the CRA.
• Updated the CRA’s Project Gate Assessment documentation requirements to include usability and accessibility artifacts to ensure IT-enabled projects are leveraging the UX design-enabled approach.

Going forward, the plan for 2018-2019 includes:

• revising the Terms of Reference for the UX Design Board to elaborate on the importance of the role of the committee and to strengthen its reporting relationship to the Digital Services Assistant Commissioner Steering Committee, the Resource Management Committee and the Agency Management Committee.

7. Acknowledgement

In closing, the evaluation team would like to acknowledge, recognize, and thank all CRA stakeholders for the time dedicated and the information provided during the course of this engagement. Auditors and collectors across the country and headquarters personnel shared their valuable insight and experiences with the evaluation team to enrich the analysis and interpretation of findings.

8. Appendices

8.1 Appendix A: Glossary