Language selection

Government of Canada / Gouvernement du Canada

Search

Sign in

Sign in

Local solutions audit

Corporate Audit and Evaluation Branch
June 2007

Executive Summary

Background: Canada Revenue Agency (CRA) systems that reside on the mainframe, or are deployed nationally in the distributed environment, are referred to as “national systems”. They are developed, modified and maintained within controlled processes. Headquarters (HQ) program branches, the owners of these systems, identify business needs and modifications for new and existing systems. Information Technology Branch (ITB) staff design and develop new, and modify existing systems. Development and maintenance of national systems require official approvals and a thorough testing process before being implemented.

“Local solutions” refer to applications or systems developed by employees or by local IT staff, for use at the local level in CRA Branches or Regions. These solutions are developed outside of Headquarters development processes for national solutions.
The increased power of desktop personal computers, and increased sophistication of standard software has made it possible for non-IT experts to leverage their use.

Local solutions are typically developed to generate productivity gains, automate administrative functions or to improve local interfaces with national line-of-business databases. In this audit a distinction was made between local desktop applications, referred herein as local applications, and Mainframe Macro Applications, referred to herein as MMAs. Both types are referred to generically as “local solutions”.

There are numerous motivations for employees and managers to develop and use local solutions. Managers are working to do more with less, meet program objectives, and provide better service to taxpayers. The Awards and Recognition program, one of the CRA's initiatives to encourage innovation and creativity, has recognized many employees, managers and teams for developing local solutions to improve program management. Employees at the local level have developed and use many of their own approaches and solutions to address specific needs.

Objective: The objectives of the audit were to assess the extent to which elements of governance of local solutions are in place, and to determine whether local solutions are in compliance with existing policies, procedures, and guidelines, or with generally accepted development methodology.

The examination phase was conducted from January 2006 to September 2006. Headquarters program branch debriefings occurred in November 2006.

The audit was conducted in accordance with the International Standards for the Professional Practice of Internal Auditing.

Conclusion: Local development is occurring within HQ functions and regional operations. The benefits of locally developed solutions are not being quantified in a documented or systematic manner. Considerable anecdotal evidence points to improved effectiveness in terms of better analysis of data, and consistency of fact gathering due to the ability of MMAs to “mine” data from mainframe systems. Efficiencies are gained due to the capability of MMAs to expedite repetitious tasks, reduce keystrokes, or to assist in the analysis of complex and voluminous data.
The reduction in keystrokes and automation of repetitive tasks also provides improved functionality for employees with disabilities. There is considerable dependence on MMA use in some workflows.

There are also significant risks associated with the development and use of local solutions, in particular MMAs, relating to data security, data integrity and potentially, system performance. Given these risks, best practices specify that local solutions development and use should be governed within a framework that includes a national inventory of existing solutions; policies that define requirements for security, privacy, language, and accessibility; and System Development Life Cycle (SDLC) standards.

The governance framework used in the Agency to manage the development and use of local solutions is not complete and requires significant improvements.

Compliance with existing policies, procedures, guidelines and accepted development methodology is low in areas including functional approval, registration, technical security reviews, and development and testing in a non-production environment.

Some areas of the Agency have recognized the need to improve the management of local solutions. A few basic elements of a governance framework such as inventories and standard processes for the development and roll out of local solutions exist in the Pacific region and in IRC West, within Taxpayer Services and Debt Management Branch (TSDMB). This is particularly notable given the importance that MMAs have taken in the management of debt by TSDMB.

To better manage local solutions, the Agency needs to address the significant risks identified in the audit and to implement good practices across branches and regions.

The challenge for CRA is to extend full due diligence over local solutions to mitigate the risks while supporting operational innovation .

Action Plan: ITB is developing an action plan and implementation strategy that will deploy an enhanced governance framework surrounding the development, design, and deployment of local solutions. This new framework will balance the need for a tighter end-to-end governance regiment to more effectively mitigate risks, while enabling and supporting the creativity and innovation of local development.

The framework will be introduced in two phases. The first phase, specifically addressing the issue of MMAs, will be finalized in the third quarter of 2007, while the second phase, addressing the broader issue of non-macro local solutions, is scheduled to roll out in the fourth quarter.

Given the cross-branch, cross-regional nature of the local solutions space, and the necessity to strengthen the compliance continuum in all impacted areas, much effort will be needed to ensure that ITB's action plan receives wide communication and penetration. To this end, it may be necessary to form an ad hoc local solutions working group to obtain the broadest possible exposure.

Some specifics of the action plan are:

Introduction

Canada Revenue Agency (CRA) systems that reside on the mainframe, or are deployed nation-wide in the distributed environment, are referred to as “national systems”. They are developed, modified and maintained within controlled processes. Headquarters program branches, the owners of these systems, identify business needs and modifications for new and existing systems. Information Technology Branch (ITB) staff design and develop new, and modify existing systems. Development and maintenance of national systems require official approvals and a thorough testing process before being implemented. Controls over data security are typically stronger within the central mainframe environment.

“Local solutions” refer to applications or systems developed over the years by employees or by local IT staff in CRA Branches or Regions, for use at the local level. Local solutions have been and continue to be developed and used in most Branches and Regions. These solutions are developed outside of Headquarters development processes for national solutions. Local solutions are typically developed to generate productivity gains, automate administrative functions, improve local interfaces with national line-of-business databases, or to address specific needs.

In this audit a distinction was made between local desktop applications, referred herein as local applications, and Mainframe Macro Applications (MMAs). Both types are referred to generically as “local solutions”. Local applications are generally built by local IT staff on a fee-for-service basis or sometimes by other CRA employees in the Branches and Regions. Local applications do not generally access mainframe data directly, but may still provide information for administration and management decision-making.

MMAs are generally built to provide a user-friendlier interface for employees accessing data in mainframe databases. MMAs augment the functionality of traditional mainframe systems and are used and developed to automate and speed up certain routine tasks conducted by CRA employees. They are developed in Headquarters branches and local field offices. By using an MMA, an employee can access mainframe screens with fewer manual keystrokes, and can copy large volumes of data from these screens - a process referred to as “screen-scraping”, to local data storage devices.

Focus of the Audit

The objectives of the audit were to assess the extent to which elements of governance of local solutions are in place, and to determine whether local solutions are in compliance with existing policies, procedures, and guidelines or with generally accepted development methodology.

The scope of this audit was national. It included an analysis of random samples of local applications and MMAs, selected from most branches and regions, to obtain information and measure compliance. The audit sample consisted of 92 MMAs and 111 local applications, extracted from available sources in the spring of 2006.

The local applications samples were drawn randomly from an aggregation of the numerous inventories of local applications that the audit could find/identify. The aggregation contained well over twelve hundred applications, some of which (web-applications and some smaller purchased applications) were not within the scope of this audit. But there were a large number of local applications, in use and being developed that require governance.

Anecdotal information suggested a national population of over 500 MMAs, but there was no national inventory of any kind from which to draw a sample of MMAs. Therefore MMA audit samples were derived by randomly selecting UserIDs executing higher than normal transaction rates on the mainframe, from Host Technology Management (HTM) statistics.

Questionnaires were issued to users and developers of the audit samples, and the responses gathered and analyzed. File reviews and interviews were also conducted with ITB and functional headquarters managers, and with managers and employees in field operations who develop and use MMAs and local applications.

Although Canada Border Services Agency (CBSA) information systems still reside on the same infrastructure as those of the CRA, CBSA was not included in the scope of this audit.

The audit was conducted in accordance with the International Standards for the Professional Practice of Internal Auditing.

Findings, Recommendations and Action Plans

1.0 The Strategic Benefits and Risks of Local Solutions

There is considerable anecdotal information indicating that local solutions, especially MMAs, provide significant productivity benefits for users. Although quantitative data was unavailable, the audit team was made aware, throughout interviews and tests, of benefits including:

MMAs add the additional benefits of:

There is considerable dependence on local solutions use in some workflows, and numerous awards and recognition have been given for local solution development.

There are also significant risks, but documentation of these risks at a strategic level was not evident. Some MMAs are large, complex applications involving activities such as drawing data from a host of databases, facilitating taxpayer enforcement actions, issuing letters to taxpayers, updating mainframe database information, and more. Hence there are risks to data integrity, especially when consistent and proven development methodology is not in use.

There are also risks to data security given the facility that MMAs provide to screen-scrape and download large amounts of protected data. Although employees still require to be assigned the mainframe profiles in order to obtain access to mainframe data, the same profiles now provide the capability to download large volumes of client data, and to store and transport this data on storage devices such as floppy diskettes, Universal Serial Bus (USB) memory sticks, CD's, DVD's, or any other portable device.

Seventy-eight percent of the MMA sample respondents said that they download data from the mainframe, but the audit did not investigate the media on which this data is stored. CRA FAM Security policy Chapter 18 indicates protected information that is stored on portable storage devices is to be encrypted with an Agency approved encryption algorithm. However, encryption is not automatic, requiring manual intervention and thus there is no assurance that protected data when downloaded to portable media is being encrypted per policy. Also there is currently no automated method to monitor if data is stored to portable media devices. Thus considerable risks exist.

Recommendations

Management needs to determine where local solutions, particularly MMAs, fit in the CRA's application toolkit. This can only occur after there is a clear understanding by both the functional branches and ITB, of the local solutions that exist, the purposes for which they exist, the risks associated with their use, and the benefits offered by this development approach.

More immediate steps should be taken to mitigate the inherent risks of MMAs, particularly in light of the capability to download large volumes of unencrypted client data to local data stores and easily transportable media devices.

Action Plan

The new MMA Development Policy is scheduled for release in June 2007. It will require the registration of any MMA that accesses data on an Agency mainframe data store. The MMA will be logged in the newly redesigned Local Application Repository (LAR), which will also capture tombstone data on the MMA such as:

The LAR, in conjunction with the requirement that all MMAs be registered, will be the foundation that enables the Regions and HQ Branches to more fully understand the extent of local solutions development and the utility of those solutions, and will allow the Agency to closely control and mitigate any associated risks. By having a single, central repository maintained by ITB, and enhancing its effectiveness by enforcing registration, the LAR will become a comprehensive national inventory of existing local solutions.

This audit correctly notes that MMAs inherently introduce risks, notably in the instance where data at rest is unmanaged. The Terminal Services Platform (TSP) rollout will help resolve this, as data (large or small) will not reside on local infrastructures.

The IT Security Services Division in ITB will undertake a review of MMAs that have the functionality of downloading large volumes of data to evaluate if appropriate security measures are in place. In addition, within the past year, IT Security has certified encrypted media (DriveMate and eNova) and PointSec encrypted software has been procured and will be made available by March 2008.

2.0 Existence of and Compliance with a National Governance Framework

According to best practices found in industry literature, local solutions development and use should be governed within a framework that would include items such as a national inventory of existing solutions; defined and communicated roles and responsibilities of the various participants and stakeholders; policies that define requirements for security, privacy, language, and accessibility; standards and procedures to guide the System Development Life Cycle (SDLC), including initiation, approval, development, implementation and use of local solutions, and post-implementation monitoring. The audit examined whether these elements of a framework were in place and whether the solutions developed were in compliance with the framework.

Overall, the audit found that efforts to build a governance framework have not kept pace with the number and capabilities of local solutions, and identified few cases where there was compliance with it or best practices. The audit findings and recommendations on governance are provided in more detail in the sections that follow.

Internal Audit (IA) acknowledges that the risk posed by some local solutions may be insignificant, as such not warranting full-fledged project planning, formal cost benefit analyses, HQ function oversight, and approvals. IA did not attempt to define significant and non-significant development, but the distinction needs to be made when determining the level of control and governance required.

2.1 Creating and Maintaining a National Inventory of Local Solutions

A complete, accurate, up-to-date and available inventory containing all registered and approved local solutions is currently not in place. Such an inventory should contain documentation for each solution including but not restricted to: the purpose, benefits, risks, program within which it is used, population of users, developer, and authorizing Branch. This information would be useful to users looking for a potential “tool”, and to developers, prior to undertaking a new development, in order to reduce duplicate development.

Local development has been occurring since the advent of desktop computing in the CRA. By the late 1990's, MMAs had begun to be developed by users as more
user-friendly and productive interfaces to client data in various mainframe application systems, particularly in Pacific Region where Vancouver TSO and Surrey Taxation Centre were two locations undertaking considerable MMA development.

There have been several corporate efforts to inventory local solutions. For example, in 2003, the A9 review by senior management of the managed distributed environment, mandated that local applications be inventoried. The resulting “Application list” was incomplete. Another effort in late 2003 to quantify local data holdings associated with local IT Solutions, including those resulting from MMAs, also produced incomplete results.

In 2003, the Pacific Regional Assistant Commissioner mandated Pacific managers to inventory all local solutions, and indicated sanctions would apply for non-compliance. Pacific Region now has available on the Intranet a Regional Definitive Software Library (RDSL) of local applications, and also maintains a separate offline MMA inventory.

In late 2004, in order to register new proposals in addition to already existing solutions, Development Centre in ITB initiated the Local Solution Development Project (LSDP) Intranet site. However the site's existence and the need to comply with its requirements were not well communicated. Subsequently the LSDP site was renamed the Local Application Support Project (LASP), and the registration process was named the “Local Application Registration and Repository” (LARR) process. The LARR contains many solutions that have been registered, but it is incomplete.

The audit noted several weaknesses in the registration process, including a lack of verification of functional approval, lack of communication to developers when registration failed due to missing or inaccurate information, and a lack of instruction on how the various links on the site should be used.

Commencing in mid-2005, Taxpayer Services and Debt Management Branch (TSDMB) sought to inventory MMAs in use in their operations. Since then, the End User Application Development Section (EUADS or also referred to as Integrated Revenue Collections-West or IRC-West), within TSDMB has inventoried the MMAs they have developed. Nonetheless, audit tests conclude that the majority of MMAs in use in TSDMB programs are not registered in the LARR national registry. Furthermore, no other Branch has a complete inventory of the local solutions used in their programs.

There is significant non-compliance with the requirement to register all local solutions, a necessary step in building a national inventory. Audit testing of samples from throughout the Agency showed that of the 111 local applications sampled, 67% of the HQ-developed, and 77% of the regionally developed local applications were not registered. Of the 92 distinct MMAs sampled, 31% of the HQ-developed and 87% of the regionally developed MMAs were not registered. It should be noted the majority of the HQ-developed MMAs that were registered were developed by IRC-West.

Recommendations

ITB and functional branches need to work together to develop and maintain a comprehensive national inventory of existing local solutions that includes appropriate documentation and is made available to appropriate stakeholders.

Clear guidelines and procedures should be developed and communicated by ITB to support the registration of new local solutions for inclusion in the national inventory, including verification that new registrations have been approved by the function.

Action Plan

Working with other HQ Branches and the regions, ITB is redesigning the LAR to enhance its capabilities as the comprehensive national inventory of existing and in-development local solutions. The LAR will be more utile than it is at present, and will include the capability to track an MMA in its development cycle, and to allow business clients to sign-off on MMA deployment. Release of the redesigned LAR is scheduled for June 2007.

ITB, in cooperation with, and using best practices in areas such as IRC West, is strengthening and reissuing the MMA Development Policy, and reissuing new, more comprehensive Standards for Development by June 2007. A Local Solutions Policy will be released in the fourth quarter of 2007.

2.2 Existing Governance Documents and Guidelines

The existing national governance model to guide local development generally defines the respective role of all stakeholders, including local offices, regional offices, functional branches, the appropriate areas in ITB including Information Technology Protection Centre (ITPC), and in Finance and Administration (F&A – Security Directorate). However, it focuses on the IT governance over development, and provides insufficient detail of the other areas that should be involved in the process. In particular, functional governance needs to be better defined. The governance model document includes some contact names but others are either missing or out-of-date. As a result, prospective developers do not know whom to contact. The document also does not reflect recent CRA organizational changes.

The various documents that generally cover the IT and security issues that should be considered when developing and implementing MMAs and local applications are referred to as “guidelines” instead of “standards”. As such, developers may assume it is not necessary to adhere to them. A stronger title such as National Standards for Local Development would be more appropriate and may encourage stronger compliance.

The various documents that comprise the existing governance model have not been approved. Also, responsibility for ensuring compliance with the governance and guidelines has not been clearly assigned.

Recommendation

The “National Standards” should be updated where appropriate and approved, and compliance to these standards should be enforced.

Action Plan

ITB, in cooperation with, and using best practices in areas such as IRC West, is strengthening and reissuing the MMA Development Policy, and reissuing new, more comprehensive Standards for Development by June 2007. Working with its partners, ITB will develop an oversight regimen to ensure compliance by September 2007.

2.3 Processes for Functional Oversight of Development, Review and Use

Processes for functional review and approval of local solutions development and use need improvement. Industry best practices established for end-user computing and end-user solution development require a clear statement of respective roles and responsibilities. HQ functional branches are also responsible for program funds allocation. As such they should exert oversight in terms of potential benefits, costs and risks, proposed business functionality, and approval of significant local solution proposals. Without functional oversight there is an increased risk of redundant, unnecessary or unauthorized development, less opportunities for sharing useful solutions, and less control over budget allocation. The expected benefits and costs of significant new development efforts should be described and tracked, so that senior branch managers are aware of what is being spent on local development.

The audit sample contained local solutions in programs belonging to most Headquarters functional branches. Based on the sample, the audit found that the roles and responsibilities for HQ functional review and approval of development proposals have not been assigned in most branches. With the exception of evolving initiatives in TSDMB, there is no national functional (Headquarters Branch) oversight in place to understand benefits, costs and risks, and review and approve local development, or to consider whether other options exist or should be considered.

TSDMB drafted a business governance process for MMAs only, including the creation of the IRC-West team in 2005 to manage MMA development. Although it continues to evolve, there is a process whereby MMAs developed by IRC-West are reviewed by functional experts and by the appropriate areas within ITB. Furthermore, development costs are tracked and known. Notwithstanding, other non-IRC-West MMA development that does not follow this process is occurring within TSDMB programs.

Compliance Programs Branch (CPB) has the Compliance Systems Access Request (CSAR) process, whereby prospective or proposed changes to existing CPB systems are reported, reviewed and prioritized. However the audit noted this process does not directly address situations where local development is already proceeding, or locally developed tools are already in use.

Analysis of MMA audit sample responses indicated that in only 13% of the samples was HQ input requested regarding possible alternatives to local development, such as making modifications to existing systems. Furthermore, while local management approval was sought in at least 77% of the cases, HQ function approval was sought in 29% of the cases. A cost-benefit analysis report was prepared in only 5% of the cases.

Analysis of the local application audit sample responses indicated that HQ input regarding operational needs, or for possible alternatives to local development, such as making modifications to existing systems, was requested in only 18% of the samples. Furthermore, while local management approval was sought in at least 86% of the cases, HQ function approval was sought in only 13% of the cases. A cost-benefit report was prepared in only 11% of the cases. In 72% of the cases, maintenance costs were not known or tracked.

Local solutions are often developed to address a lack of functionality in existing national applications. However, there are no indications that the lack of functionality that is being addressed by local solutions is being communicated or considered for inclusion into existing or new national applications and systems. Therefore, there is a risk that changes to the national applications may not be addressing all of the users requirements, because the need has not been made known to the functional Branch.

Recommendations

Headquarters functional branches should define, document, and approve internal processes with respective roles and responsibilities for business approval of proposals, and for business oversight of development including the requirements to perform cost-benefit analyses and to identify and mitigate risks.

Responsibilities within functional Branches, including contact names when assigned, should be made known to prospective developers.

Headquarters functional branches should ensure the functionality being addressed by local solutions is considered when defining requirements for existing and new national applications and systems.

Action Plan

ITB will work with its partners to develop and enable the processes necessary for effective HQ approval of local solutions' deployment. An approval and monitoring process will be formulated by June 2007. The newly redesigned LAR will track functional approval, capture contact and sign-off authorities, and provide a repository where requirements can be identified and considered in enterprise-class development. The LAR will be available in June 2007.

2.4 Systems Development Life Cycle (SDLC) Process

A standard SDLC methodology is not being consistently used. SDLC principles require that policies and procedures for development be in place and communicated. These policies and procedures should cover requirements for documentation, testing, reuse, and audit trails when client data is accessed. Other issues that should be included in such policies and procedures include the requirement to develop and test in a
non-production environment, and to conduct acceptance testing on behalf of the client. In regard to distribution of new products or updates to existing products, rollout should be controlled so that only the CRA employees authorized to use the new versions receive them.

Overall for all audit samples, documentation was weak. Forty-six percent of the respondents for the MMA samples and 66% of the local application sample respondents indicated no documentation was available describing the purpose and general use of the local solution. The results of development testing were not available for 76% of the MMA audit samples and 94% of the local application samples.

Some elements of SDLC principles are in place and complied with at local levels, but at the national level, only TSDMB, and specifically IRC-West has initiated some best practices regarding MMA development. For example, they have begun to utilize the services of Acceptance Test Division (ATD) for 11 IRC-West developed MMAs. Despite the additional rigour utilized by the IRC-West development processes, there were still considerable numbers of errors found. Nonetheless, it is notable that of all MMAs in use in CRA, only these IRC-West developed MMAs have undergone ATD testing.
This raises the question of what ATD would find if they tested MMAs that have been developed elsewhere, with possibly less stringent development methodology.

IRC-West has developed some local internal procedures and methodology to add robustness to their products. These include the Emergency Response and Operational Strategy Plan (EROSP) and processes to encourage reuse. IRC-West has developed MMA modules that meet the requirements outlined in the governance framework for reuse. However, these are not yet fully implemented and are not shared with other developers.

Rollout and access to MMAs is inconsistent, with some areas having more stringent distribution processes than others. The MMA development framework guidelines require rollout and access to be conducted via a controlled national process developed by ITB. This process is only being used to distribute those national MMAs that
IRC-West has developed on behalf of TSDMB.

The sample analysis also indicates that other program areas do have various levels of MMA development, testing, approval, and distribution processes in place, but their methodology is not consistent, nor as stringent as the IRC-West processes. Most of the audit samples from these areas also have not undergone functional review or approval.

The lack of a consistent SDLC increases the risks of inadequate documentation, data integrity and other errors, and improper version control.

Recommendation

The “National Standards” should be strengthened to promote a consistent SDLC methodology for local development, and include a process so that all local solutions are properly developed, tested, documented, rolled out, and versions controlled.

Action Plan

The introduction of a Quality Process will guide local development through an SDLC commensurate with the risk that that development poses. With the rollout of TSP, the publication of all applications will become necessary and will include a much more rigorous testing and release cycle. Closing this loophole in the current SDLC will identify issues and incompatibilities earlier in the cycle and will solidify the strengths of aligning to National Standards.

A comprehensive test strategy will be engineered, as will an effective testing platform. Testing best practices, together with proper Release, Change, and Configuration Management processes and procedures will help support more risk-controlled local solutions. Implementation of the testing platform is scheduled for September 2007.

2.5 Reviews for Technical Compliance

Several technical reviews on new and existing local solutions are conducted within ITB, to ensure that they conform to national guidelines. These reviews are conducted on only those solutions that are reported to Development Centre, and are not well integrated.

Development Centre in ITB conducts a preliminary check after solutions are registered on the LARR. This check is done to ensure compliance to governance framework guidelines.

The Integrated Revenue Collections Division in ITB-Solutions (IRC-Solutions) also reviews the national MMAs developed by IRC-West, against recommendations in the Development Centre Guidelines for MMA development. This process is not linked to the LARR process. It should be noted that IRC-Solutions considers their review to only constitute technical advice and guidance to the developer, but not technical approval.

The results of these various reviews are not well integrated with each other, and with the security review conducted on MMAs by ITB Security, IT Protection Centre (ITPC) (see Security section 2.6.1). In addition, there is no single tracking system in place where the status of any solution undergoing these reviews is recorded. This makes it difficult for developers or others to determine which reviews any particular solution has undergone.

Recommendations

The process for all ITB technical reviews should be integrated, streamlined, embedded in the “National Standards” and communicated. The “National Standards” should include ITB roles and responsibilities for the sequence of review processes taking place within ITB.

A tracking system should be provided to display the status of Headquarters review (business or ITB) for any solution submission, and the status of new development.

All solutions that are currently in use that have not previously been reviewed should undergo the required reviews prior to publishing in the Approved Applications listing in the LARR repository.

Action Plan

ITB will integrate, streamline, and communicate its technical review procedures, and embed these in the LAR website. The LAR itself will enable the sequencing and tracking of review processes within ITB, as well as those of HQ branches.

Solutions currently in use that have not previously been reviewed will be subject to the same rigours as solutions under development.

2.6 Security Policies

The CRA, Finance and Administration Manual (FAM) Chapters on Security contain numerous security policies that inherently apply to local solutions or corresponding data, transmittal, and storage. There is low compliance with many of these policies, security reviews are fragmented, and there are significant security risks.

2.6.1 Technical Security Reviews

FAM Security Volume, Chapter 20, Security Risk Management-Information Technology Threat and Risk Assessments, states that Technical Security Reviews (TSRs) are to be completed early in the development lifecycle for local/regional/branch applications, by application managers, to ensure prevention and control measures are implemented for the Agency's networks, systems and applications. Various communiqués were distributed by ITB in 2004 and 2005 to AMC members, and to IT Directors and IT Advisors stating that it is necessary to complete and submit a TSR for all existing local solutions. However, the audit found that:

There were other issues with TSR procedures as well:

2.6.2 Audit Trails

FAM, Security Volume, Chapter 22 states, “All accesses to client identifiable data must be logged (audit trail), unless exempted by the Security, Risk Management and Internal Affairs Directorate (SRMIAD) through the Threat and Risk Assessment (TRA) process”. Regardless of the security measures provided for this data, there is still a requirement that audit trails be kept, identifying the employees who accessed this data.
Seventy-eight percent of the MMA sample, and 10% of the local application sample questionnaire responses indicated taxpayer data is downloaded from the mainframe. Audit trails are not being kept on accesses to locally stored taxpayer data. This is currently the case for all MMAs in use. As such CRA cannot provide assurance that it can identify all employee access of taxpayer data.

2.6.3 Testing MMAs in the Production Environment

Conducting development and unit testing in the production environment is risky, due to the possibilities of damaging the integrity of live data or producing unintended live outputs. Therefore, the CRA security policy requires waivers to be issued by F&A Security in cases where testing is done in the production environment.

Audit testing indicated that there is no separate development and testing environment for local solutions in the CRA at present. Therefore developers of MMAs have no option but to develop and unit test MMAs in the production environment. There was no evidence provided to IA that waivers were issued by F&A Security for MMAs tested using client-identifiable data on the production environment.

2.6.4 Business Continuity and Disaster Recovery

CRA security policy requires that business continuity plans, including disaster recovery plans be in place for systems that support essential business functions. This requirement is unaddressed for most local solutions upon which business operations are heavily dependent. IRC-West has begun to consider this issue, with the development of the Emergency Response Operational Strategic Plan (EROSP).

Recommendations

The “National Standards” should summarize or reference the security considerations that apply, and the procedures that should be adhered to when undertaking local development, such as the requirement to complete a TSR and the process for submitting them. Sanctions should apply for non-compliance.

Senior management needs to ensure compliance with the policy requiring waivers when development and testing is being conducted on the production environment with live data.

ITB should consider a solution to address the lack of audit trails on accesses to taxpayer data stored outside the mainframe environment.

The process for ITPC security reviews needs to be clarified, and communicated, and steps need to be taken to ensure all local solutions undergo these reviews.

The “National Standards” should require that local solutions be included as part of business continuity plans, where local solutions support essential business functions.

Action Plan

ITB will implement a mandatory Quality Process that will guide a developer through a security regime commensurate with the risk that a local solution poses. For solutions of sufficient extent, the Quality Process will demand that business continuity plans be completed and registered in the LAR.

The development of an effective testing platform will render obsolete the dangerous practice of developing and testing in the production environment.

A 2-year cycle review of the Information Security Policies occurring FY 2007-2008 by the Security Risk Management and Internal Affairs Directorate (SRMIAD) of F&A Branch will allow for greater re-enforcement of existing policies and better refinement of roles and responsibilities vis-à-vis local solutions applications. The Security Awareness program will further stress existing policies, standards, and operational procedures regarding the creation of MMAs. In addition, F&A Branch's ongoing inspection and review program will also enable the SRMIAD to identify MMAs and other applications being developed without proper standards, certification, accreditation, or without being risk assessed.

A 5-year audit trail modernization project has been initiated. SRMIAD has started a preliminary analysis of requirements for the redesign of the National Audit Trail Modernization System. The current content of all existing audit trails are being documented in detail and stakeholder requirements are being determined for a revised system.

2.7 Compliance with Policies on Official Languages, Adaptive Technology, and Privacy Impact Assessments

Fifty-seven percent of MMA sample respondents and 66% of local application sample respondents indicated that developers had not considered the Adaptive Technology and Official Languages (OL) policies when developing these local solutions.

In regards to Privacy Impact Assessments (PIA), the broad variety of mainframe databases accessed for data, utilizing MMAs, suggests that Preliminary PIAs should be considered. However, none of the respondents to the audit sample questionnaire for MMAs and only 4% for the local applications indicated that a PIA had been completed and submitted.

A blanket Preliminary PIA (PPIA) has been completed by IRC-West for the MMAs they have developed, but the committee that reviews PPIAs for possible requirement of a formal PIA has not been approached as of December 2006.

A single point of reference does not exist for relevant policies and guidelines (Security, Adaptive Technology, Official Languages, PIA, Sustainable Development, etc.), for developers to be aware of when developing local solutions. Such a tool would be helpful to developers to assure that all of the relevant policies have been considered. The Local Solutions governance model and supporting documentation drafted by Development Centre is a good start as a national standard, but does not list all of the policies that developers should consider.

Recommendations

The “National Standards” should include specific references to policies on Official Languages, Adaptive technology, and Privacy Impact Assessments that need to be considered when undertaking local development.

Action Plan

The new Standards for Development will reference OL policies and Adaptive technologies. These Standards are scheduled for release in June 2007. The Quality Process will drive the requirement for PIAs.

2.8 Communication of Development Standards and Other Important Information

Communication of Development Standards and other important information can be improved.

Recommendation

Improvements should be made to communications channels so that all information from Headquarters, (both business and technical), that affects local solutions is received by those who need it.

Action Plan

Work on a communications strategy is continuous. Given the cross-branch, cross-regional nature of the local solutions space, much effort will be needed to ensure that the action plan receives wide communication and penetration. To this end, it may be necessary to form an ad hoc local solutions working group to obtain the broadest possible initial exposure, and to develop sustainable communication channels.

ITB (Production Assurance) currently publishes and maintains release management oriented information related to application releases and major infrastructure changes in its Branch Forward Schedule of Change (BFSC). Information on all IT infrastructure changes is found in the Change Management system within Infoman. While these reports are currently distributed widely, ITB is examining the utility of extending the distribution more widely.

2.9 Monitoring

In the CRA's traditional development approach there are many controls exerted over the development process. As a result of these numerous controls, business functionality is known, risks to security and data integrity are mitigated, and infrastructure and mainframe performance concerns are relatively well known and documented.

Local solutions, by their very nature are not subject to the same controls. Given the significant tasks that some local solutions perform, it merits the attention of management to not only identify the functionality and the benefits these offer, but also the risks.

Local applications are often utilized in administrative or Human Ressources processes and often handle employee data, or generate information used by management for decision-making purposes. They too are important and should be subject to the same control processes as any application or MMAs that access or process client data. MMAs warrant attention for numerous functional, technical, and security reasons outlined elsewhere in this report.

Specifically in regards to MMAs, although the execution of MMAs that invoke high transaction levels on the mainframe has been monitored informally by ITB - Host Technology Management (HTM), and by an employee in TSDMB, utilizing the HTM statistics showing the number of transactions processed each hour, the Agency has no formal process in place to monitor MMAs.

In HTM, the focus of monitoring utilizing the HTM statistics is limited to identifying whether the execution of an MMA has caused any mainframe performance problems. Other MMAs might not invoke high transaction volumes, and hence not appear on these statistics. In these cases there is no capability to exercise any monitoring or oversight to determine the extent and purpose for which these MMAs are being used. These MMAs could still be significant in terms of the functions they perform, and therefore pose important risks.

Many executions of large, high volume MMAs occur during core hours, in contravention of guidelines issued by ITB intended to reduce daytime loads on the mainframe. Mainframe system performance and stability has not yet been significantly impacted by the use of MMAs. However, continued lack of governance over the development and use of MMAs could begin to negatively impact mainframe system performance and stability. The high transaction volumes, which occur as a result of MMA use, may result in the need to provide additional mainframe resources, or cause other concurrency, performance and stability issues.

Local solutions have an effect on the network and network servers as well. For example, the proliferation of local solutions on the network was a critical factor requiring ITB in quarters 3 and 4 of 2006-2007, to significantly reduce the scope of its Server Consolidation project.

Given the variety of uses being made of local solutions, there is a need to monitor local solutions activity and examine risks and benefits as mentioned elsewhere in this report.

Recommendations

Headquarters Branches and ITB should implement formal monitoring on the use being made of local solutions in terms of program functionality, and compliance with National Standards.

For MMAs, the Headquarters functional branches should consider using the existing HTM statistics as a starting point to help identify and understand who is using MMAs and ensure they are authorized and registered. They should also consider seeking more information on MMAs that do not appear on the current HTM report due to lower transaction volumes, but which may still pose significant risks or benefits.

For MMAs, ITB should implement more formal monitoring of their execution on the technical infrastructure and as well report on compliance with the National Standards.

Action Plan

An approval and monitoring process will be formulated by June 2007.

The expansion in functionality of the LAR will increase the Agency's ability to monitor the use of local solutions, the drivers behind their development, and overall compliance with National Standards. The mandatory registration of MMAs and the implementation of a Quality Process will ensure they are authorized for deployment, and have undergone an SDLC commensurate with complexity and risk.

ITB will re-examine its monitoring regimen to ensure that impacts on the technical infrastructure are understood and controlled.

Conclusion

Local development is occurring within HQ functions and regional operations. The benefits of locally developed solutions are not well quantified, but include efficiency, effectiveness and productivity gains. Some examples are automation of previously manual administrative functions, and improvements of local interfaces with national
line-of-business databases, which can also lead to reduction in the use of resources such as time and paper.

There are also significant risks associated with the development and use of local solutions, in particular MMAs, relating to data security, data integrity and potentially, system performance. Given these risks, local solutions development and use should be governed within a framework that includes clear ownership and accountability, a national inventory of existing solutions; policies that define requirements for security, privacy, language, and accessibility; and System Development Life Cycle standards.

Agency governance over local solutions development and use needs to be improved. There are elements of a national governance framework in place, but these are incomplete, have not been approved by stakeholders, and have not been well communicated. Compliance with these and with best practices is low.

Functional governance over local solutions development and use is limited. HQ functional branches are ultimately responsible for the systems, tools, and procedures that support their programs, and are the fund managers for budget allocations for their operations. As such, each Branch should have authority and oversight over the significant local solutions developed and used to facilitate achievement of their program goals. The costs of development of local solutions are usually unknown to the funding Branch.

There is a need to identify and inventory all significant local solutions in existence, the purpose, the population of users for each, and the benefits and risks. Significant solutions that have not undergone HQ functional, technical and security review as required by existing guidelines and policy, need to be reviewed. Local development, especially MMAs, needs to be better understood, the risks and benefits need to be quantified, and the positioning of local development in the spectrum of automated tools and systems needs to be clarified. Monitoring needs to be instituted.

Some areas of the Agency have recognized the need to improve the management of local solutions. A few basic elements of a governance framework such as inventories and standard processes for the development and roll out of local solutions exist in the Pacific region and in IRC West, within Taxpayer Services and Debt Management Branch (TSDMB). This is particularly notable given the importance that MMAs have taken in the management of debt by TSDMB.

Page details

Date modified: