Audit of Project Management Governance

Audit Report

Office of Audit and Evaluation
March 2015

Executive Summary

What we examined

Effective project management is critical for Shared Services Canada (SSC) to deliver on its mandate. To mitigate vulnerabilities in project management, the SSC Project Management Centre of Excellence (PMCoE) defined and implemented SSC’s Project Management Framework (PMF). A primary component of the PMF was the Project Management (PM) Directive.

This audit provides assurance as to whether appropriate systems, processes and controls for managing projects were in place at SSC to support the achievement of SSC’s mandate.

The scope of the audit included SSC’s project management systems, processes and controls, including the application of these systems, processes and controls from August 1, 2012, to October 31, 2013.

Why it is important

SSC’s organizational priorities include the maintenance and streamlining of information technology infrastructure, the launching of a single email solution, and the consolidation of data centres and networks. The achievement of these priorities will depend on the successful management of a variety of projects.

When SSC was created, approximately 1,500 projects were transferred to the Department. An extensive review process resulted in the identification of about 700 projects to be continued by SSC. The remaining projects were determined to be operational requests, were grouped with other projects, placed on hold or cancelled. The PMCoE has stated that 52 new projects have been started since August 2011.

SSC was established by transferring selected staff and resources from 43 other federal organizations in August 2011. Merging project staff and processes from many different organizations with varying levels of experience presented a challenge for SSC’s project management approach. There was a strong need to effectively communicate and implement this horizontal approach.

What we found

SSC’s PMF was compliant with the Treasury Board Policy on the Management of Projects; however, there were gaps in the PMF documentation, tools and templates in terms of completeness and consistency of information.

Roles and responsibilities for project oversight and approval bodies for all project tiers and accountability for project outcomes were documented in the PM Directive. Although some oversight was taking place, some of the oversight mechanisms were not functioning as intended.

There were issues with the Enterprise Portfolio System data completeness and accuracy. The system may not provide accurate and complete information for reporting and decision making to senior management or for adequate project monitoring and governance

Yves Genest
Chief Audit and Evaluation Executive

Background

1. The Government of Canada established Shared Services Canada (SSC) on August 4, 2011, to modernize how the federal government manages its information technology (IT) infrastructure in order to better support the delivery of programs and services to Canadians.
   
2. According to SSC’s 2013-14 Report on Plans and Priorities (RPP)^{Footnote i} , effective project management is critical in order to deliver on its mandate and is a key priority area of focus. SSC’s organizational priorities include maintaining and streamlining IT infrastructure, launching a single email solution and consolidating data centres and networks. As stated in the RPP, to mitigate vulnerabilities in project management, SSC’s Project Management Centre of Excellence (PMCoE) defined and implemented a Project Management Framework (PMF) in the spring of 2013.
   
3. When SSC was created, approximately 1,500 projects were transferred to the Department. An extensive review process resulted in the identification of about 700 projects to be continued by SSC. The remaining projects were determined to be operational requests, grouped with other projects, placed on hold or cancelled. The PMCoE has stated that 52 new projects have been started since August 2011. Based on information available, project values ranged from $800 to $146M.
   
4. SSC was created by transferring IT and internal services employees from 43 federal organizations. Merging project staff and processes from many different departments with varying levels of experience presented a challenge for SSC’s project management approach. This approach was based on a Plan, Build, Operate and Manage model. The model captured the key elements of the Department’s mandate to both operate and transform.
   
5. Reporting to the Senior Assistant Deputy Minister (SADM), Projects and Client Relationships (PCR), the PMCoE was the functional authority for project management at SSC. The PMCoE’s responsibilities included:
   • Establishing a project management culture across SSC;
   • Building and sustaining project management capacity in support of SSC’s mandate;
   • Engaging and supporting the SSC project management (PM) community;
   • Providing supporting tools, methodologies, practices and a level of expertise in PM; and
   • Providing oversight and support roles for both IT and non-IT PM across SSC.
6. The Treasury Board (TB) Policy on the Management of Projects applies to all Government of Canada projects. It defines a project as an activity or series of activities with a beginning and an end, required to produce defined outputs and realize specific outcomes within specific time, cost and performance parameters. The policy’s objective is to ensure that appropriate systems, processes and controls for managing projects are in place, at a departmental, horizontal or government-wide level, and support the achievement of project and program outcomes while limiting the risk to stakeholders and taxpayers.
   
7. In the synthesis report prepared by the Office of Audit and Evaluation, “What prevents large IT projects from being successful”, it was noted that a weakness in any project management component can result in long delays, cost overruns, scope creep and ultimately project management failure. A project management framework ensures the correct prioritization and co-ordination of projects.

Objective

8. The objective of this audit was to determine whether appropriate systems, processes and controls for managing projects were in place at SSC to support the achievement of SSC’s mandate.

Scope

9. The scope of the audit included SSC’s project management systems, processes and controls including the application of these systems, processes and controls from August 1, 2012, to October 31, 2013. This included:
   
   • All files, documents and data pertaining to projects managed by SSC in this timeframe.
   • Any internal or external reports or assessments related to project management governance and organizational project management capacity.
   • SSC’s PMF and all related documentation.

Methodology

10. During the conduct of the audit, we:
    • Interviewed senior management, project managers and technical experts;
    • Conducted project file walkthroughs;
    • Reviewed relevant documents, such as previous audits, government guides and policies with regard to project management, and SSC project management process documentation; and
    • Performed data analysis.
11. Field work for this audit was substantially completed by October 31, 2013.

Statement of Assurance

12. Sufficient and appropriate procedures were performed and evidence gathered to support the accuracy of the audit conclusion. The audit findings and conclusion were based on a comparison of the conditions that existed as of the date of the audit, against established criteria that were agreed upon with management. This engagement was conducted in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing. A practice inspection has not been conducted.

Detailed Findings and Recommendations

Project Management Governance

13. SSC adopted the TB Policy on the Management of Projects definition of a project as “an activity or series of activities that has a beginning and an end. A project is required to produce defined outputs and realize specific outcomes in support of a public policy objective, within a clear schedule and resource plan. A project is undertaken within specific time, cost and performance parameters.” SSC excluded all activities intended to sustain regular operations or systems. In addition, external projects that are subject to partner gating, and whose governance provides evidence of requiring SSC sign-off at each gate, were also exempt from SSC’s gating process.
    
14. The Project Governance Framework (PGoF) established five stages of project development: Idea Generation, Initiation, Planning, Execution, and Deployment & Closeout. Each project was required to produce documentation and obtain approval at each stage to move on to the next stage of work. These approval points were referred to as gate approvals. The PGoF defined the approval structure for gate and stage depending on the assessed level of complexity and risk of individual projects.
    
15. As part of the PMF, individual projects were to be assessed on their level of complexity and risk. Each project was required to complete a Project Complexity and Risk Assessment (PCRA). Projects with a planned cost under $1M were to complete a PCRA “Lite” and all others were to complete a full PCRA. The PCRA tool was composed of 64 questions to be completed by the project team and approved by the Executive Sponsor (a stakeholder in the PM process ultimately accountable for realizing the benefits and outcomes sought from the project). This assessment resulted in the assignment of a “Tier” classification, ranging from Tier 1 for small low-risk projects to Tier 4 for evolutionary and transformational projects. The PCRA levels did not directly correspond to the tiers. PCRA levels 1 and 2 fell into Tier 3 and PCRA levels 3 and 4 fell into Tier 4 (see Table 1 below). Projects under $1M were generally expected to be either Tier 1 or Tier 2 projects.

16. The resulting tier of a project determined the documentation, approvals and gating processes required, as described in the PGoF (see Annex B); the project was expected to comply with the PGoF as it progressed through the stages. Higher tiered projects require more oversight and documentation.

Management of Project Management Framework Documentation

17. The TB Policy on the Management of Projects requires Deputy Heads to ensure that a department-wide governance and oversight mechanism was in place, documented and maintained. We found SSC’s PMF to be compliant with the TB Policy on the Management of Projects.
    
18. SSC developed and implemented a PMF to direct and guide effective management and delivery of projects from start to finish. The PMF was documented and made available electronically through the PMCoE’s GCpedia wiki page. One of the PMF’s primary components was the PM Directive, which took effect on March 26, 2013. This directive set the standards for the management of projects within SSC. The project management structure set out in the PM Directive relied on three main phases: planning, execution and operations.
    
19. The PGoF, included in the PM Directive, described the stages and gating requirements based on project complexity and risk. The PMCoE was responsible for communicating the contents of the PM Directive and ensuring that the content and associated processes, templates and guides were managed and maintained.
    
     

     

20. The launch of the PMCoE GCpedia wiki to all staff occurred in March 2013. Updates to PMF documentation were also communicated through the wiki. However, stakeholders did not receive notification when updates occurred. Stakeholders indicated that it was challenging to determine what had been updated and that there was no communication to staff when updates occurred. In addition, it was not always clear as to what changes were made. Stakeholders advised that they have used outdated PMF information or templates as they were not aware of the updates, when they occurred or what had been update.
    
21. We reviewed the primary components of the PMF, such as the PM Directive and PGoF, as well as key tools, guides and templates that have been provided for project management by the PMCoE. We found several examples where the documentation lacked completeness and consistency of information.
    
22. The incompleteness and inconsistencies identified in the documentation may lead to a lack of understanding of processes, incorrect assessments of project tier resulting in reduced oversight and certain key documents not being used.

The Governance Structure

23. SSC’s PM Directive stated that all projects must follow the PMF and it provided a structure for project management governance across the four tiers. We expected that the PMF was functioning as intended for each of the four tiers. We found that there were inconsistencies in the application of the PMF.
    
24. The following committees had key project oversight responsibilities, as noted in their Terms of Reference (TOR) or the PM Directive, for the organization:
    
25. The project oversight and approval committees identified above all had approved TORs. These TORs, for the most part, set out their project-related responsibilities. The exception was the Senior Management Board (SMB) TOR, which did not include the board’s responsibility for making gating decisions on Tier 4 projects, as set out in the PM Directive and PGoF. In addition, we found that the TORs for these committees did not set out decision making procedures for committee proceedings.
    
26. We found the roles and responsibilities for project oversight and approval bodies for all project tiers and accountability for project outcomes documented in the PM Directive. The PM Directive outlined the project management and governance responsibilities for the SMB, Senior Project and Procurement Board (SPPOC) and Business Transformation Committee. The Investment Review Board and Director General (DG) Project Management Advisory Committee (DGPMAC), both indicated in the PGoF as having an oversight function and the Intake Review Committee did not have their roles and responsibilities defined in the PM Directive.
    
27. According to the PGoF, the SPPOC provides gating approvals for Tier 3 projects and gating recommendations for Tier 4 projects, while the SMB provides gating approvals for Tier 4 projects. However, a Tier 4 project obtained a conditional gating approval from the SPPOC and we found no evidence that the project went to the SMB for approval.
    
28. Reviews, as part of the oversight and approval mechanisms for Tier 1 and 2 projects, were not functioning as set out in the PM Directive and PGoF. The DGPMAC review was not occurring as identified on the PGoF; therefore, Tier 1 and 2 projects did not receive some of the oversight expected.
    
29. Oversight and approval mechanisms for Tier 3 and 4 projects were not fully functioning as intended. We noted that, as of June 2013, the SPPOC began properly enforcing the use of the PGoF and refused gate passage for projects that did not have the required artefacts and/or signatures in place.
    
30. We found weaknesses in the tracking of executive committee action items to ensure that these were being consistently addressed. The tracking of SPPOC action items was reviewed and we found that there was no single repository for the tracking of action items. The Executive Committees Secretariat and the PMCoE both maintained a register of SPPOC action items. However, both tracking systems were found inconsistent and did not always provide information on due dates and details to explain the work done to action the items.
    
31. The incomplete oversight identified for Tier 1 and 2 projects increases the risk of projects not following the PGoF, in whole or in part. The lack of clarity and information gaps in the PMF surrounding the various committees with project-related responsibilities, their interactions and the inconsistent tracking of action items may increase the overall lack of understanding with stakeholders, resulting in project delays, productivity losses, reduced cost-effectiveness and increased likelihood that projects may not be following the PGoF.

Project Information

32. SSC implemented the Enterprise Portfolio System (EPS) as its corporate project repository. Once fully implemented, EPS was intended to provide the ability to forecast costs, manage risks, issues and change requests, report project status, track records of decisions (ROD), store project artefacts, and generate reports and personalized views. Its objectives included providing project information in real-time, standardizing how project information was reported and recorded, providing decision makers with strategic insight to make investment decisions as well as supporting accountability and transparency.
    
33. We expected SSC to maintain complete and accurate project information for oversight, monitoring and reporting purposes. We found that EPS was an inaccurate and incomplete source of information for monitoring and oversight. Therefore, other sources of information were also used when reporting on the progress of a project. However, the SPPOC decided that all future ongoing portfolio reviews of projects presented to the Committee would be based solely on information provided from EPS.
    
34. EPS had only one automated control. The manual controls on EPS information included project reviews performed by the PMCoE at gate approval points; weekly delta reports generated by the PMCoE and compared with artefacts and RODs; EPS user training; and the roles and responsibilities stated in the PM Directive.
    
35. We selected a sample of 270 projects^{Footnote ii} to base our analysis of the data in EPS. We tested the completeness of the information in EPS. We found missing project information such as tier, stage, planned project cost and project status indicators. For example, almost 40% of projects had no stage and/or tier and a majority of projects did not have planned project costs entered in EPS. The missing information made it difficult to determine whether projects were following the appropriate project governance steps as project governance was determined from this information.
    
36. The PM Directive also required that all project artefacts and project status reports be stored and maintained in EPS. Many of the reviewed projects were found lacking some or all of the required artefacts in EPS.
    
37. We tested the accuracy of information by performing a number of tests to identify whether the project information recorded in EPS was reliable. We tested the PGoF threshold between Tier 1&2 and Tier 3&4. Projects with a planned project cost of over $1M should only be Tier 3 or Tier 4. We found a high instance (nearly 60% of projects) of missing planned project cost, and almost half of these projects had an assigned tier. This missing information prevented us from performing a full assessment. Nonetheless, even out of the limited sample of projects with a planned project cost of over $1M reported in EPS, three of 25 (16%) projects were identified as Tier 2, which implied a lower governance level than expected was being applied.
    
38. The PM Directive stated that the PCRA tool was “the definitive determination of a project’s tier”. Once completed, the tool provided a resulting “level”, from 1 to 4, for the project. During our completeness tests we found only 29 of 270 (11%) projects had a PCRA rating entered in EPS. In addition, based on the PGoF, we expected Level 1&2 PCRA scores to result in a Tier 3, and Level 3&4 PCRA scores to be Tier 4 projects. We found that 24 of 29 (83%) projects had a different Tier than expected.
    
39. According to the PM Directive, project managers have to follow the Treasury Board Secretariat (TBS) Guide to Executive Dashboards for each project. The TBS Guide identified that five metrics (cost, schedule, scope, risk and issues) were to be completed to produce an overall status level. Each metric had to be assigned a status of red, yellow or green.
    
40. We found that 224 of the 270 (83%) projects reviewed had at least one “not assessed” project metric. We found only 44 (16%) projects with all five metrics completed and an overall status.
    
41. Members of the senior management team stated that the information in EPS was not kept current and may not always suit their needs. SADMs mentioned that dashboards reports, produced from EPS, were unclear, contained inaccurate and non-current data and were not used for decision making. However, the SPPOC recently performed reviews of the entire project portfolio aimed at increasing the accuracy of information captured in EPS.
    
42. Oversight of project information in EPS was essentially a manual process. The impact of the issues with EPS data completeness and accuracy was that the system did not provide accurate and complete information for reporting and decision making to senior management, or for adequate project monitoring and governance.

Conclusion

43. The objective of this audit was to determine whether appropriate systems, processes and controls for managing projects were in place at SSC to support the achievement of SSC’s mandate.
    
44. We found that SSC implemented a PMF compliant with the TB Policy on the Management of Projects; however, there were gaps in the PMF documentation, tools and templates in terms of completeness and consistency of information.
    
45. Roles and responsibilities for project oversight and approval bodies for all project tiers and accountability for project outcomes were documented in the PM Directive. Although some oversight was taking place, some of the oversight mechanisms were not functioning as intended.
    
46. There were significant issues with the EPS data completeness and accuracy in that the system may not provide accurate and complete information for reporting and decision making to senior management, or for adequate project monitoring and governance.

Management Response and Action Plans

Overall Management Response

As the management of the PMCoE, we would like to thank the Office of Audit and Evaluation at SSC for all its efforts in conducting an internal audit of Project Management Governance at SSC. One of the PMCoE’s core principles is continuous improvement. When the audit was launched, we welcomed your team’s independent review of the work and the mandate that was set for us. We viewed this audit as an early opportunity for us to continuously improve upon the systems, processes and controls in place for managing the portfolio of projects at SSC.

The audit report contains findings regarding gaps between systems’ intended purposes and actual outcomes, and makes practical recommendations to bridge the gaps. We were already aware of some of the deficiencies when the audit was substantially completed on October 31, 2013. For many of these deficiencies, mitigation strategies were put in place as interim measures until we finalized a comprehensive improvement strategy targeting the implementation of version 2 of SSC’s Project Management Directive. We are now in a position to provide detailed action plans, below, addressing each recommendation.

Recommendation 1

The Senior Assistant Deputy Minister, Projects and Client Relationships, should address documentation inconsistencies, completeness and communication of revisions to stakeholders.

Recommendation 2

The Senior Assistant Deputy Minister, Projects and Client Relationships, should review and streamline the oversight responsibilities for all project tiers and ensure that they are accurately documented and functioning as intended.

Recommendation 3

The Senior Assistant Deputy Minister, Projects and Client Relationships, should implement and maintain one action tracking system that ensures consistent documentation, tracking and following up committee action items

Recommendation 4

The Senior Assistant Deputy Minister, Projects and Client Relationships, should implement effective controls for the completeness and accuracy of the information captured in the Enterprise Portfolio System, such as the identification of a project tier based on planned project cost and Project Complexity and Risk Assessment level.

Annex A: Audit Criteria

The following audit criteria were used in the conduct of this audit:

1. A department-wide governance and oversight mechanism was established at SSC and was in compliance with the TB Policy on the Management of Projects.
2. SSC’s PMF, including project oversight and approval mechanisms, was implemented and functioning as intended.
3. SSC’s OPMCA was accurate and SSC’s PMF was consistent with the assessment class and the TB Standard for Project Complexity and Risk.*
4. SSC maintained accurate and complete project information for monitoring and reporting purposes.

* Audit work conducted for this criterion was overtaken by events as SSC’s OPMCA submission was tabled prior to completion of the reporting phase and therefore is not included in this report.

Annex B: Project Governance Framework

Notes:

• There is a control point (gate) when moving from one stage to the next, except for Tiers 1 and 2, between Stages 2 and 3 and between Stages 4 and 5, where there is no gate.
• All projects are to provide status updates to the delegating authority. If a project changes status, the delegating authority must be notified as soon as possible.
• At the discretion of the approving authority, a project may be escalated to a higher tier regardless of its Project Complexity & Risk Assessment (PCRA) score.
• Projects require strong horizontal integration throughout the organization. Plan, Build and Operate functions must be involved to varying degrees throughout the full project life cycle.

Annex C: Acronyms

Endnotes

Free PDF download

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet:

• Adobe Reader
• Foxit Reader
• Xpdf
• eXPert PDFReader