Public Services and Procurement Canada
Audit of Land, Aerospace and Marine Equipment Procurement

Office of the Chief Audit Executive

Introduction

This engagement was included in the Public Services and Procurement Canada (PSPC) 2018 to 2023 Risk-Based Audit Plan.

PSPC plays a central role in the acquisition of goods and services for the Government of Canada. Among other things, PSPC has the exclusive authority (with limited exceptions) under section 10(2) of the Defence Production Act to acquire defence goods and services of all kinds on behalf of the Department of National Defence and Canadian Armed Forces, the Canadian Coast Guard and other government departments. These procurement activities consist of establishing and managing contracts to acquire a wide range of technically complex systems for the land, naval and aerospace environments.

In an effort to improve defence procurement practices, the Government of Canada established the Defence Procurement Strategy in 2014. The key objectives of the strategy are delivering the right equipment to the Canadian Armed Forces and the Canadian Coast Guard in a timely manner, leveraging purchases of defence equipment to create jobs and economic growth in Canada, and streamlining defence procurement processes.

This government-wide initiative involves four core federal departments: National Defence and the Canadian Armed Forces; Fisheries and Oceans; PSPC; and, Innovation, Science and Economic Development Canada. These departments work together to develop more efficient, timely and streamlined defence procurement processes and each department is responsible for a distinct aspect of the procurement process. In this context, PSPC is primarily responsible for leading the stakeholder/industry engagement, as well as the solicitation process, and for developing the procurement strategy.

Within PSPC, the responsibility for defence procurement falls under the Acquisitions Program, which is organized into 2 branches: the Defence and Marine Procurement Branch; and the Procurement Branch. Various sectors within the aforementioned branches, as well as the 5 regions, are responsible for conducting procurement activities for the acquisition of land, aerospace and marine equipment (and the associated in-service support), goods and services.

Definition of internal audit

An audit provides a reasonable level of assurance by designing procedures so that the risk of an inappropriate conclusion being drawn based on the audit procedures performed is reduced to a low level. This includes inspection, observation, inquiry, confirmation, recalculation, re-performance and analytical procedures.

Focus of the audit

This internal audit focused on PSPC procurement related roles and responsibilities structure, policies and procedures (referred to as the procurement environment), business processes and monitoring and reporting activities of low to medium complexity land, aerospace and marine procurements, for the sectors within the audit scope. The objective was to provide assurance that these were planned and implemented in a manner that supports the delivery of equipment, goods and services in a timely, efficient and effective manner.

Scope inclusions

The audit focused on select activities under PSPC's responsibility per the Defence Procurement Strategy, including receiving procurement requests, developing the procurement strategy, leading the solicitation process and monitoring the procurement information and the client feedback throughout the procurement process beginning with the pre-contractual planning phase to the post-contract close-out phase. Specifically, the audit focused on PSPC's responsibility as the contracting authority for land, aerospace and marine procurement activities, and assessed a sample of low and medium complexity defence contracts (for example, procurement complexity level 1 and level 2, see Annex A for more details) awarded by the Acquisitions Program from April 2017 to July 2019.

Selected sectors within the Defence and Marine Procurement Branch, and the Procurement Branch were included within the audit scope. For more information, please refer to About the audit.

Scope exclusions

Complex procurement projects (for example, complexity level 3 to level 5, see Annex A for more details) were excluded as they are designed to acquire commodities that are not readily available in the marketplace, and often use unique procurement strategies.

The sectors which were excluded from our audit scope, due to the fact these sectors focus on complex procurement projects are detailed in the About the audit section of the report.

As of July 2019, the regions make up 6% of the total dollar value of contracts within the audit's scope. Therefore, the regions have been excluded as the population data used as the basis of the audit's file review identified regional contracts being of comparatively low-dollar value compared to the dollar value of the entire population within the audit's purview.

For more information, please refer to About the audit.

Statement of conformance

The audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

High level overview of findings and conclusion

In the course of the audit, the audit team identified a number of key findings based on the assessment of the procurement environment, business processes and monitoring and reporting activities of low to medium complexity land, aerospace and marine procurements, for the sectors within the audit scope, as follows:

Procurement environment and business processes

We found that, based on our sample of contracts, the established policies, procedures, and business processes which have been designed and implemented to support PSPC in acquiring land, aerospace and marine goods, services, and equipment for its clients in a timely, efficient and effective manner were not operating as intended as there were a higher than expected number of instances where the process was not followed. This included not consistently using the Central Allocations Unit as a central point of access, as well as key contract documents were not available for verification. That high rate of non-compliance could potentially have negatively impacted the timeliness of the procurement process, as well may have had an effect on client service satisfaction, and performance measurement for decision making and continuous improvement

Information management

We found that there are monitoring and reporting activities, however there are weaknesses related to information management practices, in particular with regards to accessibility and availability of procurement files and related information. We also found that information retained in the procurement information systems may not be reliable, accurate or complete, which negatively impacts the quality and timeliness of reporting used for decision-making.

Quality control activities, client feedback and lessons learned

We found that procurement related quality control activities and client feedback mechanisms have been designed and implemented to support the Department in acquiring land, aerospace and marine procurement in a timely, efficient and effective manner. However, an opportunity for improvement exists with respect to formalizing and sharing informal best practices and lessons learned for use across the Acquisitions Program.

Observations

Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the findings and conclusions in this report and to provide an audit level of assurance. The findings and conclusions are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The findings and conclusion are only applicable to the entities examined and for the scope and time period covered by the audit.

Procurement environment and business processes

Expectations

The procurement environment within the selected sectors allows PSPC to acquire land, aerospace and marine equipment, and goods and services for its clients in a timely, efficient and effective manner. This requires the implementation and operationalization of a clear roles and responsibilities structure, and ensuring that related business processes are designed, implemented and operating as intended.

Conclusion

We found that, based on our sample of contracts, some of the established policies, procedures, and business processes which have been designed and implemented to support PSPC in acquiring land, aerospace and marine goods, services, and equipment for its clients in a timely, efficient and effective manner were not operating as intended as there were a higher than expected number of instances where the process was not followed. This included not consistently using the Central Allocations Unit as a central point of access, as well as key contract documents were not available for verification. That high rate of non-compliance could potentially have negatively impacted the timeliness of the procurement process, as well may have had an effect on client service satisfaction, and performance measurement for decision making and continuous improvement

The procurement environment is comprised of the roles and responsibilities structure, as well as related policies, procedures and business processes. The establishment of a clear roles and responsibilities structure supports PSPC as the contracting authority within the federal government. Clear policies, procedures and business processes, as well as accountabilities for responsible stakeholders help to ensure that PSPC meets its objectives to deliver timely, effective and efficient procurement services.

Roles and responsibilities

We found that a roles and responsibilities structure, which supports effective procurement decision-making at the operational level, has been established as evidenced through the development and implementation of the contracting policy suite, related guidance materials, and governance structure. However, awareness of and adherence to the roles and responsibilities structure was inconsistent, specifically in regards to the role of the Central Allocations Unit.

The Central Allocations Unit, as per Section 2.30 of the Supply Manual^{Footnote 1}, provides a central point of access for all new requisition requests for goods or services (Requisition Form (9200)), and is responsible for tracking new requisitions, conducting preliminary assessments of mandatory requirements on the requisitions, and then delegating the requisitions to the appropriate sector within PSPC for action. For consistency purposes, the function acts as a formal point of contact for client departments at the beginning of the procurement file is life-cycle, regardless of which sector or contracting professional is assigned to their procurement file(s). Per the Supply Manual, both client departments and PSPC sectors must use the Central Allocations Unit when initiating procurements.

Upon a detailed review of a sample of 66 procurement files, we found that adherence to this requirement was inconsistent. More specifically, we noted that 18 files (27%) did not include evidence that the Central Allocations Unit was used as the point of contact, that is, these procurement files were tracked and managed directly between the procurement officers and the clients. Interviews suggested this was not due to a lack of awareness on the part of the procurement community/interviewees, but rather a perceived lack of value or potential inefficiencies in using the unit as an intermediary.

The files in our sample that were processed by the Central Allocations Unit were all processed within a maximum of three business days before being assigned to division managers, thus within the service standards. We were informed that there may be exceptions, specifically in cases where the Central Allocation Unit did not allocate the file to the appropriate division, resulting in delays due to subsequent re-allocation of the requisitions.

From our file review, we noted instances where delays were generated when the requisitions were sent directly to the procurement officers. These were either because the procurement officers contacted were not responsible for the requisition, or because the officers to which the initial requisitions were sent, were either unavailable or no longer with the department, leaving the requisitions unassigned. During the course of the audit, interviews also noted that the procurement community is facing high rates of turnover and retention issues, underlining the need to have a consistent and central point of contact for client departments.

Furthermore, as part of measuring timeliness for the purposes of measuring and improving the performance of procurement processes, the Acquisitions Program uses the date that the requisition was received within the Central Allocations Unit as a milestone. If this step is ignored, the dates used for tracking will be inconsistent across the sectors, and affects the reporting and monitoring of performance and adherence to service standards.

Notwithstanding the lack of compliance with the requirement to use the Central Allocations Unit as point of contact as stipulated in the Supply Manual, the results of our file review and interviews demonstrated that failing to consistently use a central point of contact negatively impacted the timeliness of the procurement process. This in turn may have an effect on client service satisfaction, and performance measurement for decision making and continuous improvement.

Business processes for developing and selecting the procurement strategy and assessing contract risks

One of PSPC's responsibilities per the Defence Procurement Strategy is to develop the procurement strategy, which entails selecting the type of procurement method which will be used to fulfill the requirement: either sole-source or competitive. The development and selection of the procurement strategy is enabled by completing the mandatory requirements with the aid of the guidance outlined in the Supply Manual, including the: Procurement Risk Assessment process^{Footnote 2}; development and delivery of the Procurement Plan^{Footnote 3} or the Contract Planning and Advance Approval^{Footnote 4} process; and, the assessment of the Complexity Level^{Footnote 5}.

The completion of these requirements determines the appropriate level of approval that contracting officers need to seek, including which oversight bodies will be involved in decision-making related to the file, and whether or not a Treasury Board submission is required. It also allows the Department to ensure integrity and due diligence are applied before entering into legally binding agreements.

Upon file review, we found that, for those files sampled, 15% of the files did not include a Procurement Risk Assessment on file, and 18% were missing either a Procurement Plan or Contract Planning and Advance Approval on file. Furthermore, interviews revealed that within some of the selected sectors, the approval of a Procurement Plan/Contract Planning and Advance Approval and/or risk assessment were not seen as mandatory.

Similarly, our file review revealed that 13 of 66 files (20%) did not include a complexity assessment on file. The Complexity Level of a procurement ranging from a level 1 to 5 is determined following the conduct of an assessment, and is used to direct the requirement into the appropriate process stream, and to allocate it to a contracting officer with the necessary skill set to handle the procurement.

During the course of the audit, the selected sectors were given the opportunity to provide additional documentation not initially seen on the files; with sampling results updated accordingly to reflect the latest information provided. Some of the root causes for not having the key documents were that they were never prepared, as not seen as mandatory, as well as not available as they were not properly filed.

Based on our file review and interviews, we concluded that there is a lack of compliance to some established policies, procedures and business processes. Consistent adherence to, and proper documentation of, the above mentioned processes assists in the identification of an appropriate procurement strategy and level(s) of approval, and also supports an informed understanding of potential contract-related risks and mitigation strategies.

Recommendation 1

The Assistant Deputy Minister, Procurement Branch, and the Assistant Deputy Minister, Defence and Marine Procurement Branch, should ensure that established policies, procedures and business processes are adhered to. This would include using the Central Allocations Unit as a central point of contact, and the completion and retention of key contract documentation.

Management Action Plan 1.1

A communication will be sent out to the procurement community regarding documentation procedures. This reminder will include adherence to formal business processes and highlight the importance of using the Central Allocations Unit.

Management Action Plan 1.2

Sectors that were audited, and advised as necessary by the audit team, will:

1. have team leaders/managers review documentation in sample files to determine the nature of information management issues and identify the root cause for each file applicable to their respective division
2. a report to the assistant deputy ministers ADM's from pertinent sectors will be produced that will roll-up explanations and identify the root causes along with recommendations

Information management

Expectations

Procurement activities within the selected sectors are adequately monitored and reported on to facilitate quality and timely decision-making and continuous improvement. This includes the expectation that information management practices within the Acquisitions Program support the timely, efficient and effective delivery of procurement services for its clients.

Conclusion

We found that there are monitoring and reporting activities, however there were weaknesses related to information management practices, in particular with regards to accessibility and availability of procurement files and related information. We also found that information retained in the procurement information systems may not be reliable, accurate or complete, which negatively impacts the quality and timeliness of reporting used for decision-making.

As per the Government of Canada Policy on Information Management, information is vital for departments to function and manage effectively. The availability of high-quality, authoritative information supports the delivery of efficient and effective services and results in a responsive and accountable government. As such, sound management of procurement information is critical to enable the Acquisitions Program to achieve and sustain its goals of providing high-quality services and programs while meeting operational and fiscal responsibilities, legal obligations and accountabilities. Maintaining accessible and complete files directly supports the Department in demonstrating transparency, due diligence and sound decision-making in cases of legal risk and liability.

Accessibility and availability of procurement files and related information

Through requests for information for the purposes of testing the procurement process, we noted several weaknesses with respect to the management of procurement information, more specifically with regards to the accessibility and availability of procurement files and related documentation. At the onset of our audit examination phase, we requested access to all information available for our sample of 66 procurement files. Our initial file review indicated that 57 (or 87%) of the files had at least one key piece of documentation missing. After follow-up by the audit team, additional documentation was provided, and at the end of our audit, 35 files (or 53%) had at least one key piece of documentation missing from the file, and for which the responsible sector could not account for.

We were informed that information for each procurement file is maintained in a variety of formats depending on the source of the requisition and how each procurement officer tracks and stores it. There is no central location for all procurement files and related documentation, with each procurement officer being responsible for storing and maintaining the documentation related to the procurement files they manage.

In the absence of an identified central point of contact, the accessibility and availability of procurement files and related information creates a serious challenge for the Acquisitions Program, as there is a requirement to provide both internal and external stakeholders, such as client departments and senior management, timely and complete information for the purposes for decision making. As per the Contract Management Guide, Procurement Managers need to be aware of the "importance of properly documenting their procurement files before putting away files, and should understand that file documentation represents Crown records, and not information that should remain in the sole custody of PSPC employees".

This issue was experienced by the audit team during the course of the audit, as we were unable to obtain some files and related documentation for the purpose of our file review. Similarly, some of our interviewees during the audit indicated that when a contracting professional was not available (for example, on training, out of the office, etc.), obtaining the files under their responsibility was not always possible. The Treasury Board Submissions Unit within the Business Integration and Liaison Directorate of the Procurement Branch identified this challenge as well, given they require access to procurement file-related information in order to prepare necessary submissions in a timely manner to meet deadlines set by the Treasury Board Secretariat ^{Footnote 6}.

Finally, the latest client feedback results, which included input from 389 individuals across various client departments underlined communication and accessibility of information as reoccurring issues client departments face when working with the Acquisitions Program, with an overall negative impact on client satisfaction. Monitoring and reporting activities are discussed in detail in the Quality Control Activities, Client Feedback and Lessons Learned section.

Procurement information within the procurement information systems

At the time of the audit, the main information systems used by the Acquisitions Program to record and track procurement related information were the Automated Buyer Environment (ABE), and the reporting arm of the system, which is the Automated Information System (AIS). ABE is used to capture various procurement-file related data, including procurement complexity levels, contract award dates and amounts, etc. We were informed that ABE represents the main source of information used by the Procurement Statistics Division^{Footnote 7} to generate various reports used by stakeholders, including senior management for decision-making.

Through our file review, we noted a number of discrepancies between the information captured in ABE and the information in the actual files. More specifically, for 12 of 66 files (18%), the information captured in ABE contained inaccurate data related to contract value, number of amendments and contract award date. Also, the most recent internal reports on compliance to services standards prepared by the Acquisitions Program noted that for the selected sectors within the audit scope, for fiscal year 2018 to 2019, the number of files which did not have a complexity level entered into the system ranged between 18% and 90%, depending on the sector. Further, as mentioned previously, our file review revealed 13 of 66 files (20%) did not have a complexity assessment on file. The complexity level information was also missing in ABE. The above noted discrepancies may negatively affect data accuracy when generating reports for management, including those used to support decision-making.

The gaps in the reliability, accuracy, and completeness of procurement information retained in the above mentioned systems directly impacts the reliability of information used for performance measurement, reporting, and timely decision-making. Additionally, the above mentioned internal reports on compliance to services standards highlighted the importance that the use of complexity coding continues to be enhanced across the Program in order to "ensure high quality data upon which performance expectations are being set and tracked".

A good practice noted was the use of standardized checklists by some groups to ensure all key documents were on file. Those files under review that included a standardized checklist, were generally well organized and were not missing key documents. While the majority of questionnaire respondents stated that their groups have standardized checklists or guidance material to assist them, our file review revealed that the majority of sampled files did not include a standardized checklist. Given the above-noted observations related to the Procurement Environment and Business Processes, where the audit found inconsistent application and documentation of key business processes within the selected sectors, a more formal and consistent use of the checklists across the Acquisitions Program would likely improve accuracy and completeness of files.

Going forward: It is worth noting that while procurement modernization initiatives within the Acquisitions Program include the design and establishment of a new procurement system, (for example, E-Procurement) we were advised that the Automated Buyers Environment and Automated Information System will continue to run in parallel with the new system and will also be used for historical reporting; as such, good information practices on the documentation of procurement data in the systems remain relevant, and critical to improve the accuracy and completeness of data for reporting purposes.

Recommendation 2

The Assistant Deputy Minister, Procurement Branch, and the Assistant Deputy Minister, Defence and Marine Procurement Branch, should:

• strengthen information management practices within their responsible sectors in order to ensure that procurement files and related documentation are readily available and accessible, including the establishment of standardized checklists to ensure key documents are included in the procurement files
• ensure that information that should be entered into the Automated Buyer Environment, according to the Supply Manual and other relevant policies and procedures, is consistently entered to support completeness, accuracy, and reliability of information used for reporting and decision-making

Management Action Plan 2.1

In terms of immediate actions to be taken to address the audit findings, w communication will be sent out to the procurement community regarding documentation procedures. This reminder will include adherence to formal business processes. This communication will also emphasize that although the Automated Buyer Environment (system) is being phased out, while it is still being used, formal procedures should still be followed to ensure information integrity and adherence to policy.

Management Action Plan 2.2

It is clear that information management is a key issue to be addressed. Currently, the Electronic Procurement Solution (EPS) is being implemented over the near term by the Acquisitions Program to, among other things, improve information management related to procurement activity. For example, as part of ongoing action items to address the audit findings, EPS will:

• enable automation of the approval process
• support the creation of standardized templates for Sourcing and Contracting
• allow for increased contract data and compliance
• enable traceability and availability of historical data
• guide and automate due diligence processes
• decrease paper-based engagements

Ongoing progress on the EPS is being communicated regularly to the Department Audit Committee.

Quality control activities, client feedback and lessons learned

Expectations

Quality control activities, client feedback mechanisms and lessons learned are designed, implemented and incorporated into procurement practices to support PSPC in acquiring the right land, aerospace and marine equipment, goods and services in a timely, efficient and effective manner for its clients.

Conclusion

We found that procurement related quality control activities and client feedback mechanisms have been designed and implemented to support the Department in acquiring land, aerospace and marine procurement in a timely, efficient and effective manner.

An opportunity for improvement exists with respect to formalizing and sharing informal best practices and lessons learned for use across the Program as a whole.

Incorporating quality control activities, client feedback mechanisms and the capturing of lessons learned into the procurement lifecycle is critical in identifying opportunities for improvement and thus contributing to PSPC's mandate to render procurement practices simpler, less administratively burdensome and more user friendly among others.

We found that a number of formal and informal activities were undertaken across the Acquisitions Program to identify opportunities to improve the procurement practices, including client feedback, quality control activities and sharing lessons learned, however, an opportunity for improvement exists with respect to formalizing and sharing informal best practices and lessons learned.

Client feedback

We found client feedback regarding contracting activities within the Acquisitions Program was formally sought and captured within the Acquisitions Program via the Post Contract Assessment Survey and the Client Satisfaction Survey. Both surveys are run and managed by the Office of Small and Medium Enterprises and Strategic Engagement.

The Post Contract Assessment is an exercise conducted on a monthly basis to seek client feedback by sending online surveys to a stratified, random selection of 200 recently issued contracts. Areas of feedback in the questionnaire relate to client satisfaction of services provided by the Acquisitions Program, including the overall satisfaction with services; quality of the services provided; speed and convenience of services; and, effectiveness of communication by the Acquisitions Program, etc. The questions within the survey are related to the service standards that the Program has established for procurement-related activities. The results of the Post-Contract Assessment are communicated to procurement groups in the National Capital Area, the Regions, PSPC senior management, and client departments.

Every three years, a Client Satisfaction Survey is also sent to client departments to seek feedback on several main areas; which included service delivery; timeliness; communications; electronic services; outreach; and, overall satisfaction with services provided by the Acquisitions Program. The questions within the survey are also tied to the service standards for the Program In 2019, the survey was sent to 1,389 individuals from 55 departments, with a 28% response rate (389 respondents). The results of the survey were benchmarked against the previous survey, conducted in 2015, and trends incorporated into a report for the purposes of sharing with Acquisitions Program staff members in order to help constructively improve the Program's performance.

Lessons learned

The Contract Management Guide highlights the importance of identifying lessons learned for future procurements, through the inclusion of this step as part of the recommended activities for the contract close-out phase. According to our questionnaire results, over 96% respondents stated that their sector informally shares best practices and lessons learned within their group, and in some instances, with other sectors. However, we were unable to confirm via file review that lessons learned were documented and shared in a formal manner.

Quality control activities

The results of formal quality control activities within the Acquisitions Program are reported in a quarterly Service Standard Report, which details the results of service standards criteria established against established targets and timelines. The service standards measured include timeliness of acknowledgement of requisitions received, identification of a point of contact within Acquisitions for the client and timeliness of contract award. The data used to populate the report is derived from Acquisition Information Service (AIS) System. The Acquisitions Program advised that the report is shared with PSPC senior management for the purposes of decision making.

Additionally, as of 2014, the Procurement Support Services Sector within the Procurement Branch has implemented a Post Procurement Review Process in order to monitor procurement file compliance with applicable procurement policies. At the conclusion of each fiscal quarter, a list is generated from the procurement information system of all recent procurements in that quarter. From this list, a random sample of approximately 160 files is selected across the Program. The procurement file is then reviewed against a set of compliance statements, with results provided to the contracting officer responsible for the file review. Contracting officers are then responsible for reviewing the report with their Team Leader or Manager.

We were advised that results of these reviews were previously rolled up into an aggregate report, which is shared with PSPC senior management. We were provided with the latest available report, dated November 2017 and were informed that consultations with senior management from each sector and region within the Acquisitions Program were held to discuss the findings of the report, share sector specific data, and obtain feedback and suggestions for improving the existing quality control processes. The report identified concerns regarding the completion of required approval documents and proper documentation of procurement files. The audit team was advised that there have not been subsequent aggregate reports produced, but rather, the detailed results of quarterly reviews are provided to the appropriate senior management representatives for information and discussion.

Interviews also noted that some individual procurement teams, or groups, have adopted informal quality assurance activities, such as random reviews of procurement files by the responsible procurement Manager, and the development of standardized checklists, as mentioned above.

Recommendation 3

The Assistant Deputy Minister, Procurement Branch, and the Assistant Deputy Minister, Defence and Marine Procurement Branch, should formalize and share existing informal best practices and lessons learned for use across the Program as a whole for the purposes of continuous improvement.

Management Action Plan 3.1

Sharing best practices and lessons learned are embraced by the Acquisitions Program. In addition to formal targeted training, currently, various modes for sharing information exist, these include:

• the internal publication of learning resources (GCpedia or other)
• Acquisitions Newsfeed (via email)
• Acquisitions Program (AP) Days events

Acquisitions Program will facilitate the identification and sharing of informal best practices through various learning activities and communication methods, on both an immediate and ongoing basis.

About the audit

Authority

This engagement was included in the Public Services and Procurement Canada (PSPC) 2018 to 2023 Risk-Based Audit Plan.

Objective

This internal audit focused on the procurement environment, business process and monitoring and reporting activities of low to medium complexity land, aerospace and marine procurements, within the defence portfolio. The objective was to provide assurance that they are planned and implemented in a manner that supports the delivery of equipment, goods and services in a timely, efficient and effective manner.

Scope and approach

The audit focused on PSPC roles and responsibilities for developing the procurement strategy, leading the solicitation process and monitoring the procurement information and the client feedback throughout the procurement process; beginning with the pre-contractual planning phase to the post-contract close-out phase. Specifically, the audit focused on PSPC's responsibility as the contracting authority for land, aerospace and marine procurement activities, and assessed select low and medium complexity defence contracts (for example, procurement complexity level 1 and level 2, see Annex A for more details) awarded by the Acquisitions Program from April 2017 to July 2019.

The following selected sectors were included:

Defence and Marine Procurement Branch:

1. the Land and Aerospace Equipment Procurement Support Sector acquires a wide range of equipment, systems and life-cycle support services related to national security and defence. These commodities may include weapons systems and ammunition, defence electronics, aerospace equipment, armoured vehicles and the associated in-service support
2. the Marine Services & Small Vessels Sector handles the small vessel component of the National Shipbuilding Strategy. The sector's procurement services involve the purchasing of oil, life extension, specific machinery, refits, and other large procurements from a supply perspective for the ships

Procurement Branch:

1. the Commercial and Alternative Acquisitions Management Sector manages defence procurements for a diverse set of commodities such as vehicles, fuels, oils and lubricants
2. the Services and Technology Acquisition Management Sector acquires commodities that facilitate space communications and space surveillance for defence purposes
3. the Office of Small and Medium Enterprises and Strategic Engagement, in its role of collecting and analyzing client satisfaction survey results, provides evidence on client feedback monitoring
4. the Procurement Support Services Sector provides information on how quality assurance activities related to contracting operations are conducted
5. the Procurement Business Management Sector provides information on how the Treasury Board submission process is managed and how the complexity risk assessment is performed

Complex procurement projects (for example, complexity level 3 to level 5, see Annex A for more details) were excluded as they are designed to acquire commodities that are not readily available in the marketplace, and often use unique procurement strategies.

The following sectors were excluded from our audit scope:

Defence and Marine Procurement Branch: Excluded due to the fact these sectors focus on level 3 to level 5 procurement projects.

1. the Defence Major Project Sector establishes and manages contracts to acquire large and technically complex land and aerospace procurement projects as well as the associated in-service support on behalf of the Department of National Defence and Canadian Armed Forces
2. the Large Combat Ships Construction Sector and Large Ship Construction Sector. These sectors handle the large vessel component of National Shipbuilding Strategy. Projects are complex, and the strategy is currently being audited by the Office of the Auditor General
3. the Washington Sector as it only handles defence procurement sole sourced to the United States government through the Foreign Military Sales program which was audited by the Office of the Chief Audit Executive in 2014, and no significant issues were found

Regions: As of July 2019, the regions make up 6% of the total dollar value of contracts within the audit's scope. Therefore, the regions have been excluded as the population data used as the basis of the audit's file review identified regional contracts being of comparatively low-dollar value compared to the dollar value of the entire population within the audit's purview.

Text version

This graphic consists of a list of the 20 sectors and regions of PSPC participating in the procurement of Land, Aerospace and Marine Equipment. The graphic highlights 7 sectors that have been included in the scope of the audit.

Under the Defence and Marine Procurement Branch, both the Land and Aerospace Equipment Procurement Support sector, as well as the Marine and Small Vessels Sector were included in scope. However, the Defence Major Projects sector, the Defence Procurement Strategy Secretariat sector, the Large Combat Ship Construction Sector, the Large Ship Construction Sector, the Strategic Policy Sector and the Washington sector were excluded from scope.

Under the Procurement Branch, the Commercial and Alternative Acquisitions Management sector, the Services and Technology Acquisition Management sector, the Office of Small and Medium Enterprises and Stakeholder Engagement sector, the Procurement Support Services sector, and the Procurement Business Management sector were included in scope.  However, both the Acquisitions Program Transformation Sector and the Acquisitions Strategy and Relations sector were excluded from scope.

All Regions were excluded from scope – including the Atlantic region, the Quebec region, the Ontario region, the Western region, and the Pacific region.

Approach

A file review was performed to assess the effectiveness of key controls in the procurement process of low and medium complexity land, aerospace and marine goods and services. This included verifying consistency in the application of procedures, and identifying (where possible) bottlenecks, redundancies and weaknesses. A statistical sample of 66 files was established using procurement data provided by the Acquisitions Program.

This audit was conducted in accordance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

The survey and examination phases included over 20 interviews with key departmental personnel involved in the acquisition of land, aerospace and marine equipment, as well as associated services and materiel required to support this equipment. During the planning phase, relevant processes, policies, and documentation were reviewed. A risk assessment was conducted to assist the auditors in determining the audit objective, scope and criteria.

During the examination phase, in addition to the above mentioned interviews, documentation review and file testing were conducted. We also reviewed the results of client department feedback provided to the Acquisitions Program during the scope of the audit, in order to understand their views on the effectiveness of land, aerospace and marine procurement activities. For example, for fiscal year 2019, this included feedback received from three large departments, with a total of 389 respondents. At the end of the examination phase, the audited organization provided validation of the findings.

During the reporting phase, the audit team documented the audit findings, conclusions and recommendations in this Draft Report. This report will be internally cleared through the Office of the Chief Audit Executive's quality assessment function. The audited organization will be provided with the Draft Report and will be requested to review and comment on this report. Comments will be assessed and incorporated in the Chief Audit Executive's Draft Report. This report will be sent to the audited organization for final acceptance. A management response to the report and a Management Action Plan in response to the audit recommendations will be requested. The Draft Final Report, management response and Management Action Plan will be tabled at the Departmental Audit Committee meeting for final approval.

Criteria

The criteria for this audit were chosen based on Audit Criteria related to the Management Accountability Framework: A Tool for Auditors (Published by Office of the Comptroller General [OCG], March 2011) and the PSPC Supply Manual.

The criteria were as follows:

• 1. the procurement environment within the selected sectors allows PSPC to acquire the right land, aerospace and marine equipment, goods and services for its clients in an efficient and timely manner
  • 1.1. a clear roles and responsibilities structure is established and facilitates effective procurement decision-making
  • 1.2. PSPC's engagement with the Treasury Board Secretariat supports the timely procurement of land, aerospace and marine equipment, goods and services
  • 1.3. a process for the selection of the appropriate procurement strategy is in place
• 2. business processes are designated and implemented within the selected sectors to allow PSPC to acquire the right land, aerospace and marine equipment, goods and services for its clients in a timely, efficient and effective manner
  • 2.1. PSPC procurement process of acquiring land, aerospace and marine goods and services is consistently applied
  • 2.2. key controls are designed and implemented in the procurement process to ensure contract objectives are achieved
  • 2.3. PSPC effectively fulfills its requisition review function in relation to client requirements
  • 2.4. mechanisms are in place to mitigate delays and other impediments to the efficient and timely acquisition of land, aerospace and marine equipment, goods and services
• 3. PSPC's activities related to the procurement of land, aerospace and marine equipment, goods and services, within the selected sectors, are adequately monitored and reported to facilitate quality/timely decision-making and continuous improvement
  • 3.1. client feedback is being sought and actioned
  • 3.2. procurement information retained in the information systems is sufficient for performance reporting and timely decision-making
  • 3.3. best practices and lessons learned are captured and used for continuous improvement
  • 3.4. risk-based quality control reviews are conducted, with results being used to facilitate continuous improvement

Audit work completed

Review fieldwork for this audit was substantially completed in October 2019.

Audit team

The audit was conducted by members of the Office of the Chief Audit Executive, overseen by the Director of the Procurement Audit Directorate and under the overall direction of the Chief Audit Executive.

The engagement was reviewed by the quality assurance function of the Office of the Chief Audit Executive.

Annex A: Complexity assessment tool

Complexity Level 1 procurements include or involve one or more of the following:

• a Call-up against a Standing Offer
• a commercially available good that is readily available in the marketplace, which may include manufacturer-supplied customizable options. Includes Commercial Off The Shelf (COTS) or Military Off The Shelf (MOTS)
• a common service that is readily available in the marketplace, which may include supplier customized options
• procurement methodologies that are tried and proven using existing conventions
• Departmental Standard Procurement Templates or legally approved templates
• contractor selection methodology that is based on price and delivery alone, or where all criteria determining the successful bidder are objective and do not require subjective judgement (for example, grid)

And do not include/involve any of the criteria of Levels 2, 3, 4 and 5.

Complexity Level 2 procurements include or involve one or more of the following:

• government-specified performance requirements to commercially available goods
• a requirement which uses existing technology/processes to achieve an innovative solution
• services with subjective results (for example, consultations, recommendations)
• Departmental Standard or Sector procurement templates which have been modified; that introduce new custom clause(s), or that use terms and conditions other than those in the SACC Manual
• 1 or more evaluation criteria to determine the successful bidder that require subjective judgment
• Industrial and Technological Benefits (ITBs) and/or Value Propositions
• vendor pre-qualification (includes establishment of Supply Arrangements)
• advance payments

And do not include/involve any of the criteria of Levels 3, 4 and 5.

Complexity Level 3 procurements include or involve one or more of the following:

• a requirement which uses new, or a combination of new and existing technology/processes to achieve an innovative solution and may be developmental in nature
• a requirement and/or procurement that is highly unpredictable due to a high level of uncertainty
• untested procurement strategy

And do not include/involve any of the criteria of Levels 4 and 5.

Complexity Level 4 procurements include or involve the following:

• transformational requirements that are enterprise-wide (across Government)

and one or more of the following elements:

• high level of uncertainty:
• may involve partnerships with clients and multiple stakeholders within various federal government departments, where decision-making is shared
• contract administration is detailed and extensive and its success is unpredictable
• more than one tier of government (for example, federal, provincial, municipal) or multiple clients/stakeholders with decision-making authority

And do not include/involve any of the criteria of Level 5.

Complexity Level 5 procurements include or involve the following:

• transformational requirements that affect public policy or culture (external to Government)

and one or more of the following:

• high level of uncertainty
• involves partnerships with clients, multiple stakeholders (internal and external to the federal government) and elected officials
• reshapes public policy or introduces new public policies
• decision making is shared
• contract administration is detailed, extensive and highly unpredictable
• ultimate success is determined by the public at large