Evaluation of the Naval Experience Program

Developmental Evaluation of the Naval Experience Program

This evaluation was conducted between March 2023 and May 2024 and provides a targeted assessment against key evaluation questions regarding early program effectiveness and efficiency. The focus was on the initial stages of the recruitment process: attraction and enrolment (Program 3.1 Recruitment). Evidence was gathered and analyzed using mixed methods. Advisory assistance was provided to the program during Year 1 of pilot implementation using a Developmental Evaluation (DE) approach, with emphasis on the development of key performance metrics. DE products were developed in collaboration with the program, including:

• Performance Measurement Framework: collaborative development of immediate, intermediate and ultimate performance indicators, in alignment with the program logic model, to guide the program in the collection of data. This enabled the program to develop benchmarks for Year 1 of the pilot, track progress and report key metrics to senior management.
• Theory of change: development of “program logic and processes,” with consideration of program assumptions validated through interviews, as well as internal and external influencing factors. This enabled the program to visualize the many factors impacting the pilot’s progress and begin to address them through concrete program adjustments.

About the Naval Experience Program

The Naval Experience Program (NEP) is a five-year pilot program that falls under the Royal Canadian Navy (RCN) recruitment program. NEP addresses low recruitment levels and responds to the Arbour Report recommendation to “restructure and simplify [the Canadian Armed Forces] recruitment, enrolment and basic training processes.” In alignment with the successful Australian Defence Force Gap Year program, NEP attracts participants through marketing campaigns, enrols them using a streamlined entry process, then trains them before providing a commitment-free experience that exposes them to various RCN trades. Following NEP, participants can then apply to a specific trade of interest within the RCN or other environments, or they can off-ramp from the Canadian Armed Forces with “no strings attached.”

Marketing Campaign Trends and Results

• Overall, the NEP campaign produced multiple advertising campaigns and materials in both traditional and online marketing platforms, reaching 180 thousand accounts across five social media platforms. Engagement on Facebook remains the highest compared to other platforms (as indicated in the following graph); however, the program’s 18–24 target demographic is the lowest reached on Facebook.
• The majority of interviewees noted that the awareness campaign (in social and traditional media) went well and was effective.
• Following the anticipated decline of interest since March 2023, NEP content is levelling off to the average performance of other Navy content.

Challenges and Metric Caveats

• Navy Public Affairs and NEP collaborate in marketing efforts, while Navy Public Affairs tracks NEP marketing data. The Navy Public Affairs Social and Traditional Media Reports recommended incorporating more tangible results/program success in the digital NEP content, as well as tailoring future NEP content to the target demographic group identified as 18–24.
• Interviewees cited limited resourcing as impacting NEP’s ability to appropriately target and track success for its ideal recruits from the start of the campaign.
• As well, NEP has yet to determine targets for its attraction/marketing indicators.
• Note that the 101-day timeline to for file enrolment (Key Takeaway 4) is not currently an indicator of NEP or recruitment intake capacity and, therefore, should not influence the marketing campaign.

Engagement in social media marketing has declined since pilot inception.

“Family/public exposure would indicate that [NEP] is better than many [RCN] initiatives in terms of public understanding.”

Interviewee

Ongoing Work

• The Director of Navy Personnel (Personnel, Psychology and Attraction) team is developing a more mature approach to recruitment process guidance, as well as more effective attraction methods and performance metrics to better and discreetly target Canadian populations. These include the following activities and focus areas:
  • More tangible results/program success are being incorporated in NEP’s digital content.
  • The Director of Navy Personnel Attractions ran working groups in December 2023 to plan events promoting NEP and has already begun planning the 2024 Great Canadian Marine Careers Expo. They also continue to seek new partnerships with educational institutions to inform potential high school, college and university leads about the program and have begun offering a “concierge” service, whereby members of the team reach out directly to leads generated at events to provide one-on-one recruitment guidance in an effort to generate new recruiting prospects.
• Surveys are being designed by NEP to inform attraction plans and program performance and will be reported as trends surface.

Initial Target Met and the Strategic Intake Plan

• Based on March 1 NEP candidate statistics, with 160 candidates currently enrolled (including 7 applicants transferred from the Raven program), NEP surpassed its initial target of 144 recruits in December 2023 (as indicated in the following graph). Thereafter, the Strategic Intake Plan (SIP) cap was moved to 150 recruits and subsequently to 160 to accommodate the increasing number of recruits.

• NEP will run continual intake to the program, with variance based on the Navy’s capacity to absorb. With the RCN’s current recruitment needs, it is unlikely that its capacity to absorb will cap during the pilot’s duration.

• SIP for NEP will therefore fluctuate based on the natural flow of intake and enrolment to the program, not based on the target. Note that since the Navy is not meeting its SIPs in other occupations, there is currently flexibility for NEP in this area.

  SIP, issued by Military Personnel Command, outlines the recruitment targets by individual occupations within each CAF organization. SIP also identifies priority occupations and threshold occupations, which are differentiated by their percentages of vacant positions.

Pre-existing Files and Enrolment Ratio

• Despite successes in reaching the enrollee target for Year 1, nearly half of the NEP files were of individuals who had pre-existing files with CFRG and switched to NEP when it became available.

  Forty-six percent of NEP files were of individuals who had pre-existing files at CFRG.

• Historically, the Navy has had an applicant to enrollee (occupation) ratio of 8:1. Current NEP data indicates that the NEP applicant to enrollee ratio is 9:1; however, it is trending down, as indicated in the following graph.

  

  Figure 5 Summary

• These metrics suggest that a similar target may not be feasible in coming years. These challenges may be due to the absence of internal enabling factors such as a truncated program design phase.

During Enrolment

• As of March 1, 2024, 35 percent of NEP files are currently open with CFRG and are at different stages in the enrolment process.
• Fifty-four percent of files were closed. The majority of files (66 percent) were closed at the request of the applicant or due to inactivity (i.e., the recruiters were not able to re-establish contact with the applicant for 30 days or longer). This is depicted in the following graph.

At BMQ

• According to NEP, the drop-out rate for NEP voluntary release at Basic Military Qualifications (BMQ) is currently 8 percent. Further data collected over the pilot’s next year will help establish a baseline, with the goal of a decline in drop-out rates.

• Interviewees from CFRG and Naval Reserve recruitment centres and headquarters were asked about their experiences with file closures. All noted that the NEP file closure reasons and rates were aligned with general recruitment; however, there has been a slight increase in NEP file closures recently.
• External environmental factors that may influence this increase in dropout rates, according to Navy Public Affairs seasonal attraction/marketing data reports, include public perception of CAF culture change efforts related to sexual misconduct.
• Insights into file closures and validating related assumptions through the collection of key metrics during the enrolment process may validate program assumptions related to participants. As part of DE, continued collaboration between NEP and Director General Evaluation is planned in this area for Spring 2024.

Canadian Forces Aptitude Test

Although the Canadian Forces Aptitude Test (CFAT) is one of the most commonly cited reasons (from interviews) for participant drop out, according to recruitment data, it accounts for only 4 percent of the known reasons for file closures.

CFAT Ongoing Work:

• CMP is collaborating with the Digital Transformation Office and working on introducing an online, unproctored CFAT to reduce barriers for applicants and expedite suitability screening.
• As of December 2023, certain occupations now have CFAT delayed until BMQ, including NEP recruits.

• Despite the key program element of streamlining the enrolment process, all interviewees confirmed that there has been no change in the NEP enrolment process when compared to other recruitment programs, save for the prioritization of green files (files with minimal security implications). NEP is one of several priority files, which makes prioritization a challenge. The interviewees stated that this prioritization comes with trade-offs for other files.
• The average time to process a NEP file is 104 days for CFRG and 97 days for Naval Reserve Divisions (NRD). This is an improvement compared to the average 200 days quoted in the 2016 Auditor General Report; however, this improvement was due to a recent file backlog reduction exercise led by CFRG that was unrelated to the NEP pilot.
• Note that the timeline for the security clearance process is not currently being tracked by NEP as the focus is on green files.
• Almost all interviewees stated that the 21-day target was a challenge to meet, with suggestions for realistic timelines ranging from three to nine months.

Medical and Security Processes

• Based on NEP data from March 2024, the average amount of days for files to move from CFAT to completing the medical process is 40. Although most of the processing time remains in the processing of medical files, note that this is a significant decrease (39 percent) based on NEP’s January 2024 data, where the average was noted as 66 days.
• With regards to security processes, files with foreign implications currently take 18–24 months. NEP has implemented some efficiencies for the security process, such as allowing Director General of Defence Security to sign off on security (as opposed to CFRG having sole sign-off authority). NEP has also ensured participants awaiting clearance can still observe all trades, with Commanding Officers assuming the associated risk in this situation.

Naval Reserve Detachment Support

NRDs’ ability to support NEP is an influencing factor in the streamlined enrolment process; however, many interviewees were unaware of this recruitment support for NEP and noted that NRDs are short-staffed, with files often rerouted to CFRG. One interviewee occupying a senior leadership position noted that the resources and time taken to have CFRG recruitment teams train NRDs led to trade-offs and underperformance in many other areas of CFRG recruitment.

Ongoing Work

• CFRG is actively collaborating with Canadian Forces Health Services on health aspects, while CMP is currently providing support to conduct trials and adjustments for the Common Enrolment Medical Standards.
• Positive progress continues to be observed in the Recruit Medical Office’s response time, which has significantly decreased.
• CFRG is actively collaborating with Director General of Defence Security as well as Immigration, Refugee and Citizenship Canada on security aspects. This partnership aims to develop innovative solutions and streamline processes.
• NEP is developing a system to measure the time in the entry process that is within the control of NEP/CFRG. The clock will be paused whenever the process is in the applicant’s hands, providing a more accurate timeline of the enrolment process.

Pilot Sustainability

The sustainability of NEP is currently affected by the prioritization of green files (as opposed to changes to the enrolment process itself), availability of resources and supports such as NRD, and enrolment processes outside of NEP’s purview. Note that the 101-day timeline to process file enrolment is not currently an indicator of NEP or recruitment capacity.