ARCHIVED - Evaluation of The Prion Diseases Program
4.0 Success
This portion of the evaluation contains an assessment of the success of the Prion Disease Program in meeting its stated objectives. It examines the completion of activities and outputs, the progress in achieving short, intermediate and long term outcomes, and if there have been any unexpected outcomes. Findings are derived from the document reviews, key interviews, the survey of health professionals and the Steering Committee focus group input.
Given the multiple streams of funding that the PDP received over the course of its existence, rating its success has been tied to the deliverables outlined in the related funding, reporting documents and the logic model developed for this evaluation, as listed in the table below.
Annexes G and H contain a detailed analysis of the outputs and outcomes and the findings of the evaluation. Below the findings are summarized.
4.1 Expected Outputs
Are activities being implemented as planned and producing the expected outputs?
4.1.1 CJDSS Activities and Outputs
Based on a detailed document review, supplemented by targeted interviews, the activities and outputs related to CJD surveillance were broadly implemented. While some of the specific dates for achievement of the activities and outputs are not available due to the elapsed time, the surveillance activities and outputs were implemented.
Since 2005-06, formal planning and reporting has been occurring for the PDP activities. A review of annual reports found that activities and outputs related to CJDSS are either accomplished or are in progress.
The basic elements of the CJD Surveillance System, such as the case and event investigations, diagnosis, basic database and key statistics are all working well. The Research Ethics Boards approvals have increased significantly from 5 to approximately 75 in the past three years and are managed systematically.
The survey of health care professionals found high levels of satisfaction with the services provided by CJDSS. Some dissatisfaction is noted with the turnaround times for some of the tests. These results are summarized in the table below. Caution must be exercised in interpreting this data due to the extremely small sample size.
Table 4: How satisfied are you with your contact with the CJD Surveillance System?
| Satisfied | Neutral | Dissatisfied | No Opinion | ||
|---|---|---|---|---|---|
| Case Investigations and Consultations | |||||
| Follow-up | 91% | -- | -- | 9% | |
| Access to Nurse Consultants | 91% | -- | -- | 9% | |
| Access to Senior Investigators | 73% | -- | -- | 27% | |
| Family support | 36% | -- | -- | 63% | |
| Laboratory testing | |||||
| Tests offered | 82% | -- | -- | 18% | |
| Turnaround Times | 14-3-3 | 55% | 9% | -- | 36% |
| Genetic | 55% | 9% | 9% | 27% | |
| Neuropathology | 45% | 18% | 18% | 18% | |
| Support for Interpretation | 14-3-3 | 63% | -- | -- | 36% |
| Genetic | 73% | -- | 9% | 18% | |
| Neuropathology | 73% | 9% | -- | 18% | |
| Usefulness of Documentation | 82% | -- | -- | 18% | |
During the survey, a consistent theme of the comments by health care professionals was the high degree of professionalism and dedication of the CJDSS staff. Further, the PDP is recognized as an accessible, knowledgeable expert. Two less common messages were some concerns over the availability of documentation and service in French, and the relative priority of tracking a rare disease such as CJD over more common ones.
The case database is maintained and enables regular statistical reporting on the PHAC website and periodic submission of statistics to EuroCJD. The CJDSS also collects descriptive information in a lengthy questionnaire. Ten years of this data is on file however has not been entered or analyzed which limits the possibility for related epidemiological work and/or publication, noting that the original rationale for a lengthy questionnaire was to perform a case-control study of CJD. However this has not been possible to date and similar studies have now been published by other groups. This was also noted in the EuroCJD audit of 2005 along with a recommendation to consider making the questionnaire more specific.
Interview comments also indicated that the CJDSS would benefit from the standard operating procedures (SOPs) to ensure program consistency and transition when there are new staff members. The need for SOPs for the database was also identified in the 2005 EuroCJD audit.
During the interviews with PHAC and key stakeholders, interviewees were asked to rate on a five point scale how successful they felt the PDP had been in implementing its activities in this area. The results of the 15 interviews are shown below. Caution must be exercised in interpreting this data due to the small sample size.
| Area | Successful | Neutral | Poor | No Opinion |
|---|---|---|---|---|
| CJD Surveillance System | 53% | 27% | -- | 20% |
4.1.2 Applied Research and Reference Services
For the Applied Research and Reference Services, the key activities and outputs were met. However, there were expectations at the time that new diagnostic and screening methods would be developed and could be then implemented. This has proved to be quite optimistic from a scientific point of view. In the past ten years no definitive diagnostic methods have been developed world-wide. Research continues in Canada and around the world on this.
The key reference tests, the 14-3-3 CSF test and the PRNP sequencing, are being delivered using in-house and the successful ISO accreditation process has improved turnaround times and standardization. Consistent messages were expressed that the turnaround times for neuropathology analysis are too long and that this diagnostic tool is under-resourced. The Steering Committee focus group indicated some concerns that turnaround times for the 14-3-3 CSF test lags those of other labs conducting similar tests however, this is of less concern in that the results are typically not urgently required. Physician interviews identified an interest in receiving the results from genetic testing.
Time to implement technical and process improvements to the reference services may be affected by capacity. Examples of improvements include test accreditation, validating existing tests to expand the lab’s offered services and the implementation of laboratory information management system (LIMS).
Applied research is perceived as being somewhat limited to date, due to capacity constraints.
During the interviews with PHAC and key stakeholders, interviewees were asked to rate on a five point scale how successful they felt the PDP had been in implementing its activities in this area. The results of the 15 interviews are shown below. Caution must be exercised in interpreting this data due to the small sample size.
Table 6: Results of Interviewees Rating of Success
N=15
| Area | Successful | Neutral | Poor | No Opinion |
|---|---|---|---|---|
| Applied Research and Reference Services in Prion Diseases | 20% | 13% | 7% | 60% |
4.1.3 Pathobiology Research
Based on a detailed document review, supplemented by targeted interviews, the activities and outputs were broadly implemented with the exception of:
- Research on transmission through iatrogenic routes;
- Collaboration with industry on fractionation;
- Unclear from historical documentation whether specific research activities were undertaken in all indicated areas (see Annex G for details); and
- Establishment of an experimental non-human-primate model of Chronic Wasting Disease to assess the risk of transmission to humans.
For the first point, resource constraints reportedly hindered PDP’s ability to meet these deliverables. The collaboration with industry on fractionation was terminated by industry due to concerns over health and safety. While research has been underway for some time, a lack of documentation from early years impeded the ability to confirm if all of the specifically identified research had been conducted. For example, specific topics identified included the risk of prion transmission through blood transfusions, iatrogenic transmission, and tonsil and appendix studies.
In recent years, there has been significant growth in the success of the research conducted and the number of collaborationsFootnote 2. The research scientists have been recognized domestically and internationally for their scientific successes Footnote 3 Footnote 4 and ability to develop and apply novel scientific techniques. The prion disease expertise within PHAC has increased considerably as a result.
The unit has published refereed papers, conference presentations, posters and abstracts. There has been an increase in publication of research over time, though according to the Steering Committee focus group, the peer-reviewed publication productivity can be considered moderate. From 2006-2008, 8 of PDP’s 11 refereed papers were published by the research groupFootnote 5.
The original forms used by the CJDSS did not request permission to use the human tissue samples collected for research of any kind. Revised forms which have recently been submitted to Research Ethics Boards (REB) for approval include a dedicated request for use of surplus specimens in research. While research is not allowed to be conducted on the human samples collected through the surveillance system in the past impeding human-based research, the process is underway to remedy this situation.
During the interviews with PHAC and key stakeholders, interviewees were asked to rate on a five point scale how successful they felt the PDP had been in implementing its activities in this area. The results are of the 15 interviews are shown below. Caution should be exercised in interpreting the data due to the small sample size.
Table 7: Results of Interviewees Rating of Success
N=15
| Area | Successful | Neutral | Poor | No Opinion |
|---|---|---|---|---|
| Pathobiology Research in Prion Disease | 47% | -- | 7% | 47% |
Broadly, the activities are being implemented and are producing the expected outputs.
CJDSS is successfully conducting case and event investigations, maintaining the database and publishing basic statistics, although there is limited additional analysis, research or publication of the surveillance data. Research Ethics Board approvals have increased significantly. Health care professionals in contact with the CJDSS are very satisfied with the services provided, with the exception of turnaround times for neuropathology results.
Reference Services are consistently providing the testing analysis and have shown process improvements in terms of ISO certification for specific tests and improved turnaround times. Neuropathology analysis takes an extended time to complete. Some planned improvement activities, while in progress are not being completed in a timely manner, possibly related to resource constraints.
Research has increased the number of collaborations both nationally and internationally leading to a greater access to funding from a wide range of agencies. The unit has developed research capacity since its inception in 1998, and has demonstrated an increase in the number of refereed papers, conference presentations, posters and abstracts, and can be considered to have moderate research publication productivity.
4.2 Outcomes
Have the expected outcomes and reach been achieved through program activities?
Annex H contains a list of each of the outcomes identified in the original program mandate, the Annual Reports and the 2008 Logic Model along with a brief analysis of the success that has been achieved.
The outcomes have been achieved with the exception of the development of new diagnostic methods which, as identified earlier, may have been an aggressive expectation.
In summary, the CJDSS is considered to be successful with respect to providing effective surveillance and public health intervention. Active surveillance is ongoing using internationally accepted methods. Canadian surveillance is finding the number of CJD cases expected – in the range of internationally predicted frequency of cases (1 - 2 per million per year). The readily available 14-3-3 test serves to cast a wide net for referrals in Canada. The focus group consensus was that this is being met. The Canadian system compares favourably to the European model and is more comprehensive than the US system.
However, there are opportunities for further improvements to the surveillance system. Specifically, CJDSS could be making better use of data in terms of publishing and sharing. In addition, there is a widely recognized need for the surveillance system to become better linked with provinces and territories.
The laboratory reference services are also successful, and appear to be fulfilling the required functions with good high, quality work and ISO Accreditation in place. Implementation of the identified technical and process improvements to the references services is acknowledged as important.
Improving awareness of CJD is thought to be somewhat successful. Some activities have been implemented however resource and capacity constraints limit the success in this area.
Overall, the research activities of the PDP are very successful. The focus group concluded the research undertaken is of high quality and is contributing to the understanding of prion diseases. While productivity in terms of peer-reviewed publications may be lower than some other research groups, it appears to be increasing. There are opportunities to more clearly articulate the research priorities of the PDP.
During the interviews with staff and stakeholders, the interviewees were asked to rate the success of the PDP in achieving its stated objectives on a five point scale. Health care professionals were only asked about the first three objectives as they have limited awareness of the research and technical improvements. A significant number of people were unable to offer an opinion on all of the objectives as they have contact or awareness with a limited number of the PDP activities.
Overall, the survey found that the PDP is perceived to be successful in achieving its objectives, particularly in the core activities of surveillance, reference services and research. Success is technical improvements and improving awareness of CJD is thought to be slightly less.
| Objective | Well | Neutral | Poorly | No Opinion |
|---|---|---|---|---|
| Effective surveillance and public health interventionN1 | 53% | 10% | 3% | 33% |
| Laboratory reference services N1 | 63% | 3% | 7% | 27% |
| Improving awareness of CJD N1 | 50% | 17% | 13% | 27% |
| Improving the understanding of prion diseases through researchN2 | 37% | 11% | -- | 53% |
| Technical improvements N2 | 16% | 11% | 16% | 58% |
*NOTE: Health Professionals were not questioned regarding these areas
The PDP has broadly met its outcomes as set out in a range of funding and reporting documents generated since its inception in 1998. Some objectives based on expected diagnostic tests were over-ambitious and so have not been met.
The CJDSS is considered to be successful with respect to providing effective surveillance and public health intervention. It is finding the expected number of cases of CJD annually in Canada and using internationally accepted methods of surveillance.
The laboratory reference services are also successful, and appear to be fulfilling the required functions with high, quality work and ISO Accreditation in place.
The success to date in improving awareness of CJD is thought to be somewhat less, with resource and capacity constraints limiting the success in this area.
Overall, the research activities of the PDP are very successful, considered to be of high quality and are contributing to the understanding of prion diseases. There are opportunities to more clearly articulate the research priorities of the PDP.
4.3 Unexpected Outcomes
Have there been any unexpected outcomes?
The degree of interest in domestic classical CJD from physicians, families and the media has exceeded initial expectations and queries are fielded as needed. The involvement of the nosocomial side of CJD through guidelines and greater community involvement (e.g. Alzheimer’s Society) also surpassed original expectations.
Given the range of scientific developments since 1998, considerable strides have been made in the science, particularly in the areas of genomic technologies, and microarrays which have cross-applications to other diseases.
Conclusions:
There were no significant unexpected outcomes.
Page details
- Date modified: