Guidelines for systematic reviews of economic evaluations of vaccination programs: National Advisory Committee on Immunization (NACI)

February 2022

• Previous
• Next

See in PDF format the Guidelines for systematic review of economic evaluations of vaccination programs.

On this page

• Purpose for guidelines
  • Planning and development
    • Policy question, PICO (TS)
    • Inclusion and exclusion criteria
    • Existing or similar systematic reviews
    • Search strategy
      • Peer reviewed literature
      • Grey literature
      • Stakeholder engagement
    • Protocol
    • Registration with PROSPERO
  • Screening and study selection
    • Title and abstract screening and full-text screening
    • Handsearching
  • Appraisal
    • Quality appraisal
    • Vaccine model-specific appraisal
    • Appraisal of applicability
  • Data extraction
  • Synthesis
    • Studies included for synthesis
    • Currency
    • Narrative and graphical synthesis
  • Reporting
  • Templates for reporting
• Abbreviations
• References
• Version history

Purpose of guidelines

This document outlines the steps for conducting a systematic review on economic evaluations of vaccination programs for the National Advisory Committee on Immunization (NACI). This is based on existing methodological guidance for systematic reviews and adapted to meet the needs of NACI.^{Footnote 1Footnote 2Footnote 3Footnote 4}

Systematic reviews on economic evaluations for NACI are to be conducted by the Secretariat or by its contractors, in consultation with the NACI Working Group.

1. Planning and development

1.1 Policy question, PICO(TS)

The policy question should be defined in terms of PICO(TS): population, intervention, comparator(s), outcome(s) of interest; and where relevant, timing, type of study, and setting.

1.2 Inclusion and exclusion criteria

At minimum, the inclusion and exclusion criteria should specify: study population, intervention, date range, language(s) searched (English and preferably French), restrictions to countries or jurisdictions, and type of study (full economic evaluations). Note that full economic evaluations refer to cost-utility analyses (CUA), cost-effectiveness analyses (CEA), and cost-benefit analyses (CBA).

1.3 Existing or similar systematic reviews

Once the policy question and inclusion/exclusion criteria have been developed, scan the literature for existing or similar systematic reviews before proceeding. Use databases such as Cochrane Database of Systematic Reviews,^Footnote 5 Database of Abstracts of Reviews of Effects (DARE)^Footnote 6— specifically, their Health Technology Assessment (HTA) Library and National Health Service Economic Evaluation Database (NHS EED)— and the International Network of Agencies for Health Technology Assessment (INAHTA) International HTA Database.^Footnote 7 If a relevant and up-to-date systematic review is identified, a new review may not be needed. Alternatively, an existing systematic review may need to be updated.

1.4 Search strategy

If a new systematic review is needed, proceed to developing the search strategy. This should be done in consultation with a librarian.

1.4.1 Peer-reviewed literature

NACI strongly recommends the use of the following search filters/hedges: (i) Canadian Agency of Drugs and Technologies in Health (CADTH) "Economic Evaluations/Cost/Economic Models" search filter;^Footnote 8 (ii) McMaster Health Information Research Unit "Costs" and "Economics" search hedges;^Footnote 9 or (iii) Scottish Intercollegiate Guidelines Network (SIGN) "Economic studies" search filter.^Footnote 10

At minimum, the following three electronic databases should be searched: MEDLINE, EMBASE, and EconLit. Other recommended databases include: Cochrane Library, DARE (which includes the HTA Library and NHS EED), and INAHTA International HTA Database.

1.4.2 Grey literature

Firstly, NACI recommends the use of the CADTH Grey Matters tool, which is a checklist of grey literature sources including Canadian and international health technology assessment agencies.^Footnote 11 There is also a CADTH Grey Matters Light tool for a concise shortlist.^Footnote 12 Contact organizations directly for grey literature that is not publicly available.

Secondly, NACI recommends searching national immunization technical advisory group (NITAG) websites. At minimum, search the UK (Joint Committee on Vaccination and Immunisation, JCVI), US (Advisory Committee on Immunization Practices, ACIP), Germany (Standing Committee on Vaccination, STIKO), and Australia (Australian Technical Advisory Group on Immunisation, ATAGI).

Thirdly, NACI recommends searching for preprints. Consider searching the medRxiv, arXiv, SSRN Preprints, and Research Square servers. Recognize that the search interface of preprint servers is not ideal, so consider searching sites that compile preprints from multiple server, such as PubMed Central Europe and Open Science Framework (OSF) preprints.

1.4.3 Stakeholder engagement

The Secretariat will connect with stakeholders (e.g., subject matter experts, industry, NITAGs) for information on economic evaluations under development or those that have yet to be published. They may be included in the review depending on the stage of development. Studies without results will not be included in the systematic review. However, these studies and their timelines for anticipated results may be listed in the Appendix if the stakeholders agree to have the information made publicly available (see Appendices in section 6, "Reporting"). The Secretariat will discuss with stakeholders to agree on information from their unpublished studies that will be reported on, and thus, made public.

1.5 Protocol

The protocol should be developed in consultation with the NACI Working Group as an iterative process. At minimum, the protocol should include: policy question defined in terms of PICO(TS), inclusion/exclusion criteria, search strategy, screening (e.g., how many reviewers, how disagreements will be resolved), data extraction (e.g., how will inter-rater reliability be assessed), quality appraisal, analysis (e.g., will subgroup analyses be conducted, how publication bias will be assessed). The protocol should also list the reviewers' conflicts of interests.

1.6 Registration with PROSPERO

The systematic review of economic evaluations should be registered with the International Prospective Register of Systematic Reviews (PROSPERO) during the initial stages of research development. PROSPERO^Footnote 13 is an international database of registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome. Features from the protocol are recorded and maintained as a permanent record. PROSPERO is produced by the Centre for Reviews and Dissemination and funded by the National Institute for Health Research (NIHR). Consider publishing the protocol.

2. Screening and study selection

2.1 Title and abstract screening and full-text screening

Screen titles and abstracts as well as full-texts based on the inclusion/exclusion criteria. If there are two or more reviewers, it is recommended that reviewers pilot the screening to ensure consistency. For example, reviewers may pilot batches of randomly selected titles/abstracts and repeat until inter-rater reliability surpasses a predefined threshold. Conflicts should be resolved through discussion, and if needed via an additional reviewer. Document any notable excluded studies and the reasons for their exclusion.

2.2 Handsearching

Handsearch the bibliographies of included studies (e.g., publications, conference proceedings) to identify any additional studies that may not have been indexed in the databases searched.

3. Appraisal

It is preferable that the following appraisals are done in duplicate with two reviewers.

3.1 Quality appraisal

Several quality appraisal tools exist to assess the methodological and reporting quality of full economic evaluations. Some are specific to model-based economic evaluations and some are specific to trial-based economic evaluations.^{Footnote 14Footnote 15Footnote 16Footnote 17Footnote 18Footnote 19Footnote 20Footnote 21Footnote 22} NACI recommends the use of the Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Economic Evaluations,^Footnote 23 but alternative or additional quality appraisal tools may be used after consultation with the NACI Working Group based on the policy question. Note that the JBI Checklist does not generate a summary score.

Reviewers should determine the essential appraisal questions based on the policy question. Studies can be considered "high quality" if they satisfy essential appraisal questions and do not have any severe deficiencies. Studies can be considered "acceptable" if they satisfy essential appraisal questions, but have some minor deficiencies in other sections of the quality appraisal. Studies can be considered "unacceptable" if they have clear issues across essential questions in the quality appraisal.

After this appraisal, reviewers should consider the appropriateness of excluding low quality studies in the analysis. For instance, reviewers could report only the "best evidence" by including studies deemed high quality or acceptable and excluding unacceptable studies.^Footnote 24 Exclusion of low quality studies may reduce bias in the analysis.

3.2 Vaccine model-specific appraisal

To complement the JBI Checklist, consider the following three critical appraisal questions on modelling from the World Health Organization (WHO) guide for standardization of economic evaluations of immunization programmes, Table 17 "A Checklist for appraising the quality of economic evaluations of immunization programmes":^Footnote 25

• Are the model structure and implicit or explicit assumptions clearly described?
• Is the model type (static, dynamic or stochastic) clearly stated and justified in light of likely changes to the force of infection and the role of chance in the transmission process? Have the model's strengths and weaknesses been discussed?
• Has the model been validated? If so, has it been validated in as many facets of validation as possible?

In the appraisal, reviewers should consider vaccine-specific elements including:

• Community immunity (also known as herd immunity)
• Natural immunity
• Supplies (e.g., vaccines, syringes, safety boxes)
• Public health costs (e.g., contact tracing)
• Disease surveillance
• Distribution system (e.g., transport and cold storage)
• Vaccine wastage and waste management

3.3 Appraisal of applicability

Applicability (also known as transferability) refers to the appropriateness of using or adapting evidence from other countries/jurisdictions for local decision-making.^Footnote 26 NACI recommends the use of either tool to assess the applicability of included studies to the Canadian context:

• Heyland's generalizability criteria^Footnote 27
• Antonanzas' transferability index^Footnote 28

The latter tool includes a checklist and formula. NACI does not require the formula to be used.

4. Data extraction

Proper documentation is required. The development of the data extraction form should be done in consultation with the NACI Working Group. The minimal data elements required include study and population characteristics, key model inputs (e.g., vaccine cost, vaccine efficacy or effectiveness, epidemiology), outcomes (e.g., clinical, costs, and cost-effectiveness outcomes), and influential parameters. Please see the "Templates for reporting" section of this document or the NACI tool called Presentation Template for Presenting Systematic Reviews. If the entire data extraction cannot be completed in duplicate, consider verifying a subset of the extracted data with an independent reviewer.

Reviewers should contact authors of included studies if additional information or clarification is required. Document who was contacted, and whether multiple attempts were made to contact them. The NACI Secretariat will coordinate any correspondence with industry if required.

5. Synthesis

5.1 Studies included for synthesis

Reviewers should determine if they are synthesizing all studies or select studies. A subset of studies may be synthesized or a stratified analysis may be conducted based on study quality or potential conflicts of interest. Study quality and potential conflicts of interest may explain heterogeneity of cost-effectiveness results. Reviewers should explore other clinical and methodological factors contributing to heterogeneity. Results should be synthesized separately for model-based economic evaluations vs. non-model-based economic evaluations (e.g., trial-based, studies based on administrative data). For model-based studies, synthesize key model inputs to assess face validity and applicability to the Canadian context. For non-model-based studies, describe the analyses (e.g., protocol-drive care vs. clinical practice; handling of missing, censored, skewed data).

5.2 Currency

Adjust the cost-effectiveness results (such as incremental cost-effectiveness ratios, ICERs, or net health/monetary benefit) to present day Canadian dollars. Convert the currency to Canadian dollars using the Organization for Economic Co-operation and Development's (OECD) purchasing power parity rates. Then inflate using the Bank of Canada's inflation rates. Keep the unadjusted and adjusted ICERs for analysis.

5.3 Narrative and graphical syntheses

Use a combination of narrative syntheses and graphical syntheses where possible. NACI does not require a meta-analysis of cost-effectiveness results.

For the narrative synthesis, reviewers should summarize key findings, the appraisals conducted (i.e., quality appraisal, vaccine model-specific appraisal, applicability), and any subgroup analyses and stratified analyses conducted. Comment on whether there was a consensus among studies or if there was too much heterogeneity. Reviewers should describe any parameters reported to be influential to results, as well as the results of sensitivity and scenario analyses conducted by the included studies. A section should be dedicated to highlighting and comparing Canadian studies. Reviewers should discuss limitations, publication bias, and draw comparisons to other systematic reviews if applicable.

For the graphic synthesis, please see Templates for Reporting for examples of evidence tables and figures to use and edit at the reviewers' discretion. When listing cost items in evidence tables, report original costs in local currency. When synthesizing cost-effectiveness results in evidence tables, report adjusted results in present day Canadian dollars. It is preferable to report both discounted and undiscounted results, where possible.

Reviewers should avoid stating policy implications and any references to explicit or implicit cost-effectiveness thresholds. Policy implications are the responsibility of NACI. For example, reviewers may not say, "Based on the systematic review, the intervention appears to be cost-effective". Reviewers may say, "Most included studies (N = 9) concluded that the intervention is cost-effective based on their respective regional thresholds used".

6. Reporting

Reporting should be done in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.^Footnote 4 Below is the basic structure required:

1. Executive summary/ Abstract
2. Disclosure of reviewers' conflicts of interest
   • Includes financial and intellectual conflicts of interest
   • Describe how the study was funded and the role of the funder in the identification, design, conduct, and reporting of the analysis. Describe other non-monetary sources.
3. Introduction
4. Methods
   • Policy question defined in terms of PICO(TS)
   • Description of search strategy, inclusion/exclusion criteria, screening, data extraction, and synthesis
   • List appraisal tools (i.e., quality appraisal, vaccine model-specific appraisal, applicability)
   • PROSPERO registration number
5. Results
   • Overview of included studies
   • Key findings
     • Was there a consensus among studies? Were the studies too heterogeneous?
     • Preferable to report both discounted and undiscounted results
     • Describe any subgroup analyses or stratified analyses
     • Describe influential parameters on the cost-effectiveness results
     • Describe sensitivity and scenario analyses of included studies
     • Highlight and compare Canadian studies
   • Quality appraisal
   • Vaccine model-specific considerations
   • Applicability to Canadian context
6. Discussion and conclusions
   • Draw conclusions on cost-effectiveness of vaccine program
   • Describe limitations and strengths of the included studies
   • Describe limitations and strengths of the review
   • Discuss applicability of included studies to Canadian context (e.g., populations and comparators assessed in the studies, regional differences in terms of disease epidemiology, population characteristics, clinical practice patterns, resource-use patterns, unit costs, and other factors of relevance). Where differences exist, discuss the impact on the results (expected direction and magnitude), and the conclusions. Key parameters to discuss are vaccine price, vaccine effectiveness, and epidemiology.
   • Describe how sources of funding and potential conflicts of interest may affect cost-effectiveness results
   • Comment on publication bias
   • Compare to other systematic reviews, if applicable
7. 7. Tables/Figures
   • Figure: PRISMA flow diagram
   • Table of study and population characteristics
   • Tables/ figures on cost-effectiveness results
8. Appendices
   • Full search strategy
   • Table of notable excluded studies (optional)
   • Tables/figures of appraisals (i.e., quality appraisal, and vaccine model-specific appraisal, applicability)
   • Economic evaluations under development or that are unpublished (e.g., gathered from stakeholder engagement if stakeholders agree to have information made publicly available)– list of studies and timelines for anticipated results (optional)

Templates for reporting

The following tables and figures are examples. Reviewers may use and edit at their discretion.

Appendix figures:

Quality appraisal: Proportion of appraisal items met in each study

Vaccine model-specific appraisal: Proportion of appraisal items met in each study

Transferability appraisal: Proportion of criteria considered generalizable/ applicable in each study

Figure 1 - Text Description

Figure 1 - Text description: This generic figure shows a horizontal stacked bar graph. Each horizontal bar represents a study, labelled by its author and year of publication on the y axis. The x axis is numerical, ranging from 0% to 100% and represents the percentage of items/criteria met by each study. For example, if using this figure to depict the results of the quality appraisal, the stacked bar graph will show how many of the appraisal items in the checklist were met, not met, or were unclear. To show this visually, each study's horizontal bar is comprised of up to three segments: green represents "Yes", red represents "No", and grey represents "Unclear". The segments add up to 100%.

Abbreviations

ACIP

Advisory Committee on Immunization Practices

ATAGI

Australian Technical Advisory Group on Immunisation

CADTH

Canadian Agency of Drugs and Technologies in Health

CBA

Cost-benefit analysis

CEA

Cost-effectiveness analysis

CUA

Cost-utility analysis

DARE

Database of Abstracts of Reviews of Effects

DSA

Deterministic sensitivity analysis

HTA

Health technology assessment

ICER

Incremental cost-effectiveness ratio

INAHTA

International Network of Agencies for Health Technology Assessment

JBI

Joanna Briggs Institute

JCVI

Joint Committee on Vaccination and Immunisation

NACI

National Advisory Committee on Immunization

NIHR

National Institute for Health Research

NITAG

National Immunization Technical Advisory Group

NHS EED

National Health Service Economic Evaluation Database

OECD

Organization for Economic Co-operation and Development

PICO(TS)

Population, intervention, comparator(s), and outcome(s) of interest (specific timing or type of study and setting)

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

PROSPERO

International Prospective Register of Systematic Reviews

PSA

Probabilistic sensitivity analysis

QALY

Quality-adjusted life year

SIGN

Scottish Intercollegiate Guidelines Network

STIKO

Standing Committee on Vaccination

VE

Vaccine effectiveness

WHO

World Health Organization

• Previous
• Next