COVID-19 tracking and evaluation 

Abstract

As people around the world experience a devastating pandemic, it is critical that policy-makers consider the methodological and measurement issues that might be associated with coronavirus disease 2019 (COVID-19) public health indicators. This commentary uses four primary variables to illustrate measurement and methodological issues that can complicate comparisons between jurisdictions. Jurisdiction refers to a variety of geographic areas, such as a country, a state, or a province/territory. These variables play a critical role in determining how we understand the trajectory of disease spread. These variables also contribute to our understanding of prevention strategies and their associated efficacy, reflecting the impact of COVID-19 on hospitals. It is critical for public health stakeholders and the public to recognize that these four simple variables can vary substantially across jurisdictions.

Introduction

The world is going through a devastating pandemic. People from around the globe have observed the coronavirus and how its associated coronavirus disease 2019 (COVID-19) has affected various jurisdictions. No areas have been spared. As we submit this comment for publication, there are 100,746,915 total confirmed cases, and 2,170,467 deaths worldwide^{Footnote 1}. Pathologists and medical experts have been working at an unprecedented pace to develop vaccines and treatments that can prevent and counteract COVID-19 and its adverse sequelae. Working in collaborative teams around the world, though promising interventions have been identified, most stakeholders do not expect to find a definitive solution within the upcoming months or perhaps years. Despite these energetic and enthusiastic efforts, public health government sectors have the responsibility to make urgent public health policy decisions to reduce and prevent the negative impacts of the COVID-19. To illustrate, statistical surveillance metrics have been used to guide screening and testing efforts as well as to limit personal movement between jurisdictions. These metrics have been used to advance public policy, protect health workers and citizens alike. How best to protect citizens by preventing the incidence of COVID-19 has become one of the most burning public health issues during these difficult times, and it will still remain even during vaccination dispensing phase and after it, for ongoing COVID-19 related surveillance.

During their decision-making process, public policy leaders rely on sparse and evolving scientific findings to guide their decision-making. These leaders recruit public health experts, epidemiologists, infectious disease experts, microbiologists and others to help interpret the scientific findings and provide insight into a rapidly changing landscape of evidence. However, because our understanding about the nature of this virus and its consequences is nascent in the scientific community, these decisions are difficult and complex. For example, setting policies around the length of time needed for confinement in a community is imprecise at best; in part, because infectious disease experts and epidemiologist do not fully understand the nature of the virus, even if the incubation period is understood.

To assist policy-making, decision-makers often look to other jurisdictions or countries to evaluate their practices and protocols. This tendency is closely linked with a variety of methodological and measurement issues. "Natural experiments" that compare different policies and approaches to the problem across areas are surely desirable. During a pandemic, every informed-based sources is welcome. However, analyses of what is being done in other countries or jurisdictions implies that the process of "making comparisons" rests on comparable data. Although commendable and necessary at this critical moment, these evaluations must be established on a shared foundation of evidence that is reliable and valid. Evidence must be established. For example, we must identify relevant and representative variables to make comparisons between areas. Further, we must use derive these comparisons from variables that are described clearly and operationalized precisely. Anything less leaves important comparisons as questionable and ultimately flawed. The bottom line is that we must compare "apples to apples" or we risk being misled.

The purpose of this brief comment is to identify some of the key variables that provide policy-makers with the information they might use to protect the public and track the spread of COVID-19. We primarily focus on this body of data and describe how readily observers can misinterpret it. We aim to discuss these fundamental variables to illustrate some of the problems associated with COVID-19 related evidence. However, we do not intend to discuss extensively the methodological issues raised in this paper. There are many fine texts to provide guidance about these investigative methods^{Footnote 2Footnote 3}. Our main goal is to offer a clarion call that will raise cautiousness and clarify nuances about the comparisons often made between jurisdictions and countries.

Analyses of the impacts of COVID-19 in the media and among scientists tend to yield to a great deal of information. Although there are numerous variables worthy of consideration, a discussion of all measures is beyond the scope of this commentary. Here we will limit our discussion to the following four main variables selected in part because these surveillance metrics are commonly reported by the media across a wide variety of jurisdictions:

1. Number of positive COVID-19 cases
2. Hospitalized cases
3. Cases in the intensive care units in hospitals
4. COVID-19 related deaths

These variables are sensible and principal indicators of the various impacts of the coronavirus. The central question is "to what extent can stakeholders use these variables and the evidence they generate to make sound, pertinent and reliable comparisons across jurisdictions"? It is worth repeating: the opportunity to compare evidence across jurisdictions associated with different public policies provides an important opportunity to conduct natural research. During the following examination, we will raise key questions about these four variables and consider how stakeholders use them when comparing the impact of COVID-19 among different countries or other jurisdictions. In addition, we will raise similar questions about the initial preventive measures that investigators implement to mitigate the impacts of COVID-19.

Number of positive COVID-19 cases

• What is the definition of a case? Is it confirmed by a standardized test or by clinical symptoms associated with COVID-19 with or without high-risk contacts?
• Is the reporting procedure associated with identified cases the same across jurisdictions?
• How many tests were conducted per X thousands of people?
• Is the availability and accessibility of tests similar across jurisdictions?
• Do physicians handle cases similarly across jurisdictions if symptoms are mild?
• Is the validity of screening tests (sensitivity and specificity) identical across jurisdictions?
• Is the number of cases reported based on the same ratio (i.e. X number cases per X thousands of inhabitants)?
• Which individuals were tested? Volunteers, at-risk, randomly selected?
• Are the screening criteria the same across jurisdictions?

Hospitalized cases

• Are the hospitalisation criteria for COVID-19 applied identically across jurisdictions?
• Is the access to hospitals comparable across jurisdictions?
• Is the availability of hospitals identical across jurisdictions?
• Is the cost of hospitalisation identical across jurisdictions?

Cases in intensive care units

• Do doctors working in hospitals use the same criteria to transfer a patient to the intensive care unit?
• Is the availability of hospital intensive care units identical across jurisdictions?
• Is the cost of an intensive care unit stay identical across jurisdictions?

Deaths caused by COVID-19

• Are the same procedures used across jurisdictions to identify the cause of a death?
• How the authorities identify COVID-19 as the cause of death among patients suffering from other medical conditions (co-morbidity)?
• Are all deaths occurring in different human service locations included (e.g. hospitals, long-term care facilities, personal residence, etc.)?
• Is the number of deaths reported in a given time period complete, final and consistent across jurisdictions?

Implementation of preventive measures

At this moment, there are three main preventive measures that policy-makers use to compare COVID-19 influences across jurisdictions: physical distancing, quarantine and the use of personal protective equipment. These measures have been the source of considerable and heated debates across and within jurisdictions. Before making comparisons across geographic areas, we need to raise the following questions:

• Are the procedures of confinement and physical distancing operationalized and applied identically way across jurisdictions?
• Are social networks complying, confining and physical distancing the same across jurisdictions?
• Is the enforcement of confinement and physical distancing identical across jurisdictions?
• Are protective personal equipment recommendations similar across different jurisdictions?
• Is protective personal equipment availability similar across different jurisdictions?

We can address the same questions to the quarantine/de-quarantine procedure. In addition, we can raise the following questions:

• On which basis did the governmental authorities allow confinement/de-confinement to take place?
• Was it allowed in a vast or progressive way?
• Was it monitored the same way across jurisdictions?

Discussion

In this commentary, we describe some of the primary variables that influence the generation of COVID-19-related evidence across jurisdictions. Certain variables and associated measures are straightforward, but stakeholders apply others inconsistently. These differences encourage faulty comparisons and make public health policy difficult to evaluate. When comparing different products or procedures, we emphasize that a basic methodological research requirement is that measurement of these products and procedures be identical, or at least very similar.

In the case of the coronavirus crisis, stating that one jurisdiction is "doing better" or "worse" than another is questionable and potentially dangerous when politically employed. Policy-makers need to keep in mind that comparing jurisdictions about the efficacy of the methods to control the impacts of the COVID-19, without knowing whether investigators applied and enforced the measures in identical or similar ways might not be as informative as intended or, worse, misleading.

Additionally, to improve surveillance measurement for COVID-19 national and international experts such as the World Health Organization could propose standardized measures to enable cross-nation comparisons. Finally, we encourage public health stakeholders and the public to evaluate this data and its nuances more carefully when they interpret and report on the impacts of COVID-19.

Authors' statement

All authors have approved the final version sent for publication and are accountable for all aspects of the work.

The content and view expressed in this article are those of the authors and do not necessarily reflect those of the Government of Canada.

Competing interests

The authors have no competing interests.

Funding

There was no funding associated with this manuscript preparation.