From STAT:
By JOHN P.A. IOANNIDIS MARCH 17, 2020
Ioannidis is the highly respected Stanford epidemiologist / statistician whose 2005 paper “Why Most Published Research Findings Are False” has been immensely influential on what is now known as the Replication Crisis.
The current coronavirus disease, Covid-19, has been called a once-in-a-century pandemic. But it may also be a once-in-a-century evidence fiasco.
At a time when everyone needs better information, from disease modelers and governments to people quarantined or just social distancing, we lack reliable evidence on how many people have been infected with SARS-CoV-2 or who continue to become infected. Better information is needed to guide decisions and actions of monumental significance and to monitor their impact.
Draconian countermeasures have been adopted in many countries. If the pandemic dissipates — either on its own or because of these measures — short-term extreme social distancing and lockdowns may be bearable. How long, though, should measures like these be continued if the pandemic churns across the globe unabated? How can policymakers tell if they are doing more good than harm?
Vaccines or affordable treatments take many months (or even years) to develop and test properly. Given such timelines, the consequences of long-term lockdowns are entirely unknown.
The data collected so far on how many people are infected and how the epidemic is evolving are utterly unreliable. Given the limited testing to date, some deaths and probably the vast majority of infections due to SARS-CoV-2 are being missed. We don’t know if we are failing to capture infections by a factor of three or 300. Three months after the outbreak emerged, most countries, including the U.S., lack the ability to test a large number of people and no countries have reliable data on the prevalence of the virus in a representative random sample of the general population.
Emil O W Kirkegaard asks: “Why don’t they do random sampling tests for Corona? Same as they do polls for elections. Do them every day, 1000 representative people. Why not? Surely, the governments of the world can figure out how to do random sampling and get those 1000 tests done.”
Every day in my neighborhood, for example, there are lots of families out on the sidewalks during the day. People are carrying on cheerful conversations from 10 feet apart. Community feeling is strong. Publicize on neighborhood websites like Nextdoor.com that an official survey will be done by testers in white coats walking door to door on Saturday. They’ll do the test on your front porch without coming into your house. Each survey person was tested on Friday and proved negative. You could quickly get a local fairly random sample (say 10 people in each of 100 neighborhoods — or do we need 10,000 people?
It could be that the virus is more widespread than assumed, which would suggest the death rate is lower. Or it could be that it’s still rare, which would be useful to know.
Let’s make this random testing happen.
Here’s computer guru Danny Hillis saying the same thing.
Let’s test in the Seattle area this weekend. Seattle is not lacking in high-profile billionaires, such as Gates, Bezos, or Schulz, who can write a check to cover it.
We can do this.
This evidence fiasco creates tremendous uncertainty about the risk of dying from Covid-19. Reported case fatality rates, like the official 3.4% rate from the World Health Organization, cause horror — and are meaningless. Patients who have been tested for SARS-CoV-2 are disproportionately those with severe symptoms and bad outcomes. As most health systems have limited testing capacity, selection bias may even worsen in the near future.
The one situation where an entire, closed population was tested was the Diamond Princess cruise ship and its quarantine passengers. The case fatality rate there was 1.0%, but this was a largely elderly population, in which the death rate from Covid-19 is much higher.
Projecting the Diamond Princess mortality rate onto the age structure of the U.S. population, the death rate among people infected with Covid-19 would be 0.125%. But since this estimate is based on extremely thin data — there were just seven deaths among the 700 infected passengers and crew — the real death rate could stretch from five times lower (0.025%) to five times higher (0.625%).
Perhaps. Or it could turn out that the Diamond Princess was a fortuitous environment and an Italian scenario is more common. Or it could turn out that one of these new treatments works and the death rate a month from now is even lower. Or many other possibilities. At this point, the future is unwritten.
A big issue is what we might call Data Hygiene. For example, Italy has a high death rate while, so far, Germany has a low death rate. Is that due to actual differences or due to methodological differences in how numbers are reported? I don’t have a clue, but somebody out there has a clue, so we should listen to them.
… Adding these extra sources of uncertainty, reasonable estimates for the case fatality ratio in the general U.S. population vary from 0.05% to 1%.
1% case fatality rate of 60% infected is about 2 million deaths.
On the other hand, another way to measure this is years lost. For example, I’m 61. I don’t have all that many years left, and I’ve enjoyed a fine life. It’s not that big of a tragedy if this thing takes me down. (On the other hand, personally speaking, I’m very much against that outcome, so I’m washing/sanitizing my hands 60 times per day and turning on and off light switches with my elbow.)
If this is only going to kill or maim us old folks, well, that’s not so bad. But … how much statistical confidence we can have in the age structure of the effects is vague. Moreover, would this novel disease condemn many people in the prime of life to major lung problems from this point onward? At present, who knows?
That huge range markedly affects how severe the pandemic is and what should be done. A population-wide case fatality rate of 0.05% is lower than seasonal influenza. If that is the true rate, locking down the world with potentially tremendous social and financial consequences may be totally irrational. …
… If we assume that case fatality rate among individuals infected by SARS-CoV-2 is 0.3% in the general population — a mid-range guess from my Diamond Princess analysis — and that 1% of the U.S. population gets infected (about 3.3 million people), this would translate to about 10,000 deaths. This sounds like a huge number, but it is buried within the noise of the estimate of deaths from “influenza-like illness.”
On the other hand, other estimates have been a 60% infection rate before herd immunity kicks in.
It’s hard to say.
Some worry that the 68 deaths from Covid-19 in the U.S. as of March 16 will increase exponentially to 680, 6,800, 68,000, 680,000 … along with similar catastrophic patterns around the globe. Is that a realistic scenario, or bad science fiction? How can we tell at what point such a curve might stop?
The most valuable piece of information for answering those questions would be to know the current prevalence of the infection in a random sample of a population and to repeat this exercise at regular time intervals to estimate the incidence of new infections. Sadly, that’s information we don’t have.
Let’s get that information.
The U.S. has been shamefully behind in testing capacity, but we have immense resources to catch up with.
For example, a couple of days ago, I heard 4th hand that a lady had made her own COVID-19 PCR test to test her kids and their friends on her block. Granted, she works on CRISPR bioengineering projects, but still …
Similarly, the immensely well-funded Broad Institute, a Harvard-MIT joint venture that is home, among much else, to the famous paleogenomics lab of David Reich and Nick Patterson, will be processing a 1,000 COVID tests per day next week, and can ramp up to much more than that in the future. I’m a huge fan of paleogenomics, but I can live without new papers on the subject for the next year or so if these world-class brains devote themselves to The Effort for The Duration.
In the absence of data, prepare-for-the-worst reasoning leads to extreme measures of social distancing and lockdowns. Unfortunately, we do not know if these measures work. School closures, for example, may reduce transmission rates. But they may also backfire if children socialize anyhow, if school closure leads children to spend more time with susceptible elderly family members, if children at home disrupt their parents ability to work, and more. School closures may also diminish the chances of developing herd immunity in an age group that is spared serious disease.
The good news about school closures is that if this turns out to be a fiasco, we can just extend the school year into summer vacation. While a lot of businesses are likely to default on debts due to lockdowns, the only U.S. students likely to be permanently greatly hurt are high school seniors on the verge of not graduating who have been slacking off so far this semester but who would do just enough in the second half of their final semester to graduate. For high school seniors who have already applied to college, this semester is trivial.
In contrast, businesses, the self-employed, and the newly unemployed are in danger of suffering severe consequences due to default provisions on loans. So my impression is that schools are safer to engage in an abundance of caution with than businesses. But your impressions may differ.
This has been the perspective behind the different stance of the United Kingdom keeping schools open, at least until as I write this.
The UK just shut its schools, although health care workers can still drop their kids off at school as daycare, which seems wise.
It could be that shutting schools is an overreaction due to the unusual age profile of this virus, but… schools get shut down all the time: for teacher strikes, snow days, smoke days, etc. It’s not the end of the world. Good students will study online and bad students won’t, much like in the classrooms.
Of course, the exact age-profile of who is at risk is even more uncertain than the overall death rate.
In the absence of data on the real course of the epidemic, we don’t know whether this perspective was brilliant or catastrophic.
Flattening the curve to avoid overwhelming the health system is conceptually sound — in theory. A visual that has become viral in media and social media shows how flattening the curve reduces the volume of the epidemic that is above the threshold of what the health system can handle at any moment.
Yet if the health system does become overwhelmed, the majority of the extra deaths may not be due to coronavirus but to other common diseases and conditions such as heart attacks, strokes, trauma, bleeding, and the like that are not adequately treated.
Indeed.
If the level of the epidemic does overwhelm the health system and extreme measures have only modest effectiveness, then flattening the curve may make things worse: Instead of being overwhelmed during a short, acute phase, the health system will remain overwhelmed for a more protracted period. That’s another reason we need data about the exact level of the epidemic activity.
Indeed.
One of the bottom lines is that we don’t know how long social distancing measures and lockdowns can be maintained without major consequences to the economy, society, and mental health. Unpredictable evolutions may ensue, including financial crisis, unrest, civil strife, war, and a meltdown of the social fabric. At a minimum, we need unbiased prevalence and incidence data for the evolving infectious load to guide decision-making.
Indeed.
In the most pessimistic scenario, which I do not espouse, if the new coronavirus infects 60% of the global population and 1% of the infected people die, that will translate into more than 40 million deaths globally, matching the 1918 influenza pandemic.
Which was bad.
The vast majority of this hecatomb would be people with limited life expectancies. That’s in contrast to 1918, when many young people died.
We hope.
Look here, Jack, we are the United States of America, the most kick-ass country ever. Running a scientifically valid random sample survey is not beyond our national capabilities. So …
LET’S GET THE DATA.
