Better in their words…

Before we begin to cover current articles, we’re going to dive into some of the topics that are particularly rich veins for the churnalist.

Each year scientific and medical journals publish nearly 2 million new articles.  These articles include reports of clinical findings that change the practice of medicine and scientific discoveries that advance our understanding of biomedicine. They also report research that is flawed, misinterpreted, or of little importance.

The many and varied news outlets that comprise our health news ecosystem report on a tiny subset of these articles. The articles that are covered in the “lay press” are not a random sample but a carefully chosen, curated selection. They are the articles that news outlets know will interest readers, listeners, and viewers. These are the articles that will attract eyes, ears, clicks, and advertising dollars. This is where churnalism begins. Researchers publish all forms of investigation. Some of it is important now and some might be important in a decade or more and is intended to inform their colleagues working in their field. Some of it is flawed or unimportant, and published in poorly edited journals. Churnalists sensationalize the earliest, preliminary research and cast an uncritical spotlight on the poorly done or meaningless research that deserves only to be ignored.

Some topics routinely get more coverage than others. The news media is in love with nutrition science. This is not surprising. We all need to eat and what we eat has the  potential to affect our health. The media’s love, however, is not indiscriminate. Red wine, dark chocolate, berries, nuts, coffee are the catnip of health news. Other foodstuffs get short shrift. No one seems interested in studying sherry, bay leaves, pretzels, pumpkin seeds, or peppermint tea. What gets reported is based less on science than what is popularly believed to be healthy. More people think that coffee is healthful than think pumpkin seeds are.

Not only do a handful of topics get the most attention but the articles that do get covered, tend to be low quality, sensational, and poorly reproducible. A few years ago, we compared 75 articles that were covered by major newspapers to 75 articles that appeared at the same time in top medical journals. Articles in top journals were more likely to be randomized trials, inarguably the most reliable study design, than articles covered in the newspapers. Articles in top journals had larger sample sizes than those in the news, again a marker that the results of the studies suggested truth rather than a chance finding. Now, certainly, newspapers are not choosing articles to report on based on their poor design. What this investigation showed is that when newspapers choose what to cover they are valuing something other than the quality of the science.          .

So here are a few examples of the coverage of nutritional science. We start with nutritional science because, more than any other topic, these are the stories that frustrate us and our colleagues and most frequently mislead our friends and our patients. Because one of us takes pride in his Sicilian heritage, we will begin with meatballs in tomato sauce.

Cloudy with a chance of meatballs 

You are poking around the internet one afternoon and you come upon an article in the Atlanta Journal-Constitution (AJC) with the headline “Meatballs could ruin anti-cancer benefits of tomato sauce, study says.” How could you possibly not click on this article. This would be especially true if you just spent the afternoon making a pot of spaghetti and combining it with home-made tomato sauce, some Parmigiano, and four perfect meatballs.

The article’s message is perfectly stated in its first four sentences:

“Tomato sauce and meatballs make a tasty dish, but the combination could have an unusual effect on your health. Tomatoes contain some anti-cancer benefits thanks to a compound called lycopene. Not only does it protect against cell damage, but it also gives tomatoes their color. But the lycopene could disappear when eaten with foods that have lots of iron.”

In order to make sense of this study, we have to review a few basics. For years antioxidants have been a source of hope. The theory goes that when the body digests food, or is exposed to dangerous substances like cigarette smoke or radiation, free radicals are formed. A free radical is a highly reactive, short-lived molecule, which might play a role in heart disease and cancer. Antioxidants are substances that might protect cells against free radicals.

How does this all relate to tomato sauce and meatballs. Tomatoes are the largest dietary source of lycopene, a potent anti-oxidant. Therefore, if antioxidants prevent cancer and strokes, and if tomatoes contain lycopene, an antioxidant, then eating tomatoes should prevent cancer and strokes. From what you have read so far you are getting the idea that the benefit of tomatoes is far from certain. Free radicals might be bad for you. If they are, antioxidants might help. If they do, tomatoes might also help because they contain antioxidants.

Unfortunately, all the evidence linking lycopene and health outcomes comes from observational studies. Observational studies are not experiments. They are studies in which researchers just observe people who do or do not take a pill, adhere to a diet, or have a procedure and compare the outcomes of the two groups over time. An observational study of antioxidant and prostate cancer would compare people who consume lots of antioxidants with those who consume small amounts and compare their rates of cancer.  Most observational studies demonstrate correlation, not causation. They cannot tell you whether healthy people, as part of broader patterns of behavior, eat more tomatoes and also have less cancer and fewer strokes or whether these people have less cancer and strokes because they eat tomatoes. Of course, it is the latter that we really care about. We want to know whether we should eat tomatoes and whether doctors should be recommending them to their patients.

This is such an important point that we consider it the 1st deadly sin of churnalism: ignoring that correlation does not imply causation. When a journalist commits this sin he assumes that because tomato intake is associated with less cancer, then tomatoes must be that cause, and thus people would eat more tomatoes. To help you remember this, we will reproduce one of our favorite cartoons of all time: by the ridiculously creative and talented Randall Munroe

Ok, back to the article from AJC. In the fall of 2019, researchers found that, “lycopene could disappear when eaten with foods that have lots of iron.” That was the key finding when seven healthy male volunteers were given lycopene with an iron supplement.7 Iron supplements reduced absorption of lycopene.

So, are you seeing the connection to meatballs? Making that leap took us a moment too. The logic goes that because meatballs contain iron, eating them with lycopene-rich food (like tomato sauce) will lower the body’s absorption of lycopene and hence reduce the anti-cancer properties of the tomato-based sauce.

If that sounds like a stretch, it is. There are a litany of problems with such logic. First, whether lycopene actually lowers cancer risk is unclear. We only have observational studies on the topic. These studies tell us that higher levels of lycopene intake are associated with, but do not necessarily cause, lower risk. We do not have a single well-done randomized trial that proves more lycopene, let alone more tomato sauce, lowers cancer risk. Second, taking an iron supplement is not the same as eating foods that contain iron. This study does not provide evidence that foods containing iron impair lycopene absorption. Third, the effect of iron supplementation on the absorption of lycopene in seven men says nothing about how a larger, more diverse population would be affected by this intervention.

In this one article the author managed to commit the first and also the second of the seven deadly sins. First, the moment the finding became “eating tomatoes lowers cancer risk”, we had correlation equals causation. Second, there was extrapolation and generalization. A basic science finding, that iron decreases lycopene absorption, is extrapolated to meatballs preventing the lycopene in tomato sauce from being absorbed. The findings in seven men are generalized to an entire population.  What this study means for real people eating a plate of spaghetti and meatballs is absolutely unknown.

We think it is important to note that the reported experiment was fine to do. It is “hypothesis generating.” You can imagine a second researcher reading it and thinking, “hmm, that’s interesting, I wonder if that is true. I think I will design a study in which I measure lycopene in 100 people after eating tomato sauce and compare those levels to the levels in 100 people who ate tomato sauce with meatballs.” If that study showed a difference, then a researcher might go on to run a larger, longer, more expensive study in which she followed subjects for years to see if they had different rates of cancer or strokes. The ultimate study, one that would be worthy of coverage, would be a trial in which people were randomized to eat spaghetti with tomato sauce each week or spaghetti with tomato sauce and meatballs.

In this case, the issue is not that the research was flawed, but that the results should never have made the news. The results of the actual research are interesting to very (very) few people, so they were aggressively extrapolated and sold as something altogether different.

A spicy finding

In the fall of 2019, researchers from Italy published something astonishing, staggering, simply amazing: a research article “showing that” eating chili peppers four times a week cut the risk of dying from a heart attack by 40%. Multiple news outlets reported on the research. CNN ran the headline, “Eating chilies cuts risk of death from heart attack and stroke, study says.”

Before telling you anything about the research or the article that reported on it, it is worth putting into context just how impressive the results are. The cholesterol lowering statin medications are the most effective medications ever developed for treating cardiovascular disease. These drugs, marketed under names like Lipitor, Zocor, and Crestor, have revolutionized the care of patients with cardiovascular disease. The most impressive results we have for these drugs show that, in high-risk patients, statins decrease the risk of death from heart attack and stroke by 34%. If the chili pepper study were to be true, eating four chilies a week would be more effective than the best drugs for cardiovascular disease that have ever been developed. Moreover, this research contends that chili peppers appear not only to decrease the risk of cardiovascular diseases like heart attacks and strokes, but everything else as well. Chilies appear to be a long-lost wonder drug, hiding in plain sight, capable of improving all of human health.

Not surprisingly, when you analyze the research study, you find clues that chilies, although maybe delicious, are not a panacea. The demographics of the people included in the study who eat chilies are strikingly different from those of the people who do not. The demographic table lists 16 different characteristics and chili pepper eaters and non-chili pepper eaters differ in all of them. People who eat chilies are more likely to be men, more likely to be managers, more likely to take statins, less likely to have cancer, and more likely to eat garlic. Realize that these 16 characteristics are only the ones that were measured or reported. One could probably hypothesize that chili pepper eaters also have higher net worth and annual income. They probably also differ in the cars they drive, the restaurants they visit, the amount they drink, and how much they exercise.

In statistics speak, this is called confounding. Confounding occurs when you think there is a causative relationship between two variables, an exposure and an outcome, but the relationship is actually due to other variables lurking in the background. This happens all the time when you compare two groups that are different. You think it is the chili peppers that are saving lives, but in fact it is the dozen other things that chili-philes do that chili-phobes do not. One of our favorite examples of confounding is the relationships between matchbooks and lung cancer. Many people who habitually carry matchbooks (or cigarette lighters) get lung cancer while almost nobody who does not carry these implements gets sick. Looking at these data with no more knowledge, you could conclude that matchbooks caused lung cancer. Of course, the lurking, confounding, causative variable is the smoking of cigarettes.

Confounding

Researchers can use statistical methods to remove (adjust for) the effect of confounders, but only insofar as they measure them. The fact that chili-philes and chili-phobes are so strikingly different in the factors the researchers did measure suggests that even after adjusting for these factors there will still be many confounders remaining. This is called residual confounding.

Confounding underlies our first deadly sin of churnalism, failing to recognize that correlation does not imply causation. It is because observational studies are compromised by confounding that they cannot prove causation. We treat the failure to recognize confounding as a sin all its own, our 3rd sin, because, from a journalistic perspective, ignoring confounding often means ignoring the most interesting aspect of a study.

When you recognize that chilies cannot possibly be as effective as stated, and that the apparent benefit of chilies is therefore attributable to the enormous differences between people who eat chilies frequently and those who do not, the results of this study become interesting, but not for the reasons pitched by the CNN article or any of the other articles that covered it. Chili pepper eating is a marker of healthy people. Why is this? We do not know but we can certainly hypothesize. People do not eat chilies for two reasons, they either do not like them or they cannot eat them. Older people may prefer, or need to, eat blander diets. Neither of us considers ourselves old, but we admit we can no longer tolerate spicy food like we used to. This is a type of confounding called healthy-user bias and it certainly played a role in this study.

The iron and lycopene article was a reasonable study that should not have been reported. The chili pepper study was bad research. It was a flawed study that did not control for confounding variables. The authors of the study and the journal who published it are at fault here. The CNN journalists who covered it practiced churnalism. They took the study at face value and did not question the results. They committed not only the 3rd sin,  failure to recognize confounding, but the 4th and 5th as well.

Our 4th deadly sin of churnalism is ignoring biologic plausibility. There is no plausible explanation as to how matches cause lung cancer or how eating four chilies a week are better than our best medications. Now, they may be, but you have to admit that it seems implausible and if it seems implausible, you had better have some overwhelming evidence to support your claims.

The 5th sin is one that shows up in almost every article we cover, the disclaim and pivot maneuver. Journalists commit this sin because they really do know what they are doing. They see the flaws in their reporting. The CNN article points out that the study shows an association between chili peppers and better health but does not prove causation. But as in so much churnalism, the headline and the bulk of the article overwhelm the cautionary sentence or two — the sentence or two that should have been the message of the whole article.

If you aren’t eating organic, you might as well eat poison

Let’s cover one more nutrition article. In the fall of 2018 headlines rang out “You can cut your cancer risk by eating organic, a new study says” and “People who eat organic 25 per cent less likely to get cancer.”

We instantly exchanged emails:

Whole Food shoppers less likely to develop cancer than Target shoppers

Truffle eaters less likely to develop cancer than mushroom eaters

Americans who eat croissants less likely to develop cancer

Americans who pronounce croissant “crescent” more likely to develop cancer

The email chain reveals that the authors of this book have a lot of free time, think that they are very clever, and that it is pretty obvious that people who eat organic food are likely quite different from those who do not. Organic food eaters are probably more health conscious and have the means to afford organic foods. Once again, this seems like a classic case of confounding.

Taking a closer look at the underlying research, there are even more problems. The questionnaire the authors used to gauge how much organic food people actually ate had not been validated. In other words, we don’t know if the questionnaire actually measures how much organic food people eat. The questionnaire might just measure how often people think they eat organic food, or how likely they are to say they eat organic food when given a questionnaire. Validation is hugely important. It is the difference between asking someone how much they smoke and actually counting the cigarettes they consume. The numbers are often very different.

Next, as expected, the demographics of the people who reported eating organic food and those that did not were markedly different. People who reported eating organic had higher monthly income, more education, were more likely to be women, were less likely to be smokers, drank less alcohol, were thinner, and ate less red or processed meat. Now, of course, the researchers used statistical methods, in this case logistic regression, to adjust for many of these characteristics. This means they essentially removed the effect of the confounding variables leaving behind only the relationship between organic food and cancer risk. In practice, however, these adjustments are less powerful and more complicated than you might imagine.

First, as you read in the discussion of the chile pepper article, a researcher can only adjust for the characteristics she considers and measures. Second, you need to have enough people in your study to adequately adjust for confounders. A rough rule of thumb is that you need at least 10 people in your study for every confounder for which you adjust. Third, even if a researcher has considered and measured the confounding variables and has a lot of people in the study, she must consider if she has measured the confounders precisely enough. Consider the measurement of income: a researcher could adjust for household income, but she might only have a general sense of it. Maybe income groupings are <50k, 50-100,000, and > 100,000.  In many places in America, there is an ocean of difference between 51,000 and 99,000, and in others, a huge difference between 101,000, and 1,000,000.  Yet, in both cases, these people would be lumped in the same category.

What this study does show is that the type of people who say they eat a lot of organic food are less likely to develop cancer than those who say they do not—about that there is no room for argument. What is entirely uncertain is whether eating more organic food has anything to do with their cancer risk. However, given what the coverage of this research was, we expect that most people came away thinking that organic food prevents cancer. Some of these people probably increased the amount of money they spend on organic food. This is an opportunity cost. People are spending money on one thing, thus not spending money on something else, and, probably, not getting what they expect for their money.

That’s three good examples. Next week, we’ll present one more example of nutritional churnalism but with that one, we will shift the blame from journalists to researchers.