The report of 鈥渁 new study鈥� in the media is often followed by a comment on social media pointing out that scientists can鈥檛 seem to agree on anything. One week, meat consumption should be reduced; the next, it鈥檚 perfectly fine. Why is it, the Internet is often asked, that scientists keep changing their minds? If they can鈥檛 agree on anything, maybe they shouldn鈥檛 be trusted on anything.
The good news is that trust in scientists is still very high overall (though there are topics where a disturbing gulf emerges between scientists and the lay public). But there are reasons why scientists 补辫辫别补谤听to change their minds constantly on the important issues of our day. Let鈥檚 use the fictional example of whether or not eating bat wings will reduce your risk of developing cancer. Dr. Orlok is on the case.
1. The world is full of low-quality studies
Not all studies are created equal and the scientific literature is flooded with uninformative papers. If Dr. Orlok recruits ten people, puts them on a bat wing diet and only follows up on their cancer status for a year, he can still get published. Probably not in The New England Journal of Medicine, but given the existence of over 28,000 scientific journals, someone somewhere will publish this. How to recognize low-quality studies? They can lack a control group, they can have too few participants, their follow-up may be too short. These studies are easier to produce because they take less time to do and don鈥檛 cost too much. Sometimes, studies are given to Master鈥檚 and Ph.D. students who must graduate within a few years and must thus work on projects that are limited in time. But the conclusions that can be drawn from them are not reliable.
2. You can鈥檛 control for every variable
Dr. Orlok may follow people who eat bat wings and people who don鈥檛 for many years to see if more cancers arise in the latter group, and he may ensure that both groups have roughly the same number of smokers, as smoking is a known risk factor for cancer. But maybe he forgets to check on asbestos exposure in these groups, given that asbestos is also a risk factor for lung cancer. When you鈥檙e interested in finding out if eating bat wings specifically impacts your risk of developing cancer, you have to control for every other variable that influences cancer risk, and that can be quite a challenge. If you don鈥檛, you may think you鈥檝e found a scientific gem but better studies will prove you wrong, giving the illusion of flip-flopping on the part of scientists.
3. Scientists have freedoms that allow them to fudge rigour
Dr. Orlok doesn鈥檛 tell the scientific journal that will eventually publish his findings what he鈥檚 辫濒补苍苍颈苍驳听to do. This allows him to put his best foot forward. Maybe he keeps recruiting and testing participants until he gets the result he鈥檚 looking for and stops. Maybe he only puts the experiments that worked in his paper and leaves out the ones that didn鈥檛 or that gave ambiguous results, because they don鈥檛 tell a sexy enough story. Maybe he never publishes a particular project because the results were negative and scientific journals tend not to be interested in the publication of negative results. There are bad incentives in scientific research which can unfortunately nudge scientists away from diligence.
4. Exploratory research can be sold as solid knowledge
Maybe Dr. Orlok ran a study designed to look at the rates of colorectal cancer in bat-wing-eating individuals鈥� but when the results were in, he spontaneously decided to also look at diabetes. Studies are designed around an outcome measure (e.g. death, a particular cancer rate, pulmonary function), but it鈥檚 possible for researchers to decide, once they have their data set, to mine it in search of something else that might shine. This is known as exploratory research, and while there鈥檚 nothing inherently wrong with it, it鈥檚 easy to present the results of exploratory research as definitive knowledge. Exploratory research can help us design future studies that will specifically tackle a potential association, to confirm whether or not it鈥檚 real, but they cannot be sold as conclusive (though they often are, leading to confusion).
5. Show me the money
Dr. Orlok鈥檚 bat-wing-eating study is funded by a major bat wing corporation that wants to sell more bat wing burgers to the public. Now it would be one thing for the corporation to give Dr. Orlok the money and go away, but its board members are involved with the study design and with the calculations done on the data. It doesn鈥檛 mean that the final paper is inaccurate, but there鈥檚 every reason to be skeptical. That being said, university researchers are being asked more and more to collaborate with industry partners, making financial independence from vested interests harder and harder to achieve. The 鈥渄isclosure鈥� section of a paper should reveal whether or not the funding source had a hand in analyzing the results. If it did, don鈥檛 be surprised if the findings agree with its business interests.
6. Some scientists commit fraud
Dr. Orlok is desperate for a good publication but the cancer data he gets in the lab is not very convincing. So he cheats and alters the images coming from his microscope. A potentially insipid story becomes a game-changer and he gets published in a major journal. Scientific misconduct鈥攚hich includes plagiarism, ghostwriting, distorting findings or fabricating them from whole cloth鈥攊s rather rare but it can contribute to the apparent flip-flopping of science. One of the better known examples of it in medicine was Andrew Wakefield鈥檚 dishonest study into the measles-mumps-rubella vaccine and its false association with autism, in which Wakefield changed children鈥檚 diagnoses and the time of symptom onset to align his results with a particular narrative.
7. Valid findings get distorted by the hype machine
Maybe Dr. Orlok鈥檚 study did show that there was a slight decrease in rates of colorectal cancer among women who ate bat wings regularly. Since Dr. Orlok needs future funding for his research, he embellishes his conclusions in the paper. Then his university, which wants to show off on the international stage, further amplifies the strength and reach of his results in its press release. And finally a journalist picks up on the story and their coverage gets stuck with the headline: 鈥淓ating bat wings may open doors to curing cancer.鈥� Hype is the inevitable outcome in a culture of click baits. This means that exaggeration is not rare. Also, many studies that make the news are done in cells or in rodents, which are good stepping scientific stones but rarely applicable in humans. These preclinical studies, which often can鈥檛 be replicated in humans, often shine in the media spotlight, while the mice themselves are left on the cutting room floor.
8. Scientific controversies can be manufactured鈥�
Dr. Orlok may have the most rigorous data in the world showing that a bat-wing-rich diet will do nothing to protect you from cancer, but if 鈥淒r.鈥� Bathory, a naturopath, claims the opposite because 鈥渂at wings are natural鈥�, the public may be confused. 鈥淒r.鈥� Bathory may even have done studies and published them in theJournal of Naturopathic Bat-Wingery, but she is not a serious scientist. She is starting from a conclusion she likes and cherry-picking the evidence that reinforces her belief. It is easy to blind the public with 鈥渟cience鈥� and to make them believe there are two sides to an issue when in fact the science is clear. For example, the tobacco industry has acted like merchants of doubt, manufacturing a controversy (a 鈥渕anufactroversy鈥�, if you will) that cast doubt on reliable findings that smoking tobacco was an important risk factor for lung cancer. And that gives the appearance of scientific uncertainty.
9. 鈥� but there are genuine disagreements in science
When we remove all of the above, we are left with genuine disagreements in science that exist because science is hard and the universe is complex. Sometimes scientists find a weak effect and disagree over whether or not we should take it seriously. Other times, too few studies have been done to know for sure. Also, the tools we have to probe interesting questions about the universe around us may not be good enough to answer them properly, leading to debates and disagreements.
Research into what we should and shouldn鈥檛 eat certainly ticks all of the above boxes. Many studies are of poor quality, while others get selectively reported for maximum sexiness. Exploratory research is sold as near-divine revelations and hype predominates in the ensuing media cycle. And multiple interest groups get to distort the science and create manufactroversies.
In the end, science is a human enterprise that aspires to rigour. It gives us a set of tools we have been mastering and optimizing over many centuries to reduce our uncertainties about the world we live in and to make better and better predictions. The nauseating see-saw movement of science headlines may try our patience, but the answer is not to lose trust. It is to demand diligence and accept that its price will be nuance.
Take-home message:
- The idea that scientists keep changing their mind can sometimes be due to actual disagreements in a field of study
- Often though, it is the result of poorly done or selectively reported studies, of hype generated in the reporting of results, and sometimes of interest groups creating fake controversies to sow doubt
Want to comment on this article?
