In today’s world of conflicting interests, flawed science, and sensationalized media, it’s important to question new claims and findings, especially when those findings could have serious implications for your health. One of the most important things you can do to make sure you’re getting the real scoop (aside from reading my articles, of course!) is to read the scientific literature yourself.
I know many of my readers are already adept at perusing research with a critical eye, but for those of you delving into the recesses of PubMed for the first time, it can be a little overwhelming. In this article, I’ll cover the basics of epidemiological research, practical tips for navigating the scientific literature, and things to watch out for when reading studies.
Do you know the most important points to consider when reading scientific research? Check out this article to find out.
Types of Scientific Research
Let’s start by going over the different types of studies and papers you’ll find in scientific journals. At the broadest level, we can categorize research into either review papers or original research. These can also be classified as secondary or primary research, respectively.
Review papers are similar to the articles you’ll find here on this blog. They draw on prior research that has been published on a specific topic, and use that research to form their own broader conclusions. Reviews will usually give you some background on a topic, which is helpful, and they’re generally much easier to read and understand than original research.
Original research comprises anything that is based on the researcher’s original findings, whether through experiments in a lab or population-wide data collection. This graphic illustrates some types of studies you’ll encounter.
Experimental studies are the only type of study that can potentially establish causation, because the researchers actually manipulate variables instead of just observing a natural sequence of events. Randomized controlled trials are a type of experimental study, and are often called the ‘gold standard’ of scientific research.
Observational studies are the other main type of original scientific research, and they’re often criticized for inferring causation from correlation when they have no business doing so. I’ll now describe some of the main types of observational studies, using cancer as an example.
Cohort studies look at a group of people with similar characteristics (a ‘cohort’) who are free from the outcome being studied (cancer, in this case). The people in the cohort are measured for different risk factors at baseline, and then followed over a period of time to see which people develop cancer. Then researchers can see if having a certain risk factor at baseline increased people’s chances of developing cancer in the future.
Cohort studies can be further broken down into prospective and retrospective studies, based on when the study was conducted in relation to when the data was collected. The study described above would be a prospective study. A retrospective cohort study would use data from the past to define a cohort and measure baseline risk factors, and then look at present-day data to see which people in the cohort already developed cancer.
A case-control study on cancer would divide participants into two groups based on whether they have cancer (the ‘cases’) or not (the ‘controls’). Researchers would then look back in time using existing data to see if they could pinpoint any factors that could potentially explain the different outcomes.
Cross-sectional studies observe a population at a single moment in time, measuring both the occurrence of the disease in question and various risk factors. For example, researchers might look to see if populations who live closer to polluted urban areas have a higher rates of cancer.
Like what you’re reading? Get my free newsletter, recipes, eBooks, product recommendations, and more!
Navigating the Scientific Literature
Now that you know what to expect, the next step is actually finding the papers you want to read! I probably don’t have to tell you that if you’re looking for peer-reviewed articles, a Google or Yahoo search just won’t cut it. However, Google does have a search engine called Google Scholar that is helpful for finding studies and papers. It pulls results from online journals and databases around the web, and is pretty good about only providing papers from trustworthy sources. PubMed is also a great place to look for trustworthy articles.
You might run into difficulty accessing the full text for papers you find. Sometimes, Google Scholar or PubMed will link to free full-text versions, but often you’ll be asked to pay for the full text, which can be pricy. A good way to access the full text is though a university library, because they usually have subscriptions to online journals and academic databases. If you’re a college or graduate student or otherwise have access to a university library, take advantage of the wealth of resources at your disposal! Some medical libraries on college campuses are also accessible to the general public.
After you find one study or paper dealing with your topic of interest, I’d recommend looking at the ‘related citations’ section to find other relevant articles. The ‘cited by’ section is also quite helpful, because it shows articles that cite the paper you’re looking at. That way, you can see how other researchers have interpreted it, and how the study is being used to support various claims. These features are especially valuable if you’re having trouble finding the results you want using keywords.
When researching a topic, it’s usually a good idea to begin with review articles. The authors have already done the hard work of rounding up relevant studies and drawing conclusions from them, and reviews are a great starting point for looking deeper into the original research. If you don’t want to rely on the authors to draw conclusions for you, however, you can look at the reference lists and read the individual studies for yourself!
Reading the Scientific Literature
Here are some questions to ask and things to watch out for when reading original research:
- Was the study short-term or long-term? Many interventions are only effective in the short run, and have completely different effects over the long term. Fish oil is a good example of this.
- Was the study done in vitro or in vivo? In humans or animals? Clearly, an in vitro study (one done in a petri dish or test tube) isn’t necessarily generalizable to living beings, and studies done in animals don’t necessarily apply to humans.
- Was the methodology strong? If a trial isn’t randomized, double blind, or placebo-controlled, its results might not be as dependable. In observational studies, it’s vital to ensure that different variables are controlled for. You should also watch out for general cluelessness and dishonesty.
- Did they use surrogate markers or end points? Surrogate markers, such as cholesterol numbers and other lab test values, are often used to measure the effectiveness of a treatment. However, changes in surrogate markers don’t necessarily translate to changes in overall disease outcomes. If a study uses values from a blood or urine test to measure success rather than using disease progression or mortality, it’s good to be wary.
- Absolute risk or relative risk? Sometimes, scientists like to use relative risk to make their results sound more impressive. If a treatment reduces the risk of a disease from 2% to 1%, the absolute risk reduction is 1%. Treatment or no treatment, your absolute risk of getting the disease is pretty small. However, you could also truthfully say that the treatment reduces the risk by 50%. This sounds more impressive, but it gives a skewed impression of how valuable the treatment actually is.
Here are some things to watch out for when reading review articles:
- Do the citations support the claim? When a paper cites sources for a claim it makes, scroll down to the reference list and look up some of the references. You might be surprised by how often citations are misused. The results may have been taken out of context, or generalized to populations where the results don’t apply. Occasionally, researchers will completely misinterpret a study!
- Do the authors infer causation from correlation? Make sure that when a review paper claims that ‘x’ causes ‘y,’ they’re not citing an observational study. (Some good examples of this can be found in my special report on red meat.)
- How do the conclusions compare with other reviews? Different review articles on the same topic can come to very different conclusions. Often, their reference lists will look remarkably similar! It’s sobering how frequently researchers can look at the same evidence but come to completely different conclusions. In these situations, you might have to delve deeper into the original research.
- Who funded the review? It’s always a good idea to see who funded a study or a review, and what the authors’ affiliations are. If a paper comes off as biased to begin with, a questionable source of funding could further tarnish its credibility.
Those are some of my best tips for being a critical consumer of the scientific literature. Reading research for yourself can be both empowering and sobering, and you’ll likely discover that most things aren’t as clear-cut as we may want them to be. Just be aware that there is such a thing as information overload, and be careful not to fall so far down the research “rabbit hole” that you stress yourself out!
Better supplementation. Fewer supplements.
Close the nutrient gap to feel and perform your best.
A daily stack of supplements designed to meet your most critical needs.
Thanks the detailed response, Prof. Feinman. It’s very helpful!
Error in above. Second sentence should be: “If 100 people who drink coke all the time are studied….”
I’m sure Chris will give more complete answer but odds ratio is what it sounds like. If 100 patients who never drink coke are studied and 5 of them have metabolic syndrome. The odds for coke-drinkers to have MetS is 5/100 = 1/20. The control group in the study, never drink coke and they have only 3 people who fit the definition of MetS, odds = 3/100. Odds ratio = (5/100)/(3/100) = 1.67 Obviously an OR of 1 means that there is no difference between coke and no-coke. Now, to know how reliable the value of 1.67 is you have to keep doing the experiment and see if the OR changes. If the OR averages out to 1.67 that is the mean that you report and the 95 % CI (confidence interval) tells you how likely the mean is to fall in that interval, that is, is the mean reliable. The quick rule is that if the 95%CI crosses 1, e.g. 95 % CI = 0.80 – 1.87 means that the value is not statistically significant. The problem is the same as the problem with relative risk. Statistically significant is not necessarily practically meaningful. For example, suppose you had to examine 10,000 coke-drinkers before you found 5 people with MetS and among 10,000 there are only 3 people. What’s the odds ratio, now? Get it? If the paper reports an odds ratio, does that help?
Thanks for posting this, Chris. I find the most difficult part of reading and understanding medical papers is knowing what the statistical data actually mean in real terms. What does “odds ratio = 1.3, 95% confidence interval = 1.0–1.7” mean? Or “Hazards ratio = 1.35, 95% CI = 0.96 to 1.88”? And when it comes to statistical significance, what is the difference between P < 0.001 and P < 0.05? I understand the difference between relative and absolute risk, but what does a relative risk of 1.4 (95% confidence interval 1.0-2.0) mean in lay terms? Please help the statistically-challenged among us (we are many!) get a handle on these common statistical terms!
I second threonate. The whole USDA Guideilnes are designed to fix something that ain’t broke, as we say in Brooklyn. It is specfically charged by congress to make recommendations to people who are healthy. Is the association between their guidelines and the obesity epidemic causal? Hill’s criteria for causality certainly make them a contender. As for RCT, they have never recovered (intellectually — people still consider them gold) — they have never recovered from Smith and Pell (reference when I am back at the computer).
Chris, not to say that this article is not useful, but you’re missing a crucial point. Allow me to illustrate, by quoting a section of this article:
“You might run into difficulty accessing the full text for papers you find. Sometimes, Google Scholar or PubMed will link to free full-text versions, but often you’ll be asked to pay for the full text, which can be pricy. A good way to access the full text is though a university library, because they usually have subscriptions to online journals and academic databases. If you’re a college or graduate student or otherwise have access to a university library, take advantage of the wealth of resources at your disposal! Some medical libraries on college campuses are also accessible to the general public.
After you find one study or paper dealing with your topic of interest, I’d recommend looking at the ‘related citations’ section to find other relevant articles. The ‘cited by’ section is also quite helpful, because it shows articles that cite the paper you’re looking at. That way, you can see how other researchers have interpreted it, and how the study is being used to support various claims. These features are especially valuable if you’re having trouble finding the results you want using keywords.”
So, I should eat coconut oil regularly, then?
(If my point wasn’t illustrated, then I will explain it below here.)
At what point does someone stop diving into so much research, analysis, and detective work just to figure out whether or not he should eat, avoid, or minimize a certain food in his diet? I eat coconut oil because I get good results from it, I like the taste, it’s somewhat affordable, and there’s solid science behind how it benefits a person (MCTs, almost fully saturated, immune support, how it’s metabolized, et al.). I did not decide to eat coconut oil regularly only after reading scholarly articles for months, checking the citations, uncovering conflicts of interest and sources of funding, comparing different studies, and reading several books about it.
See what I mean? I think we give way too much weight to “scientific studies” and the gold standard of randomized controlled trials. We’ve lost what it means to be human by reducing everything into an isolated variable, which supposedly can apply to the “human population as a whole.”
“Just be aware that there is such a thing as information overload, and be careful not to fall so far down the research “rabbit hole” that you stress yourself out!”
Heh…
Hi Chris,
Wish I had this article when I started writing about whole health topics 9 years ago. I’d love to read more about this topic, as boring as it might be for others…
Thanks again,
Wayne
Thank you for this! I’ve been wondering the best way to find research and this article helps a lot!
This is very informative, Chris. I usually visit PubMed when I’m doing a research. BTW, it’s my first time to hear about Google scholar. I would like to try it. Thanks for the advice. 🙂
I commented earlier but wanted to add one thing. Studies (scientific or not!) are motivated by many factors. That includes excellent science,ego, greed, the desire to help, personal or family illness, your status at a university etc. My point is that results are sometimes accidentally or maliciously aligned to lead the investigator to the conclusion (hypothesis) they initially hoped to answer. It is rare to read in any study that “the study completely failed to substantiate our hypothesis!” It is usually only after additional duplicated studies of larger and more complex population groups can definitive results be evaluated. Look at the recent FDA results which have shown that sometimes only after enough patients have used a drug or additive for a long enough time can the full benefit or danger of a new compound or herbal plant based derivative be known. (ie Plavix (gene based side effects), etc…….
The one thing i try to tell people is that when a new therapy or drug comes out it might be better to wait 6-12 months to try it until a large number have people have been on the med/herb/supplement. The issue is some patients are suffering so much they will try anything for relief. My mantra is to make sure their diagnosis is correct and “to take the blinders off to make sure we haven’t missed something!” I have a list of cases where that was exactly what was the problem. Previous physicians and allied health care providers did not stop, look and listen, to the patient long enough!!!!! (celiac disease, bipolar disorder, food allergy, etc etc.)Then i remember the Oath i took after medical school “Above All Do No Harm!” thank you for reading……………………..
If you are a college student you can also gain access to articles that are published in journals to which your school does not subscribe; all you have to do is use the interlibrary loan service.
You can also directly search Pubmed central, which is exclusively full-access. Pubmed central (you can even search for journals by category…if you want to browse through a journal instead of searching for a particular topic) and PLOS are replete with free open-access peer-reviewed research articles and reviews.
I have even been able to get books from fairly recent symposia via interlibrary loan at my local library.
Another timely article that figures don’t lie but some are very good at using them to muddy the waters. Industrial agriculture and grain/soy GMO VESTED INTERESTS AND SO CALLED GOV WATCHDOGS confuse the public with cries of evidence based sound science yet appear blind or oblivious to the proof of the pudding being in the eating thereof.
Anecdotal evidence and trial and error over many centuries trumps all modern industry/gov spin. Cumulative or weight of pub med evidence for example linking glyphosphate or roundup to all manner of inflammatory disease is dismissed yet the evidence, considered as a whole is irrefutable and beyond any reasonable doubt. Unfortunately the control of many by few will continue until the escalation of disease and the connection becomes self evident as the truth will out. The evidence against some big Ag practices is now far greater than the accepted anti smoking evidence yet already legislation is being enabled to prevent culpability.
Hopefully the informed minority will eventually create enough public will to divert from the path of self destruction we appear to be on and your article can only help.
Extremely useful article although I offer a different take on things in my last blog post at http://wp.me/p16vK0-gz. My main point is that, although your chart is an excellent guide to how studies may be described, in terms of scientific value, the definitions are not so clear cut. An observational study will support causation if it is good (I listed some criteria) and a well-controlled RCT may tell you nothing if it is poorly done. The appropriate type of study is one that answers the question that you want answered. As an example: You assign 20 people to a low-carb diet, they have good compliance and all lose a moderate weight loss. When they report to you, however, 5 of the participants say that, to their surprise the irritable bowel syndrome that they had all their life, suddenly disappeared (none of the controls, who lost slightly less weight say that). Now, which is the stronger results, which has more impact, the weight loss or the, essentially anecdotal report on IBS?
Hi Richard,
I agree on all points. I was trying to keep this relatively straightforward and short; there’s obviously a lot more to consider than I was able to cover in this article. It would have been interesting to discuss the Hill criteria and GRADE approach for determining causation from observational research, along with other issues such as those you raised. Perhaps in a future article!
Fire away with a big long article on research methods. I for one will enjoy it at least.
Chris,
I think we’re in basic agreement and the problem is that science and particularly statistics is a human activity. As I mentioned off-list, I think this is a good start on what I tentatively call Visualizing Imperial Neo-Acouterments.
Chris,
I AM AN MD WHO HAS BEEN FOLLOWING YOU AND REFER PATIENTS AND FAMILY TO YOUR SITE AS WELL. JUST WANT TO SAY KUDOS FOR WRITING AN ARTICLE LIKE THIS. THERE ARE SO MANY PATIENTS WITH DIFFICULT PROBLEMS TO PIN DOWN AND DIAGNOSE THEY ARE AT THE MERCY OF THE MEDIA ADVERTISERS ETC. ALL CONSUMING MANY OF THEIR ASSETS , OFTEN WITH MINIMAL BENEFIT TO THEM!!! THANK YOU FOR YOUR CONSISTENTLY HELPFUL AND HONEST ADVISE TO PATIENTS.
MARK HOLSCHER MD KS/AR
Great article, Chris. I think its always a good idea to go back out to your search after you have read through the article and use some terms that attempt to find the opposite – in other words, are there papers that refute the authors’ claims? You can find some interesting surprises when you try to disprove the study’s findings (or even your own beliefs).
Thanks for this informative summary, Chris. One good example of ‘bad’ interpretations is the Wax meta-analysis on homebirth. Out of the gate they mix ‘apples’ and ‘oranges’ by including planned homebirth with a professional midwife with accidental homebirth and pre-term (which is not planned with professional attendant either). There’s a helpful critique at http://www.scienceandsensibility.org/?tag=is-home-birth-safe
Funding is one of the most important aspect to understand when evaluating research. There are often ghost writers for papers in which the sponsor or funder will write the article, do the research, and then turn around and ask a big name and well known primary investigator (PI) to put his/her name at the top of the paper.
Another aspect, especially in natural medicine research is whether they used the correct form of a treatment. An example is using dandelion root in a study comparing conventional diuretics to dandelion. Dandelion root is not used as a diuretic, the leaf is used instead.
In holistic medicine research, looking at one part does not mean the outcomes or results can or can not be applied to the whole. An example is Chinese medicine herbal formulas. Each particular ingredient has particular chemical compounds and constituents that can be extracted and studied on their own. Each ingredient, though, is prepared in a certain manner (e.g. – boiling aconite root for hours), and then it is added to many other ingredients to make for a whole new set of chemical constituents that would have never existed if that ingredient was on it’s own.
Thanks for this Chris. I have a degree in Psychology and spent a lot of time reading through research papers during my course and beyond. I don’t read them as much anymore but will if something really stands out to me!
I would argue looking at the funding or who wrote it first, is more likely to make you the reader more biased… First look at the study design, strength/limitations… Look at the author/funding nearly last.
I agree with Mark. If the author says that the sponsor did not influence the research, you must take them at their word. Otherwise, you are implying misconduct which is very serious. In fact, it is the more narrowly focused foundations are the most hands-off (because they are sensitive to charge of bias). We would be worse off without the Atkins Foundation. Of course, all agencies have their own interests but once funded, you have to assume they don’t interfere unless it’s clear what the mission is. In that, Chris is right, you have to know what kind of agency (If funded by Kellogg’s company I would see whether there is a disclaimer. If funded by the Kellogg’s Foundation, you pretty much assume that it’s real).
I don’t necassarily disagree with either of you. You just have to use a bit of common sense when looking at the funding, researcher bias etc. The most important part of any research is obviously using fully legit research methods.
But if the data is then being used in a way that supports a political bias, for example, or for purposes outside of what the data shows then you have ask why?
Data does not exist on its own merit, but on how it’s used.
Speaking as a reviewer or general context of within scientific community, you can accuse people of being insensitive to how their results might be used (or general stupidity) but if you accuse them of malevolence or falsification in any way you are supposed to have strong evidence.
I agree. Funding sources are listed for a reason, but it isn’t to show the research is biased. I feel it is just the opposite actually. If they were being biased, they would probably try to hide the funding source somehow. The reality is, no funding source is ever going to be completely neutral in the matter. Coming from a Canadian perspective, it is becoming increasingly difficult to lock down public/government funding. Most of the funding for research is now coming from corporations or other organizations. You either accept this funding or you say goodbye to your research program. What is more important is the integrity of the researcher and the quality of their work. I am sure there are some instances where the researcher was “bought” to publish the work but I think this is more of a rarity when you are looking at higher impact journals.
You are all right in what you say.
My point was simply that research data and how it is used can be biased by the researchers themselves and/or the organisations behind it. The pharmecutical industry is a clear example of this and hides research data that does not support their views but are twice as likely to make public research data that supports the drugs they are trying to market. It’s also important to note that the research methods they use can be pretty sound.
Whereas someone like Chris Master John has affiliations with the Western A Price Foundation but not for a minute would I say that this bias’ his own research. He’s a very clever guy who has a sound scientific philosophy.
It’s just something to consider along with a bit of common sense.
The first thing I do is to look at who’s written it. I then do a little research on them to see what they do, any links to other organisations, any political beliefs etc. to see if this could have biased the results and the conclusions in any way.
I agree with you, Jamie. It is better to find out details about the author and the reason behind the scientific research. It is really crucial to know who funded the research.