Science, or the scientific method, put most simply is just a process to figure something out. The study design, participants and many other elements are essential in making sure the study is valid. However, even when the study itself is well done, it can be reported on or described poorly. For example, a journalist, blogger, organization or individual may cherry pick one or two statistics or points from a study and claim it offers proof of something or it means something it really doesn’t. This could be intentional or accidental or even a result of their own bias (more on that here). Another problem is that studies are often written for the scientific community. They aren't typically written for the general public. This can sometimes make understanding them difficult, even for the most experienced and ethical health journalist or writer.
We talked about this a little in the red flag section (click here for that info), but very few studies actually prove something. Even when a study is published with new information or information that is at odds with existing medical knowledge, researchers want to see their peers come to the same conclusions before any recommendations are made to the public. That’s because proof lies in replication. For example, let’s say one group of scientists discovered X when they did Y. Collectively, the scientific community might say, “Great, that’s interesting, but that won’t mean much to us unless many others get X when they do Y.” So they race to see if they can get the same results. And, when enough researchers get the same results over and over, you start to develop a body of evidence and that’s the basis for most medical recommendations.
Studies are a great way to advance medicine, but one study alone should not be taken as a recommendation for you to make a major change to your health. When in doubt, talk to your health care provider.
Here are some tips for checking medical studies (Limitations: The following tips will not make you an expert at evaluating studies. And the following are just red flags, one or two of the items below do not necessarily mean a study is invalid. They should only make you seek more information):
- Specifics
A good study will contain specifics such as the names of researchers, hospitals, previous studies and specific dates. If terms like “a leading doctor,” “recently” or “a major hospital” are peppered throughout the study, credibility is questionable. - How powerful is the study?
How many people were in the study? Five or five hundred? Reliability comes in large numbers. - People or animals?
Did they study humans or animals? Studies with animals can sometimes be presented in the media or online as if you can directly translate the findings into actions you should take for your own personal health when that’s just not the case. More studies need to be done and if reasonable and ethical, with humans. - Methodology
How did they find their participants? Calling up your friends to participate in a study isn’t exactly the best way to get accurate results. A randomized sample is best. - Transparency
Does it list limitations? No data set is perfect. It’s not only important that a study list its limitations, it’s ethically required. - Conflict of Interest – Part 1
Do the authors list conflicts of interest? This could be financial or other ties to an organization, corporation, person or special interest. Conflicts of interest can corrupt the motivation or decision-making of the authors. Listing them builds trust and credibility. - Conflict of Interest – Part 2
Who funded the study? For-profits funding research is quite common. It doesn’t mean the research is tainted, but it is something to consider. - Language
Does the story or article describing the study use language like “miracle,” “breakthrough” or “game-changer”? If so, be wary. Look for more information. - Exaggeration
Are the results overstated? This can happen when the study is written about by others not involved in the study, such as the media, a website or individual. Whether that misrepresentation is intentional or just accidental is another issue altogether. Sometimes it’s best to read the study, not just a story about the study. - Simplification
On the flip side, does it offer a very simple explanation or an overly simple solution to a complex problem? Be wary; health is complicated and oversimplification should be a red flag. - Science or sales?
Is it just trying to sell something? If it sounds like an advertisement, it probably is. - Checked by peers
Is it peer-reviewed? Anonymous peers reviewing an article or study is a great way for researchers to check their work. This alone shouldn’t be enough to get your stamp of approval, because the phrase “peer-reviewed” is thrown around a lot and not always accurately, but it’s a good start. When in doubt, reach out to your local librarian. - Source
Where was it published? If it has merit, why was it in a journal or website you or your health care provider have never heard of? Not sure if a journal or website is a good source? Click here to ask your local reference librarian.
Bottom line:
Very rarely will one study prove anything. And, when scientists do find a relationship (correlation or association) between two things that does not mean one thing caused the other. It's also important to note that one study is not the same as a medical recommendation. Very rarely should one study influence a major decision you make about your health. Broad medical recommendations are based on a body of evidence (lots of studies, practice, years of gathering information, etc.). So, long story long, don’t put too much weight in that one new, “miraculous” finding, especially if it goes against all existing medical recommendations. And, when in doubt, talk to a medical provider.
Click here to leave a comment or concern.