There are so many different places to get information on health, nutrition, child development, and parenting: the news, magazines, websites, blogs, your pediatrician your mother-in-law. It can be overwhelming to sort out the facts and decided what to do for your children. Good sources at least tell you where they got their information (who is Dr. KnowItAll anyhow?); better sources cite specific studies (are they quality studies?); the best sources are peer-reviewed scientific articles. Unfortunately, these articles can be hard to access, and even harder to understand. It is my hope that this guide will introduce you to the world of research and give you some basic tools so you can go straight to the source, finding the best information for your family.
So why bother reading the study instead of the news article? Newspapers are trying to get your attention, so they often take parts of a study and write a catchy headline that can be misleading. Here’s an example: “ A cheese sandwich is all you need for strong decision making”. The actual articles is “Serotonin modulates behavioral reactions to unfairness” and the researchers didn’t even mention cheese. What they did do was give subjects a drink that affected their serotonin levels indirectly through tryptophan, then presented them with a situation where they were treated unfairly and recorded the tendency to take revenge. It so happens that cheese and chocolate contain tryptophan too, but it’s a big jump to go from an artificial tryptophan cocktail to a cheese sandwich, and from a controlled test on unfairness and revenge to “strong decision making”, whatever that means. While it’s not the easiest way to get information, reading the study itself is the most reliable way. (http://www.ted.com/talks/molly_crockett_beware_neuro_bunk.html?quote=1993. http://www.sciencemag.org/content/320/5884/1739.short)
I would like to present you with some tools to help make accessing the studies easier. First, we will talk about places to find studies to read. Second, we will go through some different types of studies you might find. Next, we will go through some of the roadblocks to understanding you may encounter. Finally, we will take everything we have learned and evaluate studies to make decisions applicable to our lives.
The easiest tool for the home investigator like yourself is probably Google Scholar. It is a free search engine that contains scholarly articles, many of which are free to read. Others may only give an abstract or summary, so you can get the gist without any of the details. If you want more details you can purchase the article, but that is quite expensive. Many college and university libraries already subscribe to the most reputable science journals and may offer free services to the public, so check their periodicals section first. I have even heard of large public libraries offering this service as well.
So now that you have started looking for papers, let’s talk about some different types of studies you will find. When we are talking about medical studies, such as you will find when you look for information on disease and nutrition, you will find three basic types: in vitro, animal, and clinical. In vitro tests take tissue from people or animals, and test them in a lab to see how they react to different treatments or exposures. Animal tests, you guessed it, are done on animals, and are usually the result of a promising in vitro test done earlier. The advantage to animal tests is that the effect the researchers are testing can be seen in a whole organism not just small groups of isolated cells as in an in vitro test. If the animal tests go well, scientists move on to clinical tests where volunteers are subjected to certain treatment and the results are reported. Clinical tests are the most applicable to our lives, and there is danger in jumping to conclusions from in vitro or animal tests although they are very important steps in the process. Clinical tests may be blind tests, where the subjects don’t know what treatment they are receiving, or double blind, where neither the subjects nor the administrators of the test know.
Of course, not all studies are strictly biological. Studies of behavior usually fall into three categories: cross-sectional, retrospective, and prospective. Cross-sectional studies are surveys about present factors and conditions that try to find relationships. (For example, how tall are you, and do you have blonde hair? might show that blondes are taller than brunettes) These are fairly easy studies to conduct and a great starting point for research, but it only shows correlations, not causes. A good quality cross-sectional study can still make really good guesses about causes though by doing some fancy statistics to weed out already known causes. Retrospective studies are a better look into causation because they ask people about past behavior and compare it to their current condition. One problem with these is that they rely on people’s memory and honesty. The best, and most time consuming and expensive, studies are prospective because they take a group of people and track them into the future.
Once you have an idea what kind of study you are looking at you start reading, and maybe it is getting complicated and overwhelming fast. You do need to have some background knowledge as you attack the literature. Check out my list of recommended books for basic information on subjects in which you are particularly interested. For more specific questions, I have found Wikipedia to be a great resource. While anyone can write a Wikipedia article, the more common articles are quite accurate thanks to that very fact. Mistakes get taken out so quickly because anyone can correct them.
The next challenge is probably all the statistics. I want all of you math-phobes to take a big deep breath; it isn’t going to be too bad. Here’s what we are covering today: confidence intervals, distributions, and significant results. There is off course a lot more to learn about statistics, but those three things should get you started.
First: confidence intervals. We can never include 100% of the population in a study, so we take a sample, maybe 300 people. We then try to take what we learned about those 300 people and generalize it to the population in question. A confidence interval tells you how sure we are that our results (the sample) match reality (the population) within a certain range. In the literature it will say something like, +/- 3 grams, 95% confidence interval, meaning that we are 95% sure that we have the “right answer” within 3 grams. The thing to look for with confidence intervals is to look at the extremes. If the bottom of the interval is in the negatives, you might question the results. For example, they may say eating eating a certain food has a positive average effect within a certain interval, but the bottom of that interval is actually a negative effect, so there is a chance that this food is actually bad for you! That’s why we never just look at an average.
Averages are also misleading when there is a wide distribution of results, meaning that a variable had much more affect on some people than others. You may be familiar with “the bell curve” or a normal distribution. This is where the vast majority of results are right around the average, and the further away from the average, the fewer data points there are. If we graphed it with results along the bottom, and number of people who got those results along the side, it looks like a bell. When studies look like this, they are easy to interpret, but that’s not always how things fall. Always pay attention to the distribution curve when it is provided in a study. If the graph has a long tail to the right, than the average will be pulled to the right, and most people/results will be less than “average”, and visa-versa for a long tail to the left. You may also see two humps, meaning that there were two common results. In that case, the average is probably between the two most common results, even though that is a low point on the graph: perhaps there isn’t a single data point at the average!
Another problem with not being about to study a whole population is that when we do find that a variable has an affect on the results we must make sure that we didn’t just get lucky. For example, say we are trying to see if 3-spotted ladybugs like water more than 4-spotted ladybugs so we go to the pond and the forest and collect five lady bugs at each site. We count the spots and find that all the ladybugs at the pond have 3 spots, and only three of the five from the forest had 3 spots. We must ask: what if we just happened to pick up the only 5 three-spotted ladybugs at the pond, and the other 50 bugs we didn’t catch all have 4-spots, and we just happened to find fewer three-spotted ladybugs in the forest even though they are much more common there? If that is the case, then our study would draw the wrong conclusion. This is what they are taking about when they say there is a “significant difference”. It doesn’t mean the difference is large, simply that considering the sample size compared to the population, and considering the difference they did find, they are reasonably sure that it is not just luck, there is a really correlation here. There is a mathematical formula to decide if a result qualifies as “significant”, but it can be a little misleading because the larger the sample, the smaller the difference needs to be before it is “significant” so check that before you base major life changes on the study. If the study included a huge number of people, the “significant” result could be tiny and perhaps not worth turning your life upside down. (Statistics for Engineers and Scientists by William Navidi)
Okay, so we have gone through how to get your hands on some scholarly papers, and how to make sense of them. Now it’s time to decide what we think. Evaluating the quality of a study is mostly about common sense. Look for anything in the methodology that might introduce bias, such as it not being a double blind study, or perhaps the sample is not representative of the population. Do a search of other papers that reference this one. Have the results been replicated? criticized? or disproven elsewhere? In the end, it is up to you to decide how this research affects you and your family.