1.4: How Not to Do Statistics

Learning Objectives

• Identify common pitfalls in statistical reasoning.
• Recognize the risks of overgeneralizing results and misinterpreting sampling errors.
• Understand the importance of avoiding incorrect assumptions about cause and effect.
• Promote accurate and reliable interpretation of statistical conclusions.

Many studies are conducted, and conclusions are made. However, there are occasions where the study is not conducted correctly or the conclusion is not correctly made based on the data. There are many things that you should question when you read a study. There are many reasons for the study to have a bias. Bias is where a study may have a certain slant or preference for a certain result. The following is a list of some of the questions or issues you should consider to help decide if there is bias in a study.

One of the first issues you should ask is who funded the study. If the entity that sponsored the study stands to gain either profits or notoriety from the results, then you should question the results. It doesn’t mean the results are wrong, but you should scrutinize them to ensure they are sound. For example, suppose a study says that genetically modified foods are safe, and the study was funded by a company that sells genetically modified food, then one may question the validity of the study. Since the company funds the study and its profits rely on people buying their food, there may be a bias.

An experiment could have lurking or confounding variables when you cannot rule out the possibility that the observed effect is due to some other variable rather than the factor being studied. For example, suppose you give fertilizer to some plants and no fertilizer to others, but the plants are not growing taller due to a lack of sunlight and not due to a lack of fertilizer. You won’t know if the plants that received the fertilizer grew taller because of the fertilizer or the sunlight. Make sure you design experiments to eliminate the effects of confounding variables by controlling all the factors.

Overgeneralization

Overgeneralization is a study of one group but makes conclusions for all groups. An example is doing cancer treatments on rats. Just because the treatment works on rats does not mean it will work on humans. Another example is that until recently, most FDA medication testing had been done on white males of a particular age. There is no way to know how the medication affects other genders, ethnic groups, age groups, and races. The new FDA guidelines stress individuals from different groups.

Cause and Effect

Cause and effect is where people assume that one variable causes the other when the variables are only correlated. Unless the study was done as an experiment where a variable was controlled, you cannot say that one variable caused the other. Most likely, there is another variable that caused both. As an example, there is a relationship between the number of drownings at the beach and ice cream sales. This does not mean that purchasing ice cream increases the chance people will drown. Most likely, the cause for both increases is the heat.

Sampling Error

This is the difference between the sample results and the true population results. This is unavoidable and results in the fact that samples are different from each other. For example, if you take a sample of 5 people’s heights in your class, you will get 5 numbers. If you take another sample of 5 people’s heights in your class, you will likely get 5 different numbers.

Nonsampling Error

This is where the sample is collected poorly, either through a biased sample or an error in measurements. Care should be taken to avoid this error.

Lastly, there should be care taken in considering the difference between statistical significance versus practical significance. This is a major issue in statistics. Something could be statistically significant, which means that a statistical test shows there is evidence to support what you are trying to prove. However, in practice, it doesn’t mean much, or there are other issues to consider. For example, suppose you find a new drug for high blood pressure that reduces blood pressure. When you look at the improvement, it doesn’t amount to a large difference. Even though statistically there is a change, it may not be worth marketing the product because it isn’t that big of a change. Another consideration is that the blood pressure medication improves a person’s blood pressure, but it has serious side effects, or it costs a great deal for a prescription. In this case, it wouldn't be practical to use it. In both cases, the study is shown to be statistically significant, but practically, you don’t want to use the medication. The main thing to remember in a statistical study is that the statistics are only part of the process. You also want to make sure that there is practical significance, too.

Surveys

Surveys have their own areas of bias that can occur. A few of the issues with surveys are in the wording of the questions, the ordering of the questions, the manner the survey is conducted, and the response rate of the survey.

The wording of the questions can cause hidden bias, which is where the questions are asked in a way that makes a person respond a certain way. An example is that a poll was done where people were asked if they believed that there should be an amendment to the Constitution protecting a woman’s right to choose. About 60% of all people questioned said yes. Another poll was done where people were asked if they believed that there should be an amendment to the Constitution protecting the life of an unborn child. About 60% of all people questioned said yes. These two questions deal with the same issue, though giving opposite results, but how the question was asked affected the outcome.

The ordering of the questions can also cause hidden bias. An example of hidden bias is if you were asked if there should be a fine for texting while driving, but the preceding question asked if you ever texted while driving. By asking a person if they partake in the activity, that person now personalizes the question and may affect how they answer the next question.

Non-response

Non-response is when you send out a survey, but not everyone responds. You can calculate the response rate by dividing the number of responses by the number of surveys sent. Most response rates are around 30-50%. A response rate that is less than 30% is very poo,r and the results of the survey are not valid. To reduce non-response, it is better to conduct the surveys in person, though these are very expensive. Phones are the next best way to conduct surveys, emails can be effective, and physical mailings are the least desirable way to conduct surveys.

Voluntary response

A voluntary response is where people are asked to respond via phone, email, or online and decide to be part of the survey. The problem with these is that only people who care about the topic are more likely to respond. These surveys are not scientific, and the results from these surveys are not valid. Note: all studies involve volunteers. The difference between a voluntary response survey and a scientific study is that in a scientific study, the researchers ask the individuals to be involved, while in a voluntary they choose to be part of the study.

Example \(\PageIndex{1}\): Bias in a Study

Suppose a mathematics department at a community college would like to assess whether computer-based homework improves students’ test scores. They use computer-based homework in one classroom with one teacher and traditional paper and pencil homework in a different classroom with a different teacher. The students using the computer-based homework had higher test scores. What is wrong with this experiment?

Solution

Since there were different teachers, you do not know if the better test scores are because of the teacher or the computer-based homework. A better design would be to have the same teacher teach in both classes. The control group would utilize traditional paper and pencil homework, and the treatment group would use computer-based homework. Both classes would have the same teacher, and the students would be split between the two classes randomly. The only difference between the two groups should be the homework method. Of course, there is still variability between the students, but utilizing the same teacher will reduce any other confounding variables.

Example \(\PageIndex{2}\): Cause and Effect

Determine if a variable causes a change in the other variable.

1. Cinnamon was given to a group of people who had diabetes, and then their blood glucose levels were measured a day later. All other factors for each person were kept the same, and their glucose levels were lower. Did the cinnamon cause the reduction?
2. There is a link between spray-on tanning products and lung cancer. Does that mean that spray-on tanning products cause lung cancer?

Solution

1. Since this was a study where the use of cinnamon was controlled, and all other factors were kept constant from person to person, any changes in glucose levels can be attributed to the use of cinnamon.
2. Since there is only a link, and not a study controlling the use of the spray-on tanning products, you cannot say there is a cause-and-effect relationship between using the product and lung cancer. You can say that there is a link, and that there could be a cause, but you cannot say for sure that the spray causes the cancer.

Example \(\PageIndex{3}\): Generalization

1. A researcher conducts a study on the use of ibuprofen on people and finds that it is safe. Does that mean that all species can use ibuprofen?
2. Aspirin has been used for years to bring down fevers in people. Originally, it was tested on white males between the ages of 25 and 40 and found to be safe. Is it safe to give to everyone?

Solution

1. No, because if a drug is safe to use on one species, it doesn’t mean it is safe for all species. Ibuprofen is toxic to cats.
2. No, because if one age group can use it doesn’t mean it is safe for all age groups. There has been a link between giving a child under the age of 19 aspirin when they have a fever and Reye’s syndrome.

Authors

"1.4: How Not to Do Statistics" by Toros Berberyan, Tracy Nguyen, and Alfie Swan is licensed under CC BY-SA 4.0

Attributions

"1.4: How Not to Do Statistics" by Kathryn Kozak is licensed CC BY-SA 4.0