1.4: How Not to Do Statistics
- Page ID
- 58246
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\dsum}{\displaystyle\sum\limits} \)
\( \newcommand{\dint}{\displaystyle\int\limits} \)
\( \newcommand{\dlim}{\displaystyle\lim\limits} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)- Identify common pitfalls in statistical reasoning.
- Recognize the risks of overgeneralizing results and misinterpreting sampling errors.
- Understand the importance of avoiding incorrect assumptions about cause and effect.
- Promote accurate and reliable interpretation of statistical conclusions.
Many studies are conducted, and conclusions are made. However, there are occasions where the study is not conducted correctly or the conclusion is not correctly made based on the data. There are many things that you should question when you read a study. There are many reasons for the study to have a bias. Bias is where a study may have a certain slant or preference for a certain result. The following is a list of some of the questions or issues you should consider to help decide if there is bias in a study.
One of the first issues you should ask is who funded the study. If the entity that sponsored the study stands to gain either profits or notoriety from the results, then you should question the results. It doesn’t mean the results are wrong, but you should scrutinize them to ensure they are sound. For example, suppose a study says that genetically modified foods are safe, and the study was funded by a company that sells genetically modified food, then one may question the validity of the study. Since the company funds the study and its profits rely on people buying their food, there may be a bias.
An experiment could have lurking or confounding variables when you cannot rule out the possibility that the observed effect is due to some other variable rather than the factor being studied. For example, suppose you give fertilizer to some plants and no fertilizer to others, but the plants are not growing taller due to a lack of sunlight and not due to a lack of fertilizer. You won’t know if the plants that received the fertilizer grew taller because of the fertilizer or the sunlight. Make sure you design experiments to eliminate the effects of confounding variables by controlling all the factors.
Overgeneralization
Overgeneralization is a study of one group but makes conclusions for all groups. An example is doing cancer treatments on rats. Just because the treatment works on rats does not mean it will work on humans. Another example is that until recently, most FDA medication testing had been done on white males of a particular age. There is no way to know how the medication affects other genders, ethnic groups, age groups, and races. The new FDA guidelines stress individuals from different groups.
Cause and Effect
Cause and effect is where people assume that one variable causes the other when the variables are only correlated. Unless the study was done as an experiment where a variable was controlled, you cannot say that one variable caused the other. Most likely, there is another variable that caused both. As an example, there is a relationship between the number of drownings at the beach and ice cream sales. This does not mean that purchasing ice cream increases the chance people will drown. Most likely, the cause for both increases is the heat.
Sampling Error
This is the difference between the sample results and the true population results. This is unavoidable and results in the fact that samples are different from each other. For example, if you take a sample of 5 people’s heights in your class, you will get 5 numbers. If you take another sample of 5 people’s heights in your class, you will likely get 5 different numbers.
Nonsampling Error
This is where the sample is collected poorly, either through a biased sample or an error in measurements. Care should be taken to avoid this error.
Lastly, there should be care taken in considering the difference between statistical significance versus practical significance. This is a major issue in statistics. Something could be statistically significant, which means that a statistical test shows there is evidence to support what you are trying to prove. However, in practice, it doesn’t mean much, or there are other issues to consider. For example, suppose you find a new drug for high blood pressure that reduces blood pressure. When you look at the improvement, it doesn’t amount to a large difference. Even though statistically there is a change, it may not be worth marketing the product because it isn’t that big of a change. Another consideration is that the blood pressure medication improves a person’s blood pressure, but it has serious side effects, or it costs a great deal for a prescription. In this case, it wouldn't be practical to use it. In both cases, the study is shown to be statistically significant, but practically, you don’t want to use the medication. The main thing to remember in a statistical study is that the statistics are only part of the process. You also want to make sure that there is practical significance, too.
Surveys
Surveys have their own areas of bias that can occur. A few of the issues with surveys are in the wording of the questions, the ordering of the questions, the manner the survey is conducted, and the response rate of the survey.
The wording of the questions can cause hidden bias, which is where the questions are asked in a way that makes a person respond a certain way. An example is that a poll was done where people were asked if they believed that there should be an amendment to the Constitution protecting a woman’s right to choose. About 60% of all people questioned said yes. Another poll was done where people were asked if they believed that there should be an amendment to the Constitution protecting the life of an unborn child. About 60% of all people questioned said yes. These two questions deal with the same issue, though giving opposite results, but how the question was asked affected the outcome.
The ordering of the questions can also cause hidden bias. An example of hidden bias is if you were asked if there should be a fine for texting while driving, but the preceding question asked if you ever texted while driving. By asking a person if they partake in the activity, that person now personalizes the question and may affect how they answer the next question.
Non-response
Non-response is when you send out a survey, but not everyone responds. You can calculate the response rate by dividing the number of responses by the number of surveys sent. Most response rates are around 30-50%. A response rate that is less than 30% is very poo,r and the results of the survey are not valid. To reduce non-response, it is better to conduct the surveys in person, though these are very expensive. Phones are the next best way to conduct surveys, emails can be effective, and physical mailings are the least desirable way to conduct surveys.
Voluntary response
A voluntary response is where people are asked to respond via phone, email, or online and decide to be part of the survey. The problem with these is that only people who care about the topic are more likely to respond. These surveys are not scientific, and the results from these surveys are not valid. Note: all studies involve volunteers. The difference between a voluntary response survey and a scientific study is that in a scientific study, the researchers ask the individuals to be involved, while in a voluntary they choose to be part of the study.
Suppose a mathematics department at a community college would like to assess whether computer-based homework improves students’ test scores. They use computer-based homework in one classroom with one teacher and traditional paper and pencil homework in a different classroom with a different teacher. The students using the computer-based homework had higher test scores. What is wrong with this experiment?
Solution
Since there were different teachers, you do not know if the better test scores are because of the teacher or the computer-based homework. A better design would be to have the same teacher teach in both classes. The control group would utilize traditional paper and pencil homework, and the treatment group would use computer-based homework. Both classes would have the same teacher, and the students would be split between the two classes randomly. The only difference between the two groups should be the homework method. Of course, there is still variability between the students, but utilizing the same teacher will reduce any other confounding variables.
Determine if a variable causes a change in the other variable.
- Cinnamon was given to a group of people who had diabetes, and then their blood glucose levels were measured a day later. All other factors for each person were kept the same, and their glucose levels were lower. Did the cinnamon cause the reduction?
- There is a link between spray-on tanning products and lung cancer. Does that mean that spray-on tanning products cause lung cancer?
Solution
- Since this was a study where the use of cinnamon was controlled, and all other factors were kept constant from person to person, any changes in glucose levels can be attributed to the use of cinnamon.
- Since there is only a link, and not a study controlling the use of the spray-on tanning products, you cannot say there is a cause-and-effect relationship between using the product and lung cancer. You can say that there is a link, and that there could be a cause, but you cannot say for sure that the spray causes the cancer.
- A researcher conducts a study on the use of ibuprofen on people and finds that it is safe. Does that mean that all species can use ibuprofen?
- Aspirin has been used for years to bring down fevers in people. Originally, it was tested on white males between the ages of 25 and 40 and found to be safe. Is it safe to give to everyone?
Solution
- No, because if a drug is safe to use on one species, it doesn’t mean it is safe for all species. Ibuprofen is toxic to cats.
- No, because if one age group can use it doesn’t mean it is safe for all age groups. There has been a link between giving a child under the age of 19 aspirin when they have a fever and Reye’s syndrome.
Authors
"1.4: How Not to Do Statistics" by Toros Berberyan, Tracy Nguyen, and Alfie Swan is licensed under CC BY-SA 4.0
Attributions
"1.4: How Not to Do Statistics" by Kathryn Kozak is licensed CC BY-SA 4.0
Exercises
- Suppose there is a study where a researcher conducts an experiment to show that deep breathing exercises help to lower blood pressure. The researcher took two groups of people, one group performed deep breathing exercises and a series of aerobic exercises every day, while the other group was asked to refrain from any exercises. The researcher determined there is a connection between deep breathing exercises and lower blood pressure. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- Suppose a car dealership offers customers a low interest rate and a longer payoff period or a high interest rate and a shorter payoff period. Most customers at the dealership choose the low interest rate and longer payoff period. The car dealership concluded that customers generally prefer low interest rates no matter the type of product sold. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- A study that has shown that coffee is linked to poor health also discovered that people who tend to drink coffee don’t sleep much, tend to smoke, don’t eat healthy, and don't exercise. The study concluded that coffee directly impacts poor health. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- When researchers were trying to figure out what caused polio, they saw a connection between ice cream sales and polio. As ice cream sales increased so did the incident of polio. They concluded eating ice cream causes polio. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- A study shows that polytetrafluoroethylene (PFOA) increases the risk of tumors in lab mice. Based on the results of the study, the researchers concluded that PFOA’s have an increased risk of tumors in people. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- Suppose a telephone poll is conducted by contacting U.S. citizens via landlines about their views on vaping. Suppose over 50% of those called do not support vaping. The ones conducting the poll concluded that 50% of all people in the U.S. do not support vaping. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- Suppose that it can be shown to be statistically significant that a smaller percentage of the people are satisfied with your business. The percentage before was 87% and is now 85%. Because it was statistically significant, they decided to change the way they conduct business. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- You are testing a new drug for weight loss. You find that the drug does statistically show weight loss but it is so small that it is considered negligible. You decide to market the drug. Please explain the issue with the way it was conducted.
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- An online poll was conducted about whether the mayor of San Diego, California, should resign due to an affair. The people decided if they wanted to participate in the poll and the majority said the mayor should resign. What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- A survey was conducted at ABC College among all employees. The survey asked: "Are you dissatisfied with the level of incompetence provided by the Security Department?" What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- A student survey was conducted at Citrus College where the research group wanted to know if the services provided at the college addressed their needs. They decided to email the students and only 10% responded to the survey. Then, they analyzed the findings of the survey responses. What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- In a survey, the first question asks, "How often do you exercise per week?" The second question asks, "How would you rate your overall health?" What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- An employee survey says, “Employees at this institution are very satisfied with working here. Please rate your satisfaction with the institution.” What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- An online poll that anyone can join and fill out showed that the majority of Americans believe that the government covered up evidence of UFOs. What is the problem with how the question or survey was done that might lead to unfair or inaccurate results?
Scan the QR code or click on it to open the MyOpenMath version of the above question with step-by-step guidance.
MyOpenMath is a free online learning platform designed to support math instruction through automated homework, quizzes, and assessments. You must register for MyOpenMath and sign in to view the question.
- Answers
-
If you are an instructor and want the solutions to all the exercise questions for each section, please email Toros Berberyan.