7.5.2: Summary of p-values and NHST
- Page ID
- 18694
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)We use p-values (probability) to determine if this could have happened by chance. We are comparing the mean of our sample to the mean of the population to ask “Is it likely that my sample is from the population?”
Since we can’t ever collect data from the whole population, we’re forced to make inferences from the available sample data, and the probability of events happening that we know from the Standard Normal Distribution. We know that any score that is more 3 or more standard deviations from the mean (z = 3) is very rare. So, we compare the mean of a sample to the population mean (after standardizing so that we can use the Standard Normal Distribution) to see if the mean of the sample is extreme enough to say that the probability of the sample being from the population is below 5%.
The p-value tells us the probability of getting an effect this different if the sample is the same as the population.
If the probability (p-value) is small enough (p< .05), then we conclude that the sample mean probably is from a different population.
Exercise \(\PageIndex{1}\)
What does the null hypothesis say?
- Answer
-
The null hypothesis says that there is no difference between the groups, that the mean of the sample is similar to the mean of the pouplation.
Example \(\PageIndex{1}\)
If we’re saying that the sample is probably different from the population, are we rejecting or retaining (failing to reject) the null hypothesis?
Solution
If we’re saying that the sample is probably different from the population, we reject the null hypothesis.
Interpreting p-values
Without having to understand everything about probability distributions and the Standard Normal Distribution, what do the p-values tell us?
Small p-values
A small p-value means a small probability that the two means are similar.
Suggesting that the means are different…
We conclude that:
- The means are different.
- The sample is not from the population.
Large p-values
A large p-value means a large probability that the two means are similar.
We conclude that
- The means are similar.
- The sample is from the population.
Note
In other words:
Reject null = \( \bar{X} \neq \mu \) = p<.05
Retain null = \( \bar{X} = \mu \) = p>.05
If the probability is less than 5% that you would get a sample mean that is this different from the population mean if the sample really is from the population, the sample is probably not from that population. If you took 100 samples from a population, less than 5 of the samples’ means would be this different from the population’s mean if the samples were different in reality. So, the sample is probably not from the population. Probably.
You have a 5% chance that your results are wrong. You have a (small) chance that your sample mean is this different from the population mean, but the sample is still actually from the population.
Statisticians are okay with being wrong 5% of the time!
Let's practice!
Example \(\PageIndex{2}\)
For each:
- Determine whether to report “p<.05 ” or “p>.05”
- Determine whether to retain or reject the null hypothesis.
- Determine whether the sample mean and population mean are similar or different.
Hint:
Reject null = \( \bar{X} \neq \mu \) = p<.05
Retain null = \( \bar{X} = \mu \) = p>.05
1. p < .05
2. p = .138
3. p = .510
Solution
- p < .05
- “p<.05 ”
- Reject the null hypothesis.
- The sample mean and population mean are different.
- p = .138
- “p>.05” (because 0.138 > 0.05, 0.138 is bigger than 0.05)
- Retain the null hypothesis.
- The sample mean and population mean are similar.
- p = .510
- “p>.05” (We’re comparing to 0.05, or 5%, not .50, or 50%)
- Retain the null hypothesis.
- The sample mean and population mean are similar.
Now, try it yourself:
Exercise \(\PageIndex{2}\)
For each:
- Determine whether to report “p<.05 ” or “p>.05”
- Determine whether to retain or reject the null hypothesis.
- Determine whether the sample mean and population mean are similar or different.
Hint:
Reject null = \( \bar{X} \neq \mu \) = p<.05
Retain null = \( \bar{X} = \mu \) = p>.05
1. p > .05
2. p = .032
3. p = .049
- Answer
-
- p > .05
- “p>.05 ”
- Retain the null hypothesis.
- The sample mean and population mean are similar.
- p = .032
- “p<.05 ” or “p>.05” (because 0.32 < .05, 0.032 is less than 0.05)
- Reject the null hypothesis.
- The sample mean and population mean are different.
- p = .049
- “p<.05 ” (even though it’s close, .049 is smaller than 0.05)
- Reject the null hypothesis.
- The sample mean and population mean are different.
- p > .05