21.4.6: Non-Parametric Bootstrap

Problem:

A sample of n=10 counts is drawn from a population with unknown distributional characteristics. From the collected data, estimate a 95% confidence interval for the population mean.

Here are the observations:

4, 3, 4, 5, 4, 4, 1, 4, 9, 8, 3

The bootstrap is a powerful and versatile tool in statistics because it provides a non-parametric method for estimating the sampling distribution of a statistic without requiring strong assumptions about the underlying population distribution. By repeatedly resampling with replacement from the observed data, the bootstrap allows statisticians to approximate the variability of estimators, construct confidence intervals, and conduct hypothesis tests.

This approach is especially valuable when theoretical distributions are difficult to derive, such as with complex estimators or small sample sizes. Furthermore, the bootstrap is widely applicable across diverse statistical methods, making it a critical tool for robustness and flexibility in real-world data analysis. Its ability to leverage computational power to provide insights where traditional methods may falter has made the bootstrap indispensable in modern statistics.

Solution:

Note that the sample size is small in this example. As such, we cannot rely on the Central Limit Theorem to assume the sample mean is (close to) Normally distributed. We do not even know if the population has a finite variance. The only thing we know is that we collected that particular sample. Thus, we will use the bootstrap.

That is, we will treat the sample as being representative of the population. Without this assumption, there is absolutely nothing we can do. With that assumption, we can effectively (or essentially) "recreate" the population as being an infinite repetition of this data. Then, to estimate the variability in the population, we simply redraw a sample of the same size from that population.

This is called the non-parametric bootstrap because it does not assume a specific (named) distribution of the data.

Here is the code to accomplish this once.

theData = c(4, 3, 4, 5, 4, 4, 1, 4, 9, 8, 3)

newData = sample(theData, replace=TRUE)
mean(newData)

That code gives us just one sample mean. It takes thousands of them to understand the distribution (centers and variabilities). Thus, the non-parametric bootstrapping code will be

theData = c(4, 3, 4, 5, 4, 4, 1, 4, 9, 8, 3)
newMeans = numeric()

for(i in 1:1e6) {
  newData = sample(theData, replace=TRUE)
  newMeans[i] = mean(newData)
}

mean(newMeans)
sd(newMeans)
quantile(newMeans, c(0.025,0.975))

From this code, I estimate the population mean is 4.45, with an estimated 95% confidence interval from 3.27 to 5.82. The figure below shows the histogram of the sample means from the simulation. The triangle on the x-axis represents the mean of these sample means. The thick bar along the axis represents the estimated 95% confidence interval.

Note

The non-parametric bootstrap is used when all you know about the population is that you randomly selected the sample. It is a very flexible procedure that should be a part of your statistical toolbox.

However, statistically, it tends to be of low power (confidence intervals are too wide; p-values are too high). The reason for this lower power is that you are making fewer assumptions on the population. If you know more about the population and are able to use it, then that procedure will have higher power (all things being equal).

How Good is the Non-Parametric Bootstrap?

Problem:

How good is the non-parametric bootstrap in terms of covering the population mean? For this investigation, let's assume the population really is standard Normal.

Solution:

The first step is to draw a sample of size n=10 from that population, then do the non-parametric bootstrap, then determine how frequently the population mean is in the confidence interval. It should happen 95% of the time.

Here is the code:

estLCL = numeric()
estUCL = numeric()

for(j in 1:1e6) {     # The loop testing the NPB 

 theData = rnorm(10)
 newMeans = numeric() 
 
 for(i in 1:1e6) {
  newData = sample(theData, replace=TRUE)
  newMeans[i] = mean(newData)
 } 
 
 estLCL[j] = quantile(newMeans,0.025)
 estUCL[j] = quantile(newMeans,0.975)
}

Note that this will take quite some time to run. There are a total of 1,000,000,000,000 iterations taking place. On a new laptop, one should probably expect it to run overnight. On an older one, it may take an entire day. However, the results will be rather precise.

Let me take this opportunity to (re-) introduce you to a measure of the quality of a confidence interval: coverage. Coverage is defined as the proportion of the time that the confidence interval contains the true mean.

For our example, the true mean is 0. So, the coverage will be the proportion of the time that 0 is between the two estimated confidence bounds.

mean( 0>estLCL & 0<estUCL )

When I run this, I obtain 0.896, which is quite different from the hoped-for 0.95. It means I am rejecting at a rate of 10.4% instead of the claimed 5%. This is not good.

Having too low of a coverage (as here) is problematic because it increases the risk that the true parameter value lies outside the confidence interval or prediction range. Coverage probability reflects the proportion of intervals that, over repeated sampling, are expected to contain the true value. If the coverage is too low, it means that the interval is overly narrow or poorly calibrated, leading to an underestimation of uncertainty. This can result in overconfidence in statistical conclusions, which may cause critical errors in decision-making or policy formulation, particularly in fields like medicine, finance, or engineering where risks are high.

Note

While the bootstrap does a fine job of estimating the center of a distribution (mean or median), it tends to be biased for other measures (quantiles, variances, etc.). Because the bootstrap is so useful to statisticians, we have spent a lot of time trying to fix these biases. This has led to several "Bias-Corrected" bootstrap estimates.

These are, however, beyond the scope of this book. But, if you are interested, check out the boot package in R and the book "Bootstrap Methods and Their Application" by A. C. Davison and D. V. Hinkley (1997).

Extension:

King Rudolph's cabinet consists of several members. Their ages are \(\{57, 62, 95, 44, 68, 65, 66, 72\}\). Repeat the above analysis to determine the coverage rate of the non-parametric bootstrap using this new data.