Skip to main content
Statistics LibreTexts

8: Resampling and simulation

  • Page ID
  • 8 Resampling and simulation

    The use of computer simulations has become an essential aspect of modern statistics. For example, one of the most important books in practical computer science, called Numerical Recipes, says the following:

    "Offered the choice between mastery of a five-foot shelf of analytical statistics books and middling ability at performing statistical Monte Carlo simulations, we would surely choose to have the latter skill."

    In this chapter we will introduce the concept of a Monte Carlo simulation and discuss how it can be used to perform statistical analyses.

    8.1 Monte Carlo simulation

    The concept of Monte Carlo simulation was devised by the mathematicians Stan Ulam and Nicholas Metropolis, who were working to develop an atomic weapon as part of the Manhattan Project ( They needed to compute the average distance that a neutron would travel in a substance before it collided with an atomic nucleus, but they could not compute this using standard mathematics. Ulam realized that these computations could be simulated using random numbers, just like a casino game. In a casino game, numbers are drawn at random; to estimate the probability of a specific outcome, you could play the game hundreds of times. Ulam's uncle had gambled at the Monte Carlo casino in Monaco, which is apparently where the name came from for this new technique.

    There are four steps to performing a Monte Carlo simulation:

    1. Define a domain of possible values
    2. Generate random numbers within that domain from a probability distribution
    3. Perform a computation using the random numbers
    4. Combine the results across many repetitions

    As an example, let's say that I want to figure out how much time to allow for an in-class quiz. Say that we know that the distribution of quiz completion times is normal, with mean of 5 mins and standard deviation of 1 min. Given this, how long does the test period need to be so that we expect everyone to finish 99% of the time? There are two ways to solve this problem. The first is to calculate the answer using a mathematical theory known as the statistics of extreme values. However, this is quite complicated mathematically. Alternatively, we could use Monte Carlo simulation. To do this, we need to generate random samples from a normal distribution.

    8.2 Randomness in statistics

    The term "random" is often used colloquially to refer to things that are bizarre or unexpected, but in statistics the term has a very specific meaning: A process is random if it is unpredictable. For example, if I flip a fair coin 10 times, the value of the outcome on one flip does not provide me with any information that lets me predict the outcome on the next flip. It's important to note that the fact that something is unpredictable doesn't necessarily mean that it is not deterministic. For example, when we flip a coin, the outcome of the flip is determined by the laws of physics; if we knew all of the conditions in enough detail, we should be able to predict the outcome of the flip. However, many factors combine to make the outcome of the coin flip unpredictable in practice.

    Psychologists have shown that humans actually have a fairly bad sense of randomness. First, we tend to see patterns when they don't exist. In the extreme, this leads to the phenomenon of pareidolia, in which people will perceive familiar objects within random patterns (such as perceiving a cloud as a human face or seeing the Virgin Mary in a piece of toast). Second, humans tend to think of random processes as self-correcting, which leads us to expect that we are "due for a win" after losing many rounds in a game, a phenonenon known as the "gambler's fallacy".

    8.3 Generating random numbers

    Running a Monte Carlo simulation requires that we generate random numbers. Generating truly random numbers (i.e. numbers that are completely unpredictable) is only possible through physical processes, such as the decay of atoms or the rolling of dice, which are difficult to obtain and/or too slow to be useful for computer simulation (though they can be obtained from the NIST Randomness Beacon).

    In general, instead of truly random numbers we use pseudo-random numbers generated using a computer algorithm; these numbers will seem random in the sense that they are difficult to predict, but the series of numbers will actually repeat at some point. For example, the random number generator used in R will repeate after 21993712^{19937} - 1 numbers. That's far more than the number of seconds in the history of the universe, and we generally think that this is fine for most purposes in statistical analysis.

    In R, there is a function to generate random numbers for each of the major probability distributions, such as:

    • runif() - uniform distribution (all values between 0 and 1 equally)
    • rnorm() - normal distribution
    • rbinom() - binomial distribution (e.g. rolling the dice, coin flips)

    Figure 8.1 shows examples of numbers generated using the runif() and rnorm() functions, generated using the following code:

    p1 <-
        x = runif(10000)
      ) %>% 
      ggplot((aes(x))) +
      geom_histogram(bins = 100) + 
      labs(title = "Uniform")
    p2 <-
        x = rnorm(10000)
      ) %>% 
      ggplot(aes(x)) +
      geom_histogram(bins = 100) +
      labs(title = "Normal")
    plot_grid(p1, p2, ncol = 3)
    Examples of random numbers generated from a uniform (left) or normal (right) distribution.

    Figure 8.1 Examples of random numbers generated from a uniform (left) or normal (right) distribution.

    You can also generate random numbers for any distribution if you have a quantile function for the distribution. This is the inverse of the cumulative distribution function; instead of identifying the cumulative probabilities for a set of values, the quantile function identifies the values for a set of cumulative probabilities. Using the quantile function, we can generate random numbers from a uniform distribution, and then map those into the distribution of interest via its quantile function.

    By default, R will generate a different set of random numbers every time you run one of the random number generator functions described above. However, it is also possible to generate exactly the same set of random numbers, by setting what is called the random seed to a specific value. We will do this in many of the examples in this book, in order to make sure that the examples are reproducible.

    # if we run the rnorm() command twice, it will give us different sets of pseudorandom numbers each time
    print(rnorm(n = 5))
    ## [1]  1.48  0.18  0.21 -0.15 -1.72
    print(rnorm(n = 5))
    ## [1] -0.691 -2.231  0.391  0.029 -0.647
    # if we set the random seed to the same value each time, then it will give us the same series of pseudorandom numbers each time.
    print(rnorm(n = 5))
    ## [1]  0.59  0.71 -0.11 -0.45  0.61
    print(rnorm(n = 5))
    ## [1]  0.59  0.71 -0.11 -0.45  0.61

    8.4 Using Monte Carlo simulation

    Let's go back to our example of exam finishing times. Let's say that I administer three quizzes and record the finishing times for each student for each exam, which might look like the distributions presented in Figure 8.2.

    Simulated finishing time distributions.

    Figure 8.2 Simulated finishing time distributions.

    However, what we really want to know is not what the distribution of finishing times looks like, but rather what the distribution of the longest finishing time for each quiz looks like. To do this, we can simulate a large number of quizzes (using the assumption that the finishing times are distributed normally, as stated above); for each of these simulated quizzes, we can record the longest finishing time. To do this, we create a new function in R called sampleMax() which simulates a sample of the appropriate size (i.e. the number of students in the class) from the appropriate distribution (i.e., normal), and returns the maximum value in the sample. We then repeat this simulation a large number of times (5000 should be enough) using the replicate() function, which will store all of the outputs into a single variable. The distribution of finishing times is shown in Figure 8.3.

    # sample maximum value 5000 times and compute 99th percentile
    nRuns <- 5000
    sampSize <- 150
    sampleMax <- function(sampSize = 150) {
      samp <- rnorm(sampSize, mean = 5, sd = 1)
    maxTime <- replicate(nRuns, sampleMax())
    cutoff <- quantile(maxTime, 0.99)
    sprintf("99th percentile of maxTime distribution: %.2f", cutoff)
    ## [1] "99th percentile of maxTime distribution: 8.81"
    Distribution of maximum finishing times across simulations.

    Figure 8.3 Distribution of maximum finishing times across simulations.

    This shows that the 99th percentile of the finishing time distribution falls at 8.81, meaning that if we were to give that much time for the quiz, then everyone should finish 99% of the time. It's always important to remember that our assumptions matter -- if they are wrong, then the results of the simulation are useless. In this case, we assumed that the finishing time distribution was normally distributed with a particular mean and standard deviation; if these assumptions are incorrect (and they almost certainly are), then the true answer could be very different.

    8.5 Using simulation for statistics: The bootstrap

    So far we have used simulation to demonstrate statistical principles, but we can also use simulation to answer real statistical questions. In this section we will introduce a concept known as the bootstrap that lets us use simulation to quantify our uncertainty about statistical estimates. Later in the course, we will see other examples of how simulation can often be used to answer statistical questions, especially when theoretical statistical methods are not available or when their assumptions are too stifling.

    8.5.1 Computing the bootstrap

    In the section above, we used our knowledge of the sampling distribution of the mean to compute the standard error of the mean and confidence intervals. But what if we can't assume that the estimates are normally distributed, or we don't know their distribution? The idea of the bootstrap is to use the data themselves to estimate an answer. The name comes from the idea of pulling one's self up by one's own bootstraps, expressing the idea that we don't have any external source of leverage so we have to rely upon the data themselves. The bootstrap method was conceived by Bradley Efron of the Stanford Department of Statistics, who is one of the world's most influential statisticians.

    The idea behind the bootstrap is that we repeatedly sample from the actual dataset; importantly, we sample with replacement, such that the same data point will often end up being represented multiple times within one of the samples. We then compute our statistic of interest on each of the bootstrap samples, and use the distribution of those estimates.

    Let's start by using the bootstrap to estimate the sampling distribution of the mean, so that we can compare the result to the standard error of the mean (SEM) that we discussed earlier.

    # perform the bootstrap to compute SEM and compare to parametric method
    nRuns <- 2500
    sampleSize <- 32
    heightSample <- 
      NHANES_adult %>%
    bootMeanHeight <- function(df) {
      bootSample <- sample_n(df, dim(df)[1], replace = TRUE)
    bootMeans <- replicate(nRuns, bootMeanHeight(heightSample))
    SEM_standard <- sd(heightSample$Height) / sqrt(sampleSize)
    sprintf("SEM computed using sample SD: %f", SEM_standard)
    ## [1] "SEM computed using sample SD: 1.595789"
    SEM_bootstrap <- sd(bootMeans)
    sprintf("SEM computed using SD of bootstrap estimates: %f", SEM_bootstrap)
    ## [1] "SEM computed using SD of bootstrap estimates: 1.586913"
    An example of bootstrapping to compute the standard error of the mean. The histogram shows the distribution of means across bootstrap samples, while the red line shows the normal distribution based on the sample mean and standard deviation.

    Figure 8.4 An example of bootstrapping to compute the standard error of the mean. The histogram shows the distribution of means across bootstrap samples, while the red line shows the normal distribution based on the sample mean and standard deviation.

    Figure 8.4 shows that the distribution of means across bootstrap samples is fairly close to the theoretical estimate based on the assumption of normality. We can also use the bootstrap samples to compute a confidence interval for the mean, simply by computing the quantiles of interest from the distribution of bootstrap samples.

    # compute bootstrap confidence interval
    bootCI <- quantile(bootMeans, c(0.025, 0.975))
    pander("bootstrap confidence limits:")

    bootstrap confidence limits:

    2.5% 98%
    164.634 170.883
    # now let's compute the confidence intervals using the sample mean and SD
    sampleMean <- mean(heightSample$Height)
    normalCI <- 
      "2.5%" = sampleMean - 1.96 * SEM_standard,
      "97.5%" = sampleMean + 1.96 * SEM_standard
    print("confidence limits based on sample SD and normal distribution:")
    ## [1] "confidence limits based on sample SD and normal distribution:"
    2.5% 97.5%
    164.575 170.831

    We would not usually employ the bootstrap to compute confidence intervals for the mean (since we can generally assume that the normal distribution is appropriate for the sampling distribution of the mean, as long as our sample is large enough), but this example shows how the method gives us roughly the same result as the standard method based on the normal distribution. The bootstrap would more often be used to generate standard errors for estimates of other statistics where we know or suspect that the normal distribution is not appropriate.

    8.6 Suggested readings

    • Computer Age Statistical Inference: Algorithms, Evidence and Data Science, by Bradley Efron and Trevor Hastie