7.6: Summary of Reasons Why You Need a Large Sample Size

• Drawing a conclusion based on a sample size of one is inaccurate. The reason why you need a large sample size is that you have no idea if the next sample you take is going to be the same or different from the first sample. Likely, the next sample is going to be different.
• Drawing a conclusion based on a sample size of two is inaccurate because you cannot establish an upward or downward trend in the scores. You have no idea if the next or third sample you take is going to continue the trend established by the first two samples. For continuous observations, the first sample could be low; the second sample could be high, suggesting an upward trend. However, the third sample could be low, suggesting a downward trend, or high, suggesting a continuous trend. The fourth sample could be low, suggesting a downward trend, or high, suggesting a higher trend. You need a larger sample size to determine the overall trend, increasing or decreasing, of the variable.
• Type I and Type II error considerations. Low sample sizes tend to result in type I or type II errors. If you have a sample size of four observations, and all four observations confirm your hypotheses, technically, you have made a Type I error. However, the likelihood of you making the Type I error increases because it is possible that if you sample another 26 observations, those 26 observations will not confirm your hypothesis, hence the Type I error. The same is true for the Type II error. If you have a sample size of four observations, and all four observations do not confirm your hypotheses, technically, you have made a Type II error. However, the likelihood of you making the Type II error increases because it is possible that if you sample another 26 observations, those 26 observations will confirm your hypothesis, hence the Type II error. The sample size and the Type II error considerations are addressed with a power analysis, which is covered in the next chapter.
• Sampling bias is a theoretical consideration for why we need a large sample size. Every sample we take has a sampling bias; in this case, the sample, specifically the sample mean, will not be a correct representation of the population's mean. Every sample we take will be higher or lower than the population's mean. We can never know because the population's mean is unknown. If we do not know, then our best alternative is to keep sampling and hope the culmination of our samples will bring us closer to the population's mean. But to do that, we keep sampling, hence the need for a larger sample size.
• The standard error of the mean is a mathematical reason for needing a large sample size. The standard error of the mean needs to be low because that is the gauge we use to determine if we are getting closer to the population's mean. The mathematical formula for the standard error of the mean involves the sample size as the denominator. When the denominator is large and divided by a small number in the numerator, the result is a small number, less than one. The larger the sample size, the larger the denominator, and the calculation will yield a number considerably less than one.
• The Central Limit Theorem is a second theoretical consideration for why we need a large sample size. The theorem states that when sampling continues at random, the distribution of sample means will theoretically match the population parameters (normal distribution, standard error, population mean). The theorem is based on continuing to sample or a large sample size.

7.6.1: Guidelines for Sample Sizes

Notice that the preceding reasons for needing a large sample size do not specify exactly how large the sample size needs to be. They do not state the lower bound or the minimal sample size you need. There is no indication of how small the sample size must be when conducting statistical analyses becomes problematic.

There are no good guarantees regarding what the sample sizes should be. There are too many considerations, such as the distribution of the variable in question, the research question itself, and the logistics of recruiting the population under consideration. The other problem is missing data. Missing data is inevitable. Participants start surveys and then do not complete the remaining questions or skip other surveys. Participants skip questions. Participants complete the demographics and nothing else. Every time you obtain a pool of participants, it is likely that not everyone in the pool will complete all our study surveys. Even if you do set a minimum sample size, you always want to plan to survey more as a backup plan because you need the extra data to compensate for the missing data that will inevitably occur.

There are some guidelines for what sample sizes should be. These are guidelines, not rules. Anyone who states that the numbers are strict and that any sample sizes below the guidelines makes the study invalid is clearly wrong and does not pay attention to the context of the study. A problem with stating what the sample size should be as a rule occurs when the sample size is just underneath the minimum sample size. If a sample size is set at 30, what happens when the sample size is 29? If you do not have that 30^th data point, does that mean your statistics are invalid? No. Statistical results do not operate on a green light/red light switchboard. The statistics do not tip over into significance when the sample size is 30 but are not significant if the sample size is 29. However, students (and faculty) get worried about not meeting the sample size mark. The sample size does not solely make or break significance.

The same is true for a maximum sample size. Why not increase the sample size? Because a large sample size does not give you any additional information about the stability of your statistical results beyond the information you obtain from your presumably sufficient sample size. A sample size that is too large would be considered to contribute to a Type I error. The more you sample, the more you will find a result anyway, not because you found an actual result but because you sampled so many times, based on chance alone, and eventually, you will find something. We call that fishing, if you recall. The more you fish, the more you’ll eventually catch, but that doesn’t make you a fisherman; it makes you lucky. Students (and faculty) might consider getting a large sample size to conduct supplemental or additional analyses to find interesting effects beyond their hypotheses. But that still is not a good reason to keep sampling because, conceptually, it does not make sense according to your hypotheses, and you are just collecting data, just in case, to conduct additional analyses.

So, what are the guidelines?

• The first guideline is that the sample size should be at least 30.
• The second guideline is that the sample size should be a maximum of 300.
• The third guideline is that there should be a sample size of five per survey item.
• The power analysis is a primary guide for determining your sample size, which is the subject of the next chapter.

Where did these guidelines come from? These sample sizes are based on computer simulations of random data. For the sample size of 30, computer simulations indicate the standard error of the mean for a given sample does not change considerably when the sample size is 30. The same goes for a sample size of 300, as computer simulations indicate the coefficients of association between variables do not change considerably when the sample size is 300.

The sample size of five participants per item on a survey instrument is based on factor analysis. Despite concerns about factor analysis as the analysis of choice for examining dimensions of scales, the five participants per survey item is a guideline for determining if an item is an indicator of a scale dimension.

The sample size issue is a complex one. Yes, it cannot be ignored, but to place so much stock in a single number as the only determinant for statistical significance, and if a student (or faculty member) should proceed with a study, is not good practice. The bottom line is if someone is criticizing your study and its sample size, or preventing you from conducting your idea solely on the basis of anticipating that you will not get your desired sample size, or making you feel nervous or uncomfortable because your sample size will be inadequate, you need to consult with someone (and you are welcome to consult with me). No one should make you feel like a study cannot be done just because you cannot collect an adequate sample. Yes, the sample size needs to be considered, but no one should make you feel bad about any consideration of your sample size. Always consult with someone so that at the end of the day, you are comfortable with any decision you have about your study, your results, and the role of the sample size for interpreting your results.

Helpful Websites

https://www.statisticshowto.com/prob...tion-examples/

https://statisticsbyjim.com/basics/c...limit-theorem/