Question from customer on small sample survey applicabilityBelow are some questions from one of our customers dealing with small sample size surveys at fast-food restaurants. I don't have much experience here, other than that the survey responses can often be skewed to include only people who are extremely pleased or extremely displeased.
Does anyone have any experience here? Or references I could point the customer to?
Thanks!
Hoping you have some advice on the following question as we get asked quite often:
From a purely statistical approach is there a way to explain how low survey counts can still be representative of a restaurants total experience?
- On average we get around 50 to 70 survey responses per month for a restaurant averaging 30 to 40,000 visits
Operators ask about how representative the results are based on what is a fairly small sample size?
- Additionally, what we see is those with above average sample tend to perform better while those with lower sample tend to have worse results.
While the program is meant to help them to gain valuable feedback from their customers it also obviously leads to some tension.
Re: Question from customer on small sample survey applicabilityMy response would be that the 50-70 responses are not representative of the population; but that doesn't mean they don't have value. In fact, they are already drawing inferences about performance from this sampling.
Re: Question from customer on small sample survey applicabilityI would ask more questions....it is not so much a question of the number as to the quality of the responses. As you noted, are they the unhappy customers? Are these responses from comment cards on a restaurant counter or are they responses from a well-designed survey sent to a well-defined set of customers?
A set of data from 50 - 70 customers over a number of months could be useful. I could see control charting/pareto plotting, etc. if these are counter type cards. The data could be used for improvement, again depending on what/how the data is being collected.
I would start with the goal of the survey....and move on from there....
Re: Question from customer on small sample survey applicabilityThese types of surveys commonly have low response rates, which reduce the representativeness and reliability of the results obtained. Small sample size is one potential problem, but it’s the sample self-selection bias that matters more.
- Additionally, what we see is those with above average sample tend to perform better while those with lower sample tend to have worse results.
While the program is meant to help them to gain valuable feedback from their customers it also obviously leads to some tension.
If I understand the above statements correctly, responses from lower-than-average samples seem to come disproportionately from the unhappy customers who’re eager to participate. The survey results from self-selected samples are useful up to the degree that they cannot be generalized to the customer base they serve. There is no magic bullet to fix this. But what they can do is when comparing results across restaurants list the characteristics of the restaurants (e.g., years in business) and surrounding area demographics (highway, residential, downtown, etc.) etc. If a regression model were built on a survey repose this information would be used as control variables.
Re: Question from customer on small sample survey applicabilityThanks for the replies so far. Much appreciated! I will forward them to the customer.
If anyone else has input, please pass it along! 
Re: Question from customer on small sample survey applicabilityI would also like to share a blog from Qualtrics which discusses sample size and its limitations.
Much of the blog is in agreeance with what has already been suggested: A limited sample size does not mean that meaningful conclusions cannot be made, but statistical significance will certainly be impacted. This blog includes a sample size calculator, which factors in your desired confidence level, population size, and margin of error. A population of 40,000 with a 90% confidence level, and 10% margin of error would require a sample size of at least 68 (according to this calculator). Of course more data would be more statistically reliable---alternatively, the trends of the data could be explored rather than placing emphasis on statistical testing. Analyzing the data aggregated on a quarterly, bi-annual, or annual basis may also be more representative than a month by month comparison.
Also something I would add is to consider the type and amount of survey questions asked as well as the method of asking. For example, customers could first be asked to answer one question: On a scale of 1 -10, how satisfied were you with our service? Then they could be prompted to share more about their experience if they so desired. More people may be willing to answer this simple question and, while not very detailed, could result in a little more data from those who would not answer a survey at all based upon the perceived length of time of time to take the survey. Further, the information from this generic question could be used to contextualize the responses of particular groups who went on to provide more detailed responses. For instance, let's say 200 people responded to the scaled satisfaction question and we found the average to be eight. We also have more detailed survey responses from twenty people and they were mostly negative. Based on the average satisfaction score, we could say that the received negative responses are likely not representative of the entire population, but may be representative of those belonging to a subset of unsatisfied customers which are in the minority.
Hope this helps!
