This blog post is guest authored by Dr. Anne Beall, President of Beall Research and Training. Anne is also the author of the succinct and very useful book Strategic Market Research. I asked Anne to write this post because I have found that it is sometimes challenging to convince clients that using a 10-point scale for their customer satisfaction surveys is the best way to go.
From Anne:
As a market-research professional, I have strong feelings about rating scales, and they are the foundation of my work. Using scales that are sub-optimal is similar to using low-grade concrete for building a house. The finished product might look nice, but the structure is built from materials that are unreliable. I recommend using a 10-point scale for several reasons.
1. It requires a smaller sample size in order to have the same degree of precision as a scale with fewer points. According to researchers who analyzed this issue, a 10-point scale requires 71% of the sample size that a 5-point scale requires in order to have the same measurement precision. Thus, if you have the same sample size, you will have 71% of the power to detect a statistical effect with a 5-point scale than with a 10-point one.
2. It is more sensitive at measuring differences than scales with fewer points. For example, with customer satisfaction research, 44% of the respondents using a 5-point scale scored Company A below a 5. However, with a 10-point scale, 82% of respondents scored Company A below a 10. Thus the proportion of customers for whom satisfaction can be improved is larger when using the 10-point scale (The Measurement Imperative by D. R. Wittink & L. R. Bayer in Marketing Research, Fall 2003). Researchers also found that a multiple regression performed on the same data yielded more predictive effects in the 10-point scale data than the same data collected with a 5-point scale.
3. It has greater statistical reliability and validity. Reliability is the extent to which the measurements of a test remain consistent over repeated tests. The reliability of data can be analyzed by looking at how much data changes when you add or subtract different predictors in a statistical analysis. Data collected with the 5-point scale has more instability and the predictive variables are less consistent than the same data collected with a 10-point scale. Given that the data is less stable, we can assume that it is less valid as a result.
So if you want a scale that has the greatest statistical reliability, validity, sensitivity and ability to detect differences, go with a 10-point scale. And to make sure you’re asking great questions overall, this post from our Top Ten Entrepreneurial Research Mistakes series may be helpful.
I loved this entry. We use a 5 point scale for everything at work, and I’ve always felt that a 10-point scale was better. I couldn’t articulate it (and have had more than a few conversations about it), but I just knew that 10 points were better. Now I feel vindicated, and I have a 3-point argument that I can use for now on. Thanks.
Its funny, as a researcher buyer i find 10 point scales incredibly frustrating. For measures of satisfaction for example, you may have a statistically significant movement from 6.4 to 7.2 in satisfaction ratings. The problem is, its not clear what that means, as you can’t associate clear response options as you can with a smaller ordinal scale. It can only be as clear as it is to the respondent being asked the question… as in its somewhere between extremely satisfied and extremely dissatisfied.
From a buyers perspective, that really isn’t useful.
And ironically, i find when research consultants do use 10 point satisfaction scales, they also tend to report it in a summarised 5 or 4-point response scale or ‘top 2/4 box’ when doing a write up. So instead of allowing the respondent to make the decision on an ordinal response scale, the researcher decides to interpret it for them based on their response to a rating out of 10. Which is in itself an element of bias which could have been avoided.
Anyway, am not disputing your points. 10 point scales have statistical benefits, but are not necessarily “better” in all cases. Personally, i’d take an ordinal 5 point with a clearly defined response scale for satisfaction measures.
Re:my comments – sorry, when i talk about ‘ordinal’ i mean your standard 5 point likert scales… which aren’t necessarily ordinal.
Brendan – Thanks for your thoughts on this! As someone who helps my clients work with research findings, it’s tougher to see small increments of improvements with smaller scales. And most of my clients are dealing with small increments. I think one of the key things is putting research in context somehow. Ideally by conducting similar research over time and benchmarking.
Most of the published research does not agree with your conclusions. Look at the work by Jon A. Krosnick at Stanford. He concludes: Our data consist of results from 706 tests of reliability taken from thirty different between-subject studies…. we found that five- or seven-point scales produced the most reliable results. Where is the empirical data to support these points? – a 10-point scale requires 71% of the sample size that a 5-point scale requires – It has greater statistical reliability and validity.
Dan – Thanks so much for your comments! I am going to pass this along to my guest blogger and ask her to reply. Stay tuned!
Thanks for your post. What published research are you referring to exactly? You mention Jon Krosnick’s research where he makes a conclusion that he posted on his web page. It’s unclear exactly what he tested and what kinds of studies he analyzed. This conclusion is based on an article that has never been reviewed or published. Correspondence with Jon Krosnick confirmed that fact.
In contrast, the empirical data that supports my claims is from an article that analyzed the exact same data that was collected over the course of several years and that used both 5-point and 10-point scales. It’s called The Measurement Imperative by Wittink and Bayer in Marketing Research (2003, Vol 15, Part 3, pages 19-22).
I always thought 5-point scale was enough; but I am having second thoughts now.