Internet / Web Surveys – Pros and Cons
Thinking about web surveying?
How valid are they and are they right for your situation?
One of the greatest problems facing online surveys is a concern that they may not capture representative samples. Accuracy of inferences about an entire population, based on survey results, depends on representative sampling. Since all members of many populations do not have access to the Internet, it is questionable whether you can make accurate inferences from online surveys, especially if those with online access systematically differ from those without.
Some may be tempted to say “Forget it’“Web data is so fast and cheap it doesn’t matter.” But if there are important differences between individuals who are and are not online, results can be biased. Massive online panels, weighting of online results, or other potential approaches cannot fully solve the problem. An online panel is still just that’“online. Weighting might help, but logically, it can only do so much. No amount of weighting can turn an “online person” into a non-online person. However, online methods can be a highly effective, practical method of conducting research under the right circumstances. Indeed, it may be the best methodology in some situations. But the choice of an optimal research design, online or otherwise, always depends on things like objectives of the research, population of interest, availability of sample, budget, timing requirements, subject matter, and other design factors. Rather than unequivocally defaulting to online as “best,” instead make your decisions about its use in light of certain considerations.
Things to consider…
A key consideration is whether an online sample truly can represent the population of interest. The answer could be yes in the technology sector where an entire population of interest may be online. Also, in some cases it may be possible and even desirable to conduct a census online (for example, employee surveys for some companies). But many other situations will not have the luxury of Internet access for all respondents. This raises a second consideration ‘“ equivalence of online/non-online groups. If access is not universal, are those online systematically different from those not on line ‘“ particularly regarding the research questions at hand? If they are similar, online surveys still may be reasonable, at least for the online group. If the groups (online/non-line) are not similar, you need to consider at a minimum a “mixed-mode” design (online + some other method). And if the mixed-mode design shows differences across the groups, there will always be the lingering question ‘“ “are differences from the survey method or from initial group heterogeneity?” That leads to a third consideration ‘“ will Web-based surveys introduce bias that is not present in other methods (for instance, phone, or mail surveys)? The only true test of this “method effect” comes from studies simultaneously comparing online and other methods, where equality of groups is ensured before administering different survey modalities. Then, any differences in results can be attributed to the method of data collection, not group differences. Random assignment to conditions is the tried-and-true way of ensuring group similarity from the start. Separate representative random samples from the same population, or, elaborate matching on characteristics (matched samples) are two other reasonable scientific approaches. Unfortunately a number of other comparative studies have not ensured equality/similarity among test groups. These contain “confounding” ‘“ a design ambiguity that obscures understanding the true sources of observed differences. When differences occur with a confounded design. It could be due to the Web-based methodology, or it could be due to initial group differences. Indeed, if the groups start out being different, should we really be surprised if they produce dissimilar survey results?
Researchers considering online surveys should decide whether it is reasonable to assume that there is no pure method effect. They may want to test for this in their own settings with a truly scientific design for comparing parallel administrations, or simply rely on published literature to reasonably rule out modality bias as a threat to validity.
Bottom line…
There is no question that cost, timing, and other advantages make online surveying very attractive. After careful attention to the considerations described, a researcher might decide to use a purely online methodology. Or, if needed, a researcher might consider a number of mixed-mode options/tests (for instance, pre-notification with a Web address to save the cost of some telephone interviews ‘“ given no pure method bias). Also, a researcher should give serious consideration to design variables that will maximize response rates. The literature on mail surveys is rich with food for thought here. Published studies are just beginning to emerge testing the effects of varying design variables on online survey response. Finally, consider the actual technology for administering the survey. Just as skips, rotations, and a host of other options exist for CATI surveys, Web-survey technologies should be evaluated for their flexibility and ability to implement time-tested research practices. All of these capabilities should be applied with principles of good research science.
What does the future hold?
Questions about viability and representativeness of Web-based surveys should diminish as the online population becomes increasingly mainstream, and as access to the Web increases through cheaper PCs, Web TV, access through schools, PCs in businesses, and telecommuting setups in homes. Industry acceptance also should increase over time. Just as telephone interviewing once was viewed with skepticism (since much research at that time was done door-to-door), no doubt Web-based surveys are also a “disruptive” technology likely to change data collection in the future.
For now novelty, speed, cost, individual convenience, and other advantages have brought Web-based surveys rapidly into the research spotlight. It is unlikely that the next few years will bring any decrease in this trend. If anything, you can expect continued exponential growth.
The advantages of Web-based surveys could make them an optimal applied methodology of choice in many situations ‘“ given careful attention to good research design considerations. In many cases, Web-based surveys offer a practical, cost-effective, time-efficient, leading edge approach to data collection that just might change the face of research in your organization.
By: Doug Grisaffe, Ph.D., Professor of Marketing, University of Texas, Arlington
