Have you ever – just for fun – played around with a photograph in Adobe Photoshop? It's an incredibly powerful tool. You can change lighting...the colors...the composition...even the content. (Remember the royal wedding pictures where Prince William wasn't smiling? So they doctored the photo...?) And we haven't even started talking about the dozens of filters that you can use to turn your photo into a watercolor from outer space with a fuchsia-to-lime mezzotint and a Gaussian blur for extra effect... In the hands of a graphics artist, Photoshop is an amazing tool. In the hands of a curious novice, it's a fun toy. We (I consider myself part of this group) can do some very basic things effectively. But for most novices, Photoshop becomes an unproductive (albeit entertaining) distraction because it is TOO powerful. The myriad of possibilities is distracting. Entrenched in the tools, we lose track of the fact that we lack the essential understanding of the subtleties that allow real graphics artists to do their thing.
Electronic survey tools present a similar problem. In recent years, a number of powerful survey tools have emerged on the Web. Some of these tools make it easy to post very sophisticated surveys and collect data on the Web. Distribution is cheap. Data entry, verification, and descriptive analysis can be completely automated. Just-in-time information survey help (e.g., links to definitions of terms) and multimedia resources can be embedded directly into the survey. Since contingent question presentation can be handled invisibly, there is no more: "If you answered, 'Yes,' skip to question 26b". Further, using electronic mail to invite participants to the survey, transitions participant recruiting from a pull to a push activity.
These are all very powerful tools. You can literally implement a survey in a few minutes. It is easy to forget, in this domain of apparently instantaneous productivity, that there is an entire field of knowledge that supports the design of effective and meaningful survey data collection. As with Photoshop, the power of the tool distracts us from the essence of the task.
A recent survey paper by Andrews, Nonnecke and Preece (2003) highlights the challenges of developing effective surveys. Based on a comprehensive review of the literature, they discuss several areas of concern for researchers creating electronic surveys. They include sampling and participant selection, and survey design.
It is not unusual today for Web content managers to simply add a link to their home page that says: "Give us feedback. Participate in a survey." Who clicks on these links? Why?
Effective sampling – or getting an appropriately large and representative set of participants to respond – presents one of the greatest challenges for effective survey implementation in any mode. If the sample isn't representative, the survey results are not very meaningful. So this is a very important concern.
Achieving an appropriate representative random sample with a reasonable return rate is never trivial. Unfortunately, the nature of the Internet makes accessing a representative sample virtually impossible. Research suggests that individuals who participate in Internet surveys use the net more frequently and have better Internet skills than those who opt out (Kehoe & Pitkow, 1997). They are just different. While the male-female gap has largely disappeared (NUA, 2001), electronic survey participants tend to have higher incomes and education levels and tend to be more homogeneously Caucasian than the general population of Internet users. The demographic gaps (economics, age, ethnicity) between on-line and off-line populations are even larger (Yun & Trumbo, 2000).
Since there is no common registry of Internet users, there is no way to identify all possible users (sometimes called "the sampling frame"). As such, participant selection is necessarily nonrandom from the offset. Surveyors must remember that generalizing the results to off-line (or even the whole on-line) population may not be viable.
Available tools notwithstanding, knowing who you want to survey, and how you will reasonably access and get responses from that group, is critical to developing a meaningful survey. Just adding a link typically won't get you there.
Couper (2000) suggests implementing strategies such as intercept sampling to achieve a pseudo-random sample. In intercept sampling every nth visitor is invited to participate in the survey – like an exit poll. Anecdotal evidence suggests that inviting individuals to participate via a pop-up with a single "Would you...?" question followed up with an e-mail message containing a link to the actual survey yields increased response rates. It's a start.
Getting people to finish the survey is also a challenge. Have you been overwhelmed in a survey by an entire page of 1-7 ratings and just bailed? Surveys that appear long and mentally intensive typically don't get finished. Andrews, Nonnecke and Preece (2003) note that it is important to make the survey appear easy and usable. They review several research-driven findings to increase the likelihood that respondents will complete the survey:
In addition to their literature review, Andrews, Nonnecke and Preece (2003) review a case study of the development of a survey. Their goal was to determine whether effective and thoughtful design would allow researchers to elicit survey responses from 'lurkers' or non-active participants in electronic communities. (Yes, it does.) More interesting though, are the (re)discoveries these trained researchers experienced as they piloted their survey through several stages of development.
Stage 1: The Colleague Test
Candidate questions were drafted and reviewed over four iterations within the survey team.
Research team observations:
Stage 2: The Cognitive Test
The entire survey system was usability tested. Participants read the invitation, completed the survey and participated in retrospective interviews using a think-aloud procedure.
Research team observations:
Stage 3: Live Test
Pilot the survey on a small scale to validate the sampling procedure and investigate the response rate. Test-drive analysis procedures.
Research team observations:
Stage 4: Clean Up
Final grammar and format checks.
Creating surveys that meaningfully collect data from the right set of participants requires a keen and thoughtful effort built on an understanding of the subtleties of survey design. The strategies above provide some key insights to issues and concerns of implementation; they do not begin to touch on what it means to write effective survey questions.
Remember that photograph that you stretched and colored and filtered in Photoshop? What did you do with it? As with Photoshop, novices can pull off some trick maneuvers with electronic survey tools. But to develop serious, longitudinal data-collecting tools you need to invest time in understanding the domain, not just accessing tools.
Andrews, D., Nonnecke, B., Preece, J. (2003). Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users, International Journal of Human-Computer Interaction. Volume 16, Issue 2, Page 185-210.
Couper, M. P. (2000). Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly. 64(4), 464-494.
Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method, New York: John Wiley and Sons.
Kehoe, C. M., and Pitkow, J. E. (1996). Surveying the Territory: GVU's Five WWW User Surveys. The World Wide Web Journal 1(3), 77-84.
Yun, G. W. & Trumbo, C.W. (2000). Comparative Response to a Survey Executed by Post, E-mail, & Web Form. Journal of Computer-Mediated Communication 6(1).
Sign up to get our Newsletter delivered straight to your inbox
HFI may use “cookies” or “web beacons” to track how Users use the Website. A cookie is a piece of software that a web server can store on Users’ PCs and use to identify Users should they visit the Website again. Users may adjust their web browser software if they do not wish to accept cookies. To withdraw your consent after accepting a cookie, delete the cookie from your computer.
HFI believes that every User should know how it utilizes the information collected from Users. The Website is not directed at children under 13 years of age, and HFI does not knowingly collect personally identifiable information from children under 13 years of age online. Please note that the Website may contain links to other websites. These linked sites may not be operated or controlled by HFI. HFI is not responsible for the privacy practices of these or any other websites, and you access these websites entirely at your own risk. HFI recommends that you review the privacy practices of any other websites that you choose to visit.
HFI is based, and this website is hosted, in the United States of America. If User is from the European Union or other regions of the world with laws governing data collection and use that may differ from U.S. law and User is registering an account on the Website, visiting the Website, purchasing products or services from HFI or the Website, or otherwise using the Website, please note that any personally identifiable information that User provides to HFI will be transferred to the United States. Any such personally identifiable information provided will be processed and stored in the United States by HFI or a service provider acting on its behalf. By providing your personally identifiable information, User hereby specifically and expressly consents to such transfer and processing and the uses and disclosures set forth herein.
In the course of its business, HFI may perform expert reviews, usability testing, and other consulting work where personal privacy is a concern. HFI believes in the importance of protecting personal information, and may use measures to provide this protection, including, but not limited to, using consent forms for participants or “dummy” test data.
HFI may use personally identifiable information collected through the Website for the specific purposes for which the information was collected, to process purchases and sales of products or services offered via the Website if any, to contact Users regarding products and services offered by HFI, its parent, subsidiary and other related companies in order to otherwise to enhance Users’ experience with HFI. HFI may also use information collected through the Website for research regarding the effectiveness of the Website and the business planning, marketing, advertising and sales efforts of HFI. HFI does not sell any User information under any circumstances.
HFI may disclose personally identifiable information collected from Users to its parent, subsidiary and other related companies to use the information for the purposes outlined above, as necessary to provide the services offered by HFI and to provide the Website itself, and for the specific purposes for which the information was collected. HFI may disclose personally identifiable information at the request of law enforcement or governmental agencies or in response to subpoenas, court orders or other legal process, to establish, protect or exercise HFI’s legal or other rights or to defend against a legal claim or as otherwise required or allowed by law. HFI may disclose personally identifiable information in order to protect the rights, property or safety of a User or any other person. HFI may disclose personally identifiable information to investigate or prevent a violation by User of any contractual or other relationship with HFI or the perpetration of any illegal or harmful activity. HFI may also disclose aggregate, anonymous data based on information collected from Users to investors and potential partners. Finally, HFI may disclose or transfer personally identifiable information collected from Users in connection with or in contemplation of a sale of its assets or business or a merger, consolidation or other reorganization of its business.
If a User includes such User’s personally identifiable information as part of the User posting to the Website, such information may be made available to any parties using the Website. HFI does not edit or otherwise remove such information from User information before it is posted on the Website. If a User does not wish to have such User’s personally identifiable information made available in this manner, such User must remove any such information before posting. HFI is not liable for any damages caused or incurred due to personally identifiable information made available in the foregoing manners. For example, a User posts on an HFI-administered forum would be considered Personal Information as provided by User and subject to the terms of this section.
Information about Users that is maintained on HFI’s systems or those of its service providers is protected using industry standard security measures. However, no security measures are perfect or impenetrable, and HFI cannot guarantee that the information submitted to, maintained on or transmitted from its systems will be completely secure. HFI is not responsible for the circumvention of any privacy settings or security measures relating to the Website by any Users or third parties.
Human Factors International, Inc.
PO Box 2020
410 W Lowe Ave
Fairfield IA 52556