Creating a Valuable Survey: Mistakes to Avoid

Surveys are an efficient and low-cost method for collecting large samples of consumer data.  Though creating a survey may appear to be pretty straightforward, there are countless aspects of the design process that can affect the validity of your results.  The overarching goal of a survey is to collect useful data that allows one to draw conclusions, leading towards a better understanding of the customer.  If your survey is faulty, your data will also be faulty.  A good survey asks questions that are clear, direct, unbiased, and unassuming.

These are a few common mistakes that should be avoided in order to retrieve accurate data from your survey.

Wording Bias

The way in which you word your questions can change the responses drastically.  Each respondent has their own individual experiences and perceptions that may affect their interpretation of the question, leaving room for misunderstandings and unaccounted variations in responses.  That being said, the questions should be as clear, and specific as possible.

The Loaded Question

A loaded question is formed on a specific presupposition that does not allow for the respondent to answer without agreeing with the implication.

Example: Does the presence of cameras decrease your willingness to shoplift?

This question relies on the assumption that the respondent has an inherent desire to shoplift, cameras being the only deterrent that would affect their decision. Whether the respondent replies yes or no, their answer would still imply a temptation to shoplift when, in reality, some people would never consider shoplifting in any circumstance. A way to avoid a loaded question is to first inquire about the initial presumption:

Example: Have you ever been tempted to shoplift?

In the case that the respondent chooses no, you can use skip logic to bypass the follow-up question that asks about the presence of cameras.

Loaded questions can lead to frustrated respondents, resulting in a higher drop-off rates.

Leading question

A leading question is phrased in such a way that implies a ‘correct’ answer, pushing the respondent towards choosing a response that may not accurately reflect their true opinion or behavior.

Example: Would you pay more for a paper bag versus a plastic bag to reduce environmental pollution?

By adding a factor of morality, you force the respondent’s character into question.  If you are using this data to predict the financial outcome of switching to paper bags, your data will not accurately predict how customers will actually behave. Realistically, while people may care about the environment, other factors could still lead them to choose plastic, such as cost or durability.  A better question that would produce more accurate results is:

Would you pay more for a paper bag versus a plastic bag?

While your initiative may be to reduce environmental pollution, survey responses alone cannot achieve it.  Success requires action, which will not be reflected in skewed data.

Double-Barreled Questions

A double-barreled question forces a respondent to answer two questions at once.  Respondents cannot agree with one statement while disagreeing with the other.

Example: Were you satisfied with the product and shipping process?

While it may be tempting to combine the two in order to shorten the length of the survey, it results in a lost opportunity to gain insight about either aspect.  Instead, ask two separate questions.

Ordering

The order in which you present your questions can have a large effect on your overall results.  Previous questions often affect the responses for following questions.

For example, you pose the question: Have you had a negative experience with our brand? 

Then directly after, you ask: On a scale of 1-10, how likely are you to recommend our brand to a friend or family member?

By asking the first question, you force the customer to consider their negative experiences while responding to the latter.  The survey creator may view each question as an isolated event, but the respondent is processing the information sequentially as they move through the survey.  Test the ordering of your survey with a blind trial using an employee or willing volunteer.  It is difficult to remove yourself from the objective of the survey, so handing it off to fresh eyes can reveal details that are missed by the creator.

Other Factors to Consider

Consistency

Consistency is key when looking to compare results over time or across segments.  When you change the phrasing of the questions, it is impossible to determine whether a discrepancy is due to a legit reason or a change in wording, ordering, or phrasing.  Make sure to maintain the same cadence throughout all instances to ensure accurate comparisons.

Length

The length of the survey has an effect on the rate of response.  SurveyMonkey research looked at the drop-off rate for each additional question for 100,000 random surveys varying in length from 1-50 questions.  The results revealed a clear trend of increased drop-off rates for each additional question. [1]  In general, the average attention span is becoming shorter and shorter as time goes on.  The key is to create a concise survey that includes only the most pertinent questions to the objective.

Keeping these aspects in mind while creating your survey will help steer you down the right path, ensuring that each question is focused and unbiased.  The main goal of conducting a survey is to gather useful data, so avoid making mistakes that jeopardize the validity of the results.

[1] https://www.surveymonkey.com/curiosity/survey_questions_and_completion_rates/


Second To None empowers customer-centric brands to deliver consistent, intentional and authentic consumer experiences.

We adeptly design and manage mystery shopping, compliance, engagement and voice of customer solutions grounded in strategic relevance, program integrity and actionable insights. Our solutions are developed on the basis of solid research and statistical science. We achieve success through a relentless focus on quality and innovation, consultative relationships and a talented team of professional associates.

 

Start typing and press Enter to search