How to make a GOD (Goal-Oriented Design) Survey

ResearchDesk™ allows easy and cost-effective access to a high-quality mobile panel in the US, but the value in the results is as much about the quality of the survey as the quality of the panel.

When creating a survey, the hardest part is working out what to ask. There are times when you already have a previous survey, and times when you just brainstorm questions, but the best approach to a pithy survey that achieves your goals is to start with the goals.

This is GOD: “Goal-Orientated Design”.

What is Goal-Orientated Design (GOD)?

Surveys created using GOD start with the question: “What are the killer charts that will absolutely tell my story?”. You then design your survey specifically to populate the data needed to make those charts.

GOD front-loads the work of the final deliverable. In fact, writing any questions at all is one of the last things you do.
The process has the following steps:

  1. Decide what you want to know
  2. Create detailed mock-ups of the charts that will tell you what you want to know
  3. If needed, get consensus from all stakeholders of the study
  4. Create a list of every piece of information that you need to know
  5. Write the questions that provide all the information that you need to know
  6. Code the questions into a survey platform
  7. Estimate roughly how the responses will fall (assuming you have some idea of this), and determine how many completes you’ll need
  8. Set up a ResearchDesk™ study to get the responses
  9. Create the report for stakeholders

Why write surveys with GOD?

Benefits of GOD approach to writing surveys

Surveys written with GOD are pithier and deliver more compelling results which serve business objectives. Because they tend to be shorter, and nicer to take, drop out rates are lower and respondents concentrate more, giving higher quality answers.

Sadly, most surveys are not Goal-Orientated. It is more usual to start with the questions, run a survey, and then spend weeks trying to make sense of it. These sorts of surveys come when the questions are thought of as “What sort of things might be interesting?”, and then everything that might be interesting is thrown together into a survey.

These question-orientated surveys tend to be longer (often significantly), and don’t hang together very well. Drop out rates are high, and respondent concentration is diminished causing lower quality answers. There is always a temptation to use questions you’ve used before, which might be inconsistent, out-of-date, or simply not appropriate for a mobile platform.

Embee Mobile does not recommend re-running a survey that has previously been fielded through another method to continue a longitudinal study (e.g. a tracker), because results cannot be meaningfully compared between two surveys where the methodology was different.

When migrating a long-running survey to ResearchDesk™, you should take the opportunity to refresh the questionnaire using the GOD approach, and then run a few waves using both methodologies. It’s likely there will be differences in the results, and it’s not a foregone conclusion that the old methodology is more accurate than the ResearchDesk™ methodology, just because it’s older.

GOD: Step-by-Step

Step 1: Decide what you want to know

Although this sounds trivial, it is often the hardest step. This is especially true when you started with the idea of running a survey, before the problem you are trying to solve has been defined.

The emphasis here is to get your stakeholders to think about the following questions, in this order:

“What decisions do I need to make?”

“What information do I need to make that decision?”.

Step 2: Create detailed mock-ups of the charts that will tell you what you want to know

For each bit of information you need to know, create using data that looks like what you’d expect, a picture of the killer chart that would absolutely answer the question.

For example, you might have in your mind a killer chart to show where people spend their money by retail outlet, distributed by the outlet they consider their primary retail outlet:

You might want to actually do all or most of the formatting of the graphic already, assuming that the graphic is being created dynamically from data (see step 9).

In the above example case, the chart was created in Excel, and the data required to make it was:

Step 3: If needed, get consensus from all stakeholders of the study

Once you have created all the charts that will answer all the business objectives, you have a mock-up of the results you plan to deliver.

(Yes – you have actually already written the report! All you’ll need to do in Step 9 is change the data!)

Why not send it round to stakeholders… they’ll be able to approve or suggest edits so much faster if they can see it.

Step 4: Create a list of every piece of information that you need to know

So, you now have all the charts you need to create, and an idea of what data is going to be needed to make each chart.

We’re about to start to think about the survey.

For each chart, you can now list all the information that is needed. There will likely be some overlap between charts, but that’s OK.

In the above chart, we need the following information:

  • Age and Gender (because we’re creating this chart multiple times for each age/gender combination, as you can see in the title)
  • Which is their preferred bricks and mortar retail location
  • How much they spend in each of the stores

Don’t forgot that if a chart is only for a subset of respondents, then the data to define that subset also needs to be determined.

Step 5: Write the questions that provide all the information that you need to know

Now that you have a list of the information that you need, you can write the questions. Age and Gender are already provided by ResearchDesk™, so we don’t need to ask that, but the following questions would get the preferred bricks and mortar location, and their spend.

  1. Which of these shops do you shop in? [list, incl. none of the above]
  2. if not checked none of the above, or only one outlet: Which of these shops do you consider your primary retail outlet? [list of those they checked]
  3. if not checked none of the above: How much do you spend per month in [list of those they shop in, as a rotational group]? [Answer options could be ranges, slider or text box].

You may note that Q1 is not actually providing any information that we need. When writing a survey for a mobile, it is always better to have multiple short questions rather than fewer longer questions. Q1 is designed to make the survey nicer to take, by limiting unnecessary options made available in Q2 and Q3.

Step 6: Code the questions into a survey platform

At this stage, you have all your questions, together with some of the logic that you’ll do.

You don’t need the questions to be in the same order as you thought of them. The only requirements for question ordering is to ensure that any logic required for a question is based on questions that come before it. In general, you should start with screener questions, then a few easy “warm-up” questions, and finish with the harder questions, but the prevailing rule is that the entire survey feels like a journey and not a collection of pieces.

The next step is to open up your survey platform, and code the questions, and the logic in. Most survey platforms will allow you to copy-paste from other applications.

Be sure to check every question satisfies all the Golden Rules.

As always, you need to test your survey thoroughly at this stage, using a mobile device. (Note – preview for mobile surveys on some platforms isn’t always accurate, so as a final check, always check using an real small Android phone).

Step 7: Estimate roughly how the responses will fall (assuming you have some idea of this), and determine how many completes you’ll need

For this, you’ll need to decide on a confidence interval and an acceptable margin of error.

Calculating the number of completes is explained in a previous blog. This provides an explanation and a calculator to help you.

Step 8: Set up a ResearchDesk™ study to get the responses

The last step is simply to set this up in ResearchDesk™, click Launch and wait for the results to come in.

Step 9: Create the report for stakeholders

Step 9 has almost already been done in Step 2. Extract the data from the survey, and plug it into the data that creates your charts, assuming that the charts are dynamically linked to the data.

Add conclusions and titles – something you couldn’t do before you had data, and if you are adding in a statistical comment, then add that in too. (Statistical comments are often written as a small footnote on a slide, which states the total number of panelists asked, sometimes the text of the question, sometimes the margin of error, and if not all panelists, a description of which panelists have been asked).

Now you can deliver the report, maybe on the same day that the survey fielding closed!