Randomization

Randomization is used to control the over or underrepresentation of certain answers based on their position within a list. For most lists, randomization is the best way to control for the effects of position bias, and there are some slight variations on this programming feature, depending on the format of the answer options.
  • Answer Choice Randomization: Answer choice randomization: This is best for unordered lists, where the options in the list have no direct relationship with each other. The options randomly show in a different order each time the survey is shown.
  • Answer Choice Reversal: This is best for ordered lists, such as numeric ranges and rating scales, because the options in these types of lists are related to each other. Thus, keeping them in the same order helps respondents navigate the list and find the answer they want. In this type of randomization, the order of the list is static, and it is randomly shown in ascending or descending order.
  • Question Randomization: This is best when a survey has multiple similar questions presented consecutively. Randomizing the order of these questions helps control the effects of waning attention spans and survey fatigue can have on respondents as they progress through the survey.

For certain lists, however, randomization would introduce confusion, and would therefore not be appropriate. Generally these are items respondents are accustomed to seeing in a certain order:

  • Alphabetical list (states in the US)
  • Logical list (days of the week)

 

Answer Choice Randomization

This is an example of randomization in a multi-select question. These options are not directly related to each other, and thus they appear in random order. However, you’ll notice that “Other” is fixed at the bottom of this list, which means that it always appears as the last option. This is done because “Other” is used to capture options not the in list, and placing it in a different spot would be confusing for respondents. Thus, it needs to be anchored at the bottom.

Randomization

Answer Choice Reversal

This is an example of random reversal in a single-select question. Since this is a linear scale, randomization would be confusing. Instead, these options are shown only in this order (negative to positive), or in reverse order (positive to negative).

Randomization Order

Question Randomization

This is an example of question randomization. These two questions have the same option lists, and the question structure is similar. If there were three or more of such questions in a row, randomizing their order could help control for the effects of survey fatigue.

Randomization Order