
Qualtrics: Designing Accessible Surveys
When designing and programming Qualtrics surveys, it is critical to consider how every one of your potential respondents will experience the survey. According to Web Content Accessibility Guidelines (WCAG) 2.1, Qualtrics surveys should have content that is:
- Perceivable. All potential respondents must be able to perceive the information being presented. That is, information can't be invisible to all of their senses.
- Operable. All potential respondents must be able to operate the interface. That is, the interface cannot require interaction that a user cannot perform.
- Understandable. All potential respondents must be able to understand both the information provided and the operation of the user interface.
- Robust. Survey content must be robust enough that it can be interpreted reliably by a wide variety of user agents, including assistive technologies.
The accessibility and usability tips below are outlined with these principles in mind. We have tips for accessible survey programming and accessible survey content.
Much of the information below is taken from the Accessible U website and their Digital Accessibility Badging program. We have adapted some of their information for Qualtrics specific information. We highly recommend checking out the Accessible U website.
Tips for Accessible Survey Programming
+
Choosing Appropriate Qualtrics Question Types
- Review Qualtrics' Accessibilty page to see which Qualtrics question types do not meet WCAG 2.0 accessibility guidelines. Of note, while Likert matrix questions are commonly used by researchers, they cannot be read by screen readers.
- If you end up using a question type that does not meet WCAG 2.0 accessibility guidelines, provide an alternative route to complete the survey that does meet WCAG 2.0 accessibility guidelines.
- If your survey participants can fill out the survey on a mobile device, ensure that your Qualtrics questions are compatible with mobile devices. Questions may appear differently on tablets or mobile devices, potentially making them harder to answer or understand than on desktop or laptop screens.
+
Creating Links
- Ensure that your hyperlinks use descriptive text. Descriptive links should provide users with the proper context of where the link will take them.
- Inaccessible example: “Check out this Wikipedia page on cats: https://en.wikipedia.org/wiki/Cat.”
- Inaccessible example: “Click here to find out more about cats.”
- Accessible example: “Check out the Wikipedia page on cats.”
- Ensure that links stay visually distinct from other text.
- No other text should use underlines for emphasis.
- The color of links should be distinct from other colors used for text emphasis.
- Have all hyperlinks open in a new tab. When inserting a link within a Qualtrics question, click on the Target tab and set the target to New Window (_blank).
+
Formatting Text
- Use unambiguous fonts. Within Qualtrics, we recommend Verdana or Tahoma.
- Choose a font size that will be comfortable for all potential survey respondents to read.
- There is no specific ‘correct’ font size. However, a general recommendation is around 16px to 20px depending on typeface. This generally translates to 12pt and 14pt font in the Qualtrics rich content editor.
- Whenever possible, break up large amounts of text.
- Use left-aligned text whenever possible. However, if you are writing in a language that is written from right to left, use right-aligned text.
- Use section headings to organize content in large blocks of text (e.g. consent forms). You can find section headings in a Qualtrics question’s Rich Content Editor.
- Don’t use underlines for emphasizing information in text. Underlines should only be used to convey that text is a hyperlink.
- Don’t use all caps for emphasizing information.
- When creating bulleted lists or numbered lists, use the bulleted list or numbered list functionality within the Qualtrics question’s rich content text editor so that adaptive technology (e.g. screen readers) will read the list correctly. Avoid creating manual bullet points or numbered lists.
- There is a lot more to accessible text than just the above points. See Accessible U’s page on text formatting for some additional tips and tricks!
+
Choosing Color
- Never use color as the only means of conveying information.
- Ensure that you have an adequate contrast ratio between background and foreground colors; this includes text color vs. background color, button color vs. background color, etc. You can review your contrast ratios using the WebAIM contrast checker tool.
- Ensure that your contrast ratio is not too high. Black text on a white background can be stark and fatiguing, especially for those with light sensitivity issues (e.g., Meares-Irlen Syndrome, Scotopic Sensitivity Syndrome). It’s often better to have a dark gray text on a white background. We like gray with the hex code #595959 for text on a white background.
- Use solid backgrounds for text. Text on non-solid backgrounds is harder to read, which means increased cognitive load for your respondents.
+
Making Tables
Ensure that column and row headers are specified within the Qualtrics programming if you have them.
Review Accessible U’s tips on accessible table design for more nuanced details on creating accessible tables.
+
Adding Images
Add alternative text (i.e., alt text) to all survey images. See Accessible U’s tips on creating effective alt text, under the Dos and Don’ts tab.
+
Adding Video and Audio
If you include media in your Qualtrics survey, ensure that the media follows Accessible U’s tips for creating accessible audio and video content.
+
Accessible User Interface Motion
Avoid unexpected, dynamic movement when creating your Qualtrics surveys. Predictable, simple movement is best when choosing how various elements move on a page. For example, never use the ‘Flip’ Page Transition in the Look and Feel menu.
Tips for Accessible Survey Content
+
Ensuring Question Comprehension
- The reading level of your survey should reflect the reading level of those who will be responding to your survey. General surveys of the United States population should aim for no higher than an 8th grade reading level.
- Provide a mechanism (e.g., accessible tooltip, additional description) for identifying specific definitions of words or phrases that are unusual, idioms, or jargon.
- Avoid abbreviations and acronyms unless it is a common abbreviation that all of your potential survey respondents will know.
+
Using reCAPTCHA Appropriately
While Qualtrics' reCAPTCHA question type aligns with WCAG guidelines by being screen reader-friendly, it's important to acknowledge that certain individuals may still struggle with reCAPTCHA tests for reasons unrelated to vision.
Recommendation: Instead of using a Qualtrics reCAPTCHA verification question, you can enable reCAPTCHA within your Survey Options menu. By doing this, Qualtrics will quietly assess participant responses in the background, flagging suspicious responses in an embedded data field. While checking these flags requires more effort for the researcher, it eases participants’ cognitive load and enhances survey accessibility for those who may find reCAPTCHA challenging.
+
Using Appropriate Attention Checks
Attention checks are a tricky subject. While researchers often want to implement attention checks to help them sort ‘bad’ quality data from ‘good’ quality data, many forms of attention checks can have issues related to participant burden and cognitive accessibility. It is important to consider how any given attention check method may affect a given survey’s group of potential respondents, as people with certain demographic characteristics may be more likely to fail attention checks (Anduiza & Galais, 2016).
Because of this, our recommendation is to keep your attention checks covert. That is, survey respondents should not know that the attention check is happening. Examples of covert attention checks include:
- Tracking survey timing - Using Qualtrics' timing question to determine how long a respondent spent on a page. If a respondent spends not much time on a page, it’s an indicator that they were not paying close enough attention to the survey content.
- Response inconsistency - Determine if a respondent’s answer to two or more survey questions are inconsistent with one another. For example, if someone answered the question, “Do you think of yourself as closer to the Republican Party or to the Democratic Party?” with “Closer to Republican“ but then answered “Which of the following best describes your political leanings?“ with “Extremely Liberal," that would be an indicator that the respondent was not paying attention.
Considerations when Designing Overt Attention Checks
Overt attention checks are attention checks that the survey respondent would actually notice while taking the survey.
- Ensure that your attention check measures attention and not other things, like memory or education level. Don’t ask participants to answer a knowledge question they may not know, even if it seems obvious to you. Don’t ask participants to recall something that previously happened in the study, as that is a measurement of memory as well, not just attention. For more information, take a look at these examples of good and bad attention checks.
- Your attention check should have an extremely clear and correct answer. When creating an attention check, think about whether or not there is any room for a participant to interpret your question in a different way than intended.
- Avoid attention checks that are intentionally tricky, as they may confuse your participants at best. Additionally, participants generally dislike them.
- If participants are to be disqualified after failure of an attention check or multiple attention checks, the attention checks should be placed earlier in the study. This is especially true if there is a monetary incentive to complete the survey. This ensures that you don’t waste the participant’s time and they don’t waste your time.
- Consider placing attention checks in the questions themselves instead of in the survey instructions or question instructions. Participants see survey instruction items at different levels of importance, suggesting that respondents may view reading instructions as an option used when further clarification is needed, instead of necessary instruction (Shamon & Berning, 2020).
- Experienced survey takers (e.g., MTurk respondents) may be actively searching for attention checks. Some even have browser extensions that highlight words commonly related to attention checks.
- Passing attention checks, by themselves, is not a sufficient stand-alone method for detecting insincere respondents (e.g., Kennedy et al., 2021). Don’t just rely on a single attention check to determine who is not taking your study seriously. Someone may accidentally fail a single attention check but perform well on other measures of data quality.
- Just introducing attention checks has been shown to inadvertently affect survey results (e.g., Agley, Xiao, Nolan, & Golzarri-Arroyo, 2022) , but findings on this are generally mixed (e.g., Gummer, RoBmann, & Silber, 2018, Abbey & Meloy, 2017). Whether or not overt attention checks affect participant behavior in a survey study likely depends on context.
If you think there’s anything missing on this page related to Qualtrics accessibility, please contact the UMN Qualtrics Brand Administrators at [email protected]. A special thank you to College of Liberal Art's LATIS Research Support Services for their collaboration on creating the content for this page.