Measurement and Evaluation

Lesson 15: Design Research

Introducing design research methods or how to gather evidence to drive design.

Objective

Introduce design research techniques for understanding users and evaluating data visualizations. Focus on practical methods including user interviews, contextual inquiry, the Think-Aloud Protocol, and co-design.

Outline

This lecture provides an introduction to design research methods for learning from users. It starts with theoretical underpinnings before covering four primary research methods, offering practical guidance for each technique and emphasizing the Think-Aloud Protocol as an easy / immediate tool for visualization evaluation.

Why design research

Understanding the user is essential to answering critical design questions.

• What are the right questions to ask within a data visualization?
• What are the most important variables to include?
• How should we prioritize encoding devices?

Note: This is not a comprehensive course on user research methods in human-centered design.

• The goal is to provide a sense of how design research is done and introduce one specific tool useful for visualization measurement and evaluation.
• Employing these techniques may involve ethical considerations and training depending on the circumstance (we focus on quality assurance tasks but some studies may require human subjects research training).

Theoretical underpinnings

Before evaluating applied methods, consider these theoretical perspectives.

• We observe the world through a cultural and experiential lens, including how we interpret visualizations (diagrammatic thinking).
• The question is whether we can, without judgment, learn about the experiences of others (design empathy).
• Design research centers outside and often lived expertise to understand experiences and how symbols (including objects) get meaning within those experiences and social systems (symbolic interactionism).

Design research evolution

According to Jane Fulton Suri (IDEO), various approaches came together into comprehensive methodology.

• In the design of physical products, the goal was to create an object that effectively signaled its function to the user.
• Other designers focused on the meaning people placed on artifacts, rituals, and behaviors and how to incorporate those interactions into their work.
• In Scandinavia, designers began including the end user in the process early on, focusing on co-creation.
• As human-centered design became more defined, these various approaches came together into a more comprehensive methodology.

User interviews

User interviews may be used in learning about a product, population, system, process, etc.

• Leave it open ended and follow them.
• Ask open-ended questions which allow the user to guide you to new topics.
• Who, what, when, where, and why are all helpful, but why, who, and how can be particularly generative.
• Ask about specific and, when possible, recent actual events.
• Example questions:
  • Why are those goals important?
  • Who works with you on those goals?
  • How did those deadlines get set?
  • Can you describe your day yesterday?
  • What did you feel going through that process?
• Interview techniques:
  • Echoing (You said that it was difficult to use, why is that the case?)
  • Silence
  • Interesting redirect (That's interesting thank you. I also wonder if you could tell me about)
  • Advice to someone else (What would you tell someone else who is thinking of using this software in the future?)
  • Advice to past self (If you could give your past self advice, what would you say?)
  • End with magic wand (typically best for last, If you change anything about this, what would it be?)
• Things to avoid:
  • Dead ends (is that your preferred software -> why is that your preferred software)
  • Judgment (why is this tool is difficult to use -> how does it feel to use this tool)
  • Leading questions (what have you done to improve this process -> has the process changed over time)
  • Double-barrel questions (what do you like and dislike about this tool -> what do you like about this tool? what do you dislike about this tool?)

Contextual inquiry

Contextual inquiry is similar to a user interview but conducted in action.

• The goal is to observe a specific situated experience.
• The primary goal is to observe and ask non-disruptive questions.
• Best when conducted in the course of regular activity.
• Approach with a beginner's mind and without judgment.

Think-Aloud Protocol

The Think-Aloud Protocol is one of the easiest methods for observing how users interact with visualizations.

• Provide users with a simple prompt: "Please go through this tutorial and while doing so, please try your best to think aloud."
• Users verbalize their thought process as they interact with the visualization.
• Reveals what they notice first, what confuses them, what assumptions they make, where they get stuck, and what they find intuitive or difficult.
• Clayton Lewis and John Riemans
• See also MILC: long term diary study (Ben Shneiderman and Catherine Plaisant)

Co-design

Co-design breaks down the distinction between users and designers.

• Instead of treating people as users, invite them in as partners.
• Allow others to fundamentally guide the process in conceptualization and design.
• Often works well when you can make a long-term repeated relationship.
• Can be effective for more involved feedback, especially for expert tools.

Take Aways

Design research provides essential methods for understanding users and improving data visualizations through their feedback and participation.

• Understanding the user is essential for making critical design decisions about visualizations.
• User interviews use open-ended questions to learn about products, populations, systems, and processes.
• Contextual inquiry observes users in action during regular activity.
• The Think-Aloud Protocol is an accessible method for visualization evaluation where users verbalize their thought process.
• MILC / diary studies can provide long term view.
• Co-design invites users as partners to fundamentally guide the design process.
• These techniques require approaching users with empathy and without judgment.

Citations

[1] "The Deep Dive: One Company's Secret Weapon for Innovation," ABC News, 1999.

[2] "The Art of the Insight: Learnings from 30 Years of Curiosity and Empathy," IDEO U, 2019. Available: https://www.ideou.com/blogs/inspiration/the-art-of-the-insight-learnings-from-30-years-of-curiosity-and-empathy

[3] C. Nickerson, "Symbolic Interactionism Theory & Examples," Simply Psychology, 2025. Available: https://www.simplypsychology.org/symbolic-interaction-theory.html

[4] J. Tong, "Diagrammatic thinking and audience reading of COVID-19 data visualisations: A UK case study," Convergence, 2024. doi: 10.1177/13548565241309886.

[5] B. Hanington and B. Martin, "Universal Methods of Design," Rockport Publishers, 2012.

[6] "User Interview Example Questions," Yale University. Available: https://usability.yale.edu/understanding-your-user/user-interviews/user-interview-example-questions

[7] J. Gallegos, "50 Powerful User Interview Questions You Should Consider Asking," Respondent, 2022. Available: https://blog.respondent.io/best-user-interview-question

[8] B. Shneiderman and C. Plaisant, "Strategies for Evaluating Information Visualization Tools: Multi-dimensional In-depth Long-term Case Studies," BELIV '06, 2006. doi: 10.1145/1168149.1168158

[9] C. Lewis and J. Rieman, "Task-Centered User Interface Design: A Practical Introduction," HCI Bibliography / IBM Research, 1993. Available: https://www.hcibib.org/tcuid/

[10] C. Lewis, "Clayton Lewis." Available: https://sites.google.com/view/clayton-lewis

[11] "What is Design Thinking? An Overview," AJ&Smart, 2020. Available: https://www.youtube.com/watch?v=gHGN6hs2gZY

License

This lesson is part of Interactive Data Science and Visualization and is released under a CC-BY-NC 4.0 license.