http://www.uxmatters.com/MT/archives/000183.php

When Observing Users Is Not Enough: 10 Guidelines for Getting More Out of Users’ Verbal Comments

By Isabelle Peyrichoux

Published: April 9, 2007

One of the principles underlying usability testing is that observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task. By observing users, you can assess whether they are actually able to use a product. By asking them, you simply cannot.

However, as you try to derive valid conclusions about how to design a user interface, relying only on—or even mostly on—observation can be

• misleading—because often user behaviors that you observe can have many different interpretations. For example, if a user did not click a link, perhaps the user did not see the link or did not understand it. You cannot know the reason with certainty without asking the user. Your assumptions might be biased.
• limiting—because you lose the opportunity to gather valuable verbal data by relying only on observational data.

While some usability professionals might claim that you cannot rely on what users say—and there are some risks in relying on users’ comments—there are means of avoiding or minimizing those risks. To understand these means, we must leave the realm of objective science and enter the realm of human relationships and empathy.

A user interview—including one that occurs during usability testing or user observation—is a relationship between two people—the interviewer and the interviewee—in which emotions, fears, and judgments come into play. Thus, my training and practice in psychotherapy have greatly enriched my technique in doing user interviews, because they have helped me avoid or minimize certain biases when eliciting and interpreting users’ verbal comments.

To help you get more out of users’ verbal comments, this article will provide ten guidelines and various interviewing techniques I’ve learned from experience. These techniques work best if they are used with genuine empathy for users. If users feel that you are not genuine—even if you are not aware of it or try to hide it—these techniques won’t work. I’ve described most of these techniques within the context of usability testing, but some techniques are also applicable to other user research activities—such as field studies and task analyses—and to stakeholder interviews.

1. Be aware of your own judgments and projections.

It’s easy to say you’re not judgmental, but it’s not so easy to achieve that in reality. And if you want your interventions to be effective and users to be comfortable speaking as freely and honestly as possible, you must actually be nonjudgmental. It is useless for you to say there are no good or bad answers to your questions if your behavior says otherwise.

So, for example, be careful about saying “Excellent” or “Good” or any word that implies a positive judgment following a user’s answer. Saying an answer is excellent might imply that a user’s answer could be good or bad and that you are judging the user’s performance. Instead, depending on the context, you might say something like “Got it” or “I understand.”

In user interviews, as in all relationships, you’ll meet all kinds of people—both people with whom you feel comfortable and those you don’t, whom you might tend to judge. Unless you are careful, you might let your first reaction to a person color an entire user interview. It’s natural for you to be uncomfortable with the personalities of some users, but you must be conscious of your feelings and overcome them if you want to get the most out of an interview.

Observe your feelings about each user. Take note of any fear you feel or judgments you make. We often negatively judge others because they remind us of aspects of our own personalities that we do not accept. This phenomenon is called projection, according to Jung’s shadow theory. Observing your own feelings will help you to become less judgmental, which will, in turn, make users feel more comfortable with you, letting them speak more openly. This will enable you to get more useful information from your user interviews.

We all tend to judge others. It’s human. By becoming aware of and taking responsibility for your judgments about users and the feelings that you project onto them, you can go beyond these and become more empathic.

Of all the guidelines I’ve given in this article, this one is actually the most difficult to apply. Doing so requires self-observation and a willingness to overcome your biases and defences. However, being nonjudgmental has a huge positive impact on your relationships with users.

2. Be genuine and transparent.

The more your behavior aligns with your words, the more users will feel comfortable with you. Being truly transparent about your interview process or anything unusual that happens during an interview helps build users’ confidence in their relationship with you. If you are genuine and open, it will encourage users to be the same with you. Don’t pretend that everything is okay when users can sense that something is not. Any disconnect between what you say and what you do will make users feel insecure, and they’ll be less open with you. Here are a couple of scenarios to show you how this works.

Scenario 1

Problem: A user tells you something, but you were distracted or were thinking of something else and lost some important information that you need.

Solution: Let the user know that you were mentally absent. Say “I missed what you said. Would you please repeat it?”

Scenario 2

Problem: You want to follow a specific process during the interview or need to move quickly from one question to another and want only a user’s first impressions.

Solution: Let the user know before you start that you will move very quickly from one question to another.

3. Adapt to each user. Do not ask users to adapt to you.

It is easy to fall unconsciously into the trap of expecting a user to adapt to your way of communicating rather than trying to adapt to the user’s.

After a usability test session, you might find yourself saying, “Oh, this person wasn’t a good test subject.” He was too something—perhaps too shy or too talkative. It’s possible that the comments a particular user made were not very helpful—no matter how hard you tried to get valuable information from him. However, to make the most of each user interview, you must ensure that you are doing your best to adapt to the user’s rhythm and personality. Otherwise, you risk losing important data.

This example reflects the differences between introverts and extraverts, as defined by Jung’s psychological types and the Myers-Briggs Type Indicator (MBTI). Extraverts usually tend to think and speak at the same time, whereas introverts usually tend to speak only once they have thought through what they want to say. You should give people enough time to think before answering your questions—especially introverts.

It also shows how easily we can misinterpret users’ behavior. You must stay objective. If a user is not talking and there is an extended silence, don’t assume you know the reason for the user’s silence. Instead, observe how quickly he answers your first few questions and adapt to his rhythm. If he takes some time before answering, but gives detailed and thoughtful answers, be sure to give him enough time to answer your questions.

This example illustrates how different people can be and how important it is to be aware of their differences to make the most of user interviews. Learning about Jung’s psychological types can help you become aware of the diversity of personality types and how they can affect your relationships with users. This understanding will also help you to be less judgmental when confronted with a user whose personality is very different from yours.

4. Be conscious of the way users are interacting with you.

Even though you’ve carefully explained to users that they are not being tested, you’ll often encounter users who feel they are being tested and are afraid of giving a wrong answer. If a person is nervous throughout a test session, even though you’re being empathetic and nonjudgmental, it is useless to try and change his or her feelings. Regardless of how hard you try, it won’t change anything. Even worse, a user might become irritated by your mothering behavior.

Observe carefully how users interact with you, and take these observations into account when interpreting your findings.

5. Get users to speak about their own experiences.

In nearly all usability test sessions, at some point, you’ll hear a user say something like one of these remarks:

• “For me, it’s okay, but the average person might find it difficult.”
•  “For my mother, it would be hard.”
•  “Older people would have difficulty with it.”
• “For someone who is looking for something like that, it’s good.”

It is very common for users to speak for someone else during a test session. It often happens when users feel uncomfortable stating their own point of view. For example, they might fear being judged or want to please the interviewer. It places users in a less compromising position to speak for someone else rather than to speak for themselves and say what they really think—for example, “I find it very difficult,” “I think it’s really bad,” or “It’s useless to me.” This is something people do unconsciously every day, but do not let yourself be fooled by this. Users really know only their own experiences, abilities, and opinions. Gathering information about what users think the user experience would be for other people has no value.

To make sure users speak from their own points of view, don’t reformulate what the user said about a product’s user experience for other people. Instead, just restate the part of the user’s answer that represents his own opinion. When you do this, users will stop talking about other people’s opinions and speak for themselves—for example:

User: For me, it’s okay, but for the average person, it might be difficult.

Interviewer: For you, it’s okay.

User: Yes. It’s okay, because….

Alternatively, you can ask a question about a user’s opinion like this:

User: For my mother, it would be hard.

Interviewer: And what about for you? What do you think?

These examples show ways you can smoothly get a user to come back to his own opinions. If you do this with genuine empathy, the user will feel comfortable speaking more freely and honestly about himself and his personal opinions. Doing this acknowledges the user’s true opinion, indicates that his opinion is important to you, and shows that you are not judging him. Reformulating a user’s answers conveys empathy and acceptance.

While this generally works very well, in the rare case that a user keeps talking about other people’s viewpoints, do not push too hard and insist that the user talk about his own opinions. Otherwise, the user may become defensive.

6. Notice when users are censoring their own comments.

You’ll often see users self-censoring their opinions. This often happens when users fear their opinions are too critical. For example, at the end of an interview, you might ask a user about his general impressions of your Web site. Perhaps the first words that come to his mind are “very complicated,” but he hesitates to express this negative judgment, fearing he might offend you. So, he tones down his original thought and says, “very complicated, but when you get used to it, it’s okay,” or “but for people who know the field, it might be easy.” In some cases, users really have mixed opinions about a product, but in other cases, they are just trying to be nice. If you have carefully observed a user’s behavior throughout a test session, you can probably judge whether the user will try to please you by self-censoring his real impressions or really has mixed impressions.

To ensure you capture a user’s real opinion, reflect back the user’s initial opinion like this:

Interviewer: What are your impressions of this Web site?

User: Oh, it’s very complicated, but I guess, for people who know the field, it’s okay. Yes, I think it’s okay.

Interviewer: You said it was very complicated.

User: Yes, it’s very complicated because….

7. Get users to speak in terms of problems, not solutions.

Often, during usability testing, users offer solutions to problems. For example, after failing to find a link on a Web page, a user might say, “I did not see that link. It should be in bold, or it should be bigger.”

The user is not a designer, so the solution the user suggests—that the link should be in bold—might not actually work. What will help you find the right solution is to investigate why the user did not see the link. So, if you can, get the user to tell you why he couldn’t see the link. Sometimes, the user won’t know, so don’t push too hard, but he might give you very interesting information that will help you identify why he didn’t see the link and, ultimately, help you find a solution. For example, he might say, “I was concentrating on another part of the screen and didn’t notice there were links in this area,” or “I thought it was just text.”

And you can help the user to provide more precise information by asking follow-up questions that are appropriate to the context—like this one, “Were you expecting to find the link on another part of the screen?” Each piece of information you glean will help you better understand the reason why the user did not see the link and help you find a solution to the problem that you identify. Only when you have accurately identified the problem can you come up with the right solution.

Here are two examples of how you can help a user to clarify a problem:

User: This label isn’t right.

Interviewer: Why isn’t it right?

Don’t initially ask, “What would be a better label?” That would be asking the user to solve rather than identify the problem.

Once you understand the problem, you can ask follow-up questions that are appropriate to the context—like “What were you expecting?” or “Did you have a word in mind?”

User: “This page is dull. I don’t like it much.”

Interviewer: “Why you don’t like it?”

Don’t ask, “How would you improve it?”

It’s actually easier for users to first explore a problem rather than thinking right away about a solution. Plus, you’ll avoid losing important data about the problem, which in the end will help you to devise the right solution. Though, once you and a user have explored a problem together, the user might come up with a very good solution.

8. Ask “Why?” and dig deeper.

When interviewing a user during usability testing, asking “Why?” and exploring users’ statements in depth is essential. If you don’t dig deeply enough in trying to understand a user’s point of view, you won’t get enough information to make the proper recommendations to improve a user interface. Statements like the following won’t provide sufficient information to your product team:

“Participants preferred the previous version of the Web site.”

“Participants did not understand the label.”

“Participants did not click the link.”

You must understand and explain why. Without your providing the reasons behind such statements, it will be hard for designers to know how to improve the design of a product’s user interface. To come up with a good design solution, they must have an in-depth understanding of the problem they are trying to solve. Thus, when interviewing users during usability testing, always keep in mind what you want to do with the findings and ensure that you gather all necessary pieces of information to help you reach your  goal—generally, helping your team to redesign a user interface.

This guideline pertains to many user research activities. For example, Indi Young points out how important it is to ask “Why?” when doing a task analysis and to “dig into the background of a topic until the interview participant has no more to say about it, or takes you on another tangent.” For a task analysis, the ultimate goal of user interviews is to clearly identify users’ tasks and build a complete mental model of their work. To succeed, you must keep your final goal in mind during the interviews.

Do not be afraid of digging too deeply or getting into too much detail. You are better off having too much detail than having an incomplete explanation of a problem when redesigning a user interface. Sometimes, when first interviewing users, it’s hard to know what specific pieces of information you need. You’ll learn what to explore by trial and error. If you find some of the details you’ve gathered aren’t relevant, you can avoid exploring them further in your next interviews. 

9. Make objective and precise observations.

During my training in the creative nondirective approach to psychotherapy, I learned something that helps me a lot in usability testing: objective and precise observation. This is a simple, but very powerful tool for avoiding misinterpretations of user behaviors and getting users to talk to you.

For example, if a user is looking at a part of the screen without doing anything, don’t interpret what the user is experiencing by saying, “You are hesitating.” You can’t really judge whether the user is hesitating. Instead, as a result of objective and precise observation, say, “I notice that you have been looking at this part of the screen for a while.” If you make an objective observation, the user will generally explain what he was thinking.

If a user smiles when looking at a Web page, but does not speak, you might wonder why he is smiling. A smile can have many different meanings, but there is no way to know the exact reason why a user is smiling without asking. If you don’t ask, you won’t learn why and might lose an interesting bit of information, so try this:

Interviewer: You are smiling.

User: Yes, because I like the image on the page.

This technique can help with any user behavior that you observe and want to understand better—whether silence, nonverbal expressions, or a user’s pattern of navigation through a user interface. It provides a lot of rich information you would not have without asking the user, and if you don’t ask, you risk misinterpreting the user’s behavior.

10. Allow users to be spontaneous and follow their flow.

In usability testing, the more spontaneous a user’s answers are, the more reliable they are. Here are a few techniques for getting more spontaneous responses from users:

Let users talk without interruption unless they go outside the scope of a usability test. Also, let users remain silent or pause for a while if they need time to think.

This is often hard to do, because you might become impatient or have difficulty bearing the silence, but you should avoid interrupting a user’s thought process. An introverted user might still be composing what she wants to say in her mind. If you interrupt, you might lose some very interesting information the user was about to tell you.

For example, if a user is scanning a page of search results, and still in the process of thinking about them, starts saying, “Ah, the search results are highlighted…,” you should not interrupt the user by asking, “What is it?” Instead, give the user time to gather her thoughts.

Always go along with a user’s flow—regardless of the sequence of questions you’ve planned for a user interview.

For example, perhaps a user starts talking about a topic you intended to address at the end of your interview. While much depends on the particular situation, I generally recommend letting users talk rather than telling them you’d prefer to go back to some point later on. If a user spontaneously raises a point you wanted to know about, it is golden.

Let users speak about their spontaneous reactions rather than asking them questions right away.

For example, once a user lands on a Web page, first wait a bit for his spontaneous comments. Don’t immediately start asking the user questions.

If you do inadvertently interrupt a user, try returning to the user’s spontaneous comments.

Fortunately, if you miss something a user says or cut a user off, it’s usually possible to go back to what the user was saying. Even when you’re careful, it’s all too easy to cut off a user’s remarks. To help get a user back on track, you might say, “A moment ago, you were saying…” and repeat the words the user was saying when you interrupted him.The user will generally go back to his previous situation and explain it to you as though it has just happened.

This technique also works if a test session is interrupted for any reason—for example, if a computer breaks down or someone comes into the room—and you want to return to what the user was saying before the interruption.

Conclusion

A usability test implies, among other things, a relationship between two people—an interviewer and a user. The way an interviewer interacts with users influences the outcome of test sessions greatly. Drawing conclusions from only observation is risky. You must elicit verbal comments from users in a way that enriches your observations and helps you avoid biases. To make the most of your user interviews, convey confidence and empathy, adapt to users’ personalities and rhythms, get users to talk about their own experiences and the reasons behind their comments, explore users’ comments in depth, and follow users’ flow.

When doing eyetracking studies, you should always elicit verbal comments to ensure that you interpret users’ behaviors correctly. For example, a hot spot on a word might have different explanations—such as interest, confusion, or surprise.

However, relying too much on users’ verbal comments can be just as risky as relying too much on observational data. For example, a user might say he likes a Web site after failing all the tasks during a test session. A successful usability test session results from the right combination of observation and verbal comments. Observational and verbal data are more reliable in combination than when used separately.