7 details to keep in mind for insightful user tests


 

Many books and blog posts have been written on how to conduct user tests, spelling out Dos and Don’ts, best practices and learnings. But the truth is that with every user test you conduct, there are new aspects you learn about the process of testing. Some things might totally fail, other aspects might work for one test but not for the other and sometimes new testing environments themselves open new perspectives on how to improve user tests.

By sharing our learnings, we hope to further expand on the topic and give some practical tips to help you get the most out of the time your users spend with you. The following insights are from various stages of different projects, starting from weekly user tests with paper prototypes to apps in alpha development stages. Although priorities and tasks change with every test depending on the product development stage, we think the following points are applicable to the majority of user testing scenarios.

 
 
 

#1 RECRUIT SUITABLE USERS

Depending on what you want to test it is worth to spend more time and effort on your user recruiting process. Especially for acceptance tests it is beneficial to recruit people who are likely to use your final service or product. A few simple examples:
If you want to test an app which aims to be used by car drivers in their own car, recruit only people who have a car and frequently use it. Otherwise you get answers like: „This is a great service, I would totally use it if I had a car.“ Also if your service is higher-priced it makes sense to focus on recruiting people who could afford to use it.

For usability tests, recruit people who are more likely to deliver valuable insights, e.g. if you are testing an Android app get hold of Android users. Otherwise users will struggle with basics platform specific interaction patterns that should be taken for granted for an effective testing environment.

Recruiting your candidates with care helps to get the most out of testing sessions. Don’t take the easy way out but think hard about prior experiences that could help your test users to better fit into the testing scenario.

 
 
 

#2 MOBILE DEVICE CHECKLIST

We all know we should have our test devices ready, have them fully charged, have a substitute device… etc. But I still had some moments sometimes, I call them ‘Oh…wait a minute…’ moments that are easily avoidable if you think about them before. To give you an idea:

‘Oh, wait a minute… I need to switch off the GPS’
‘Oh, wait a minute… I need to change the language of the phone from English to German’
‘Oh, wait a minute… I have to reload the prototype to update to the latest version.’
‘Oh, wait a minute… I have to disable the screen lock and display sleep timer.’

No matter how trivial these sound, I have to admit I had such moments and it’s okay to make these mistakes if you learn from them. So don’t forget to check all the details that could affect the features of the service or tasks you give to your users. If you want to be safe prepare a checklist. For me the following points are the most important settings:  
Language, time and date settings, choosing the appropriate keypad buttons (such as Return, Go, Next, Search, Send, Done), default settings, display sleep timer and screen lock.

 
 
 

#3 INTRODUCTORY TALK

Introductory talk often helps to create a comfortable environment for users to respond to tasks. In order to have a successful user test it is crucial for test users to understand that the product or service being tested is not a finished product. Once my colleague Eva tested early product wireframes for a service. During the test one user said ‘It’s a little sad that the whole interface is all black and white’. Another time we wanted to test the flow of a mobile app we planned to built. We prepared a click dummy with images but no functionality. We ended up repeatedly saying ‘No, you can not click there. It’s just a prototype.’ The examples demonstrate how the lack of information can have very negative impacts on the overall testing process. Before starting to test, prepare and describe:

  1. What is the aim for the particular user test
  2. What is being tested (UI, flow, etc)
  3. The level of functionality of the prototype

Try to prepare test users as best as possible to reduce time wasting clarifications during the test.

 
 
 
 

#4 TALKING ALOUD VS THINKING ALOUD

During every user test, I ask test users to think out loud to better understand their thought process. It seemed to be working fine, until the day I read about the difference between thinking out loud and talking aloud. This article on UX matters talks about how to make sure that users tell how they reach their conclusions instead of simply stating them, using positive reinforcement and no feedback at all (called extinction) for undesired feedback. For example:

User: After filling the form, I will click on the ‘next’ button. But the button is not in focus. I would expect the button to be above the keypad.
Me: Okay. That’s good to know. You would want the button to be more visible.
User: The blue and the green don’t seem to be from the same colour palette. It just doesn’t seem to belong to the brand.
Tester: Okay, the next task is…

No reinforcement on opinion based answers subconsciously separates useful and inappropriate information and focuses on the kind of information you as tester want to receive from users. A round of practise on thinking out loud before the test starts, can be helpful to make users understand what testers look for and regard as useful information.

 

#5 ASK THE RIGHT QUESTIONS

It’s been suggested a million times already, but I must emphasise it again. Steer clear from flattery and compliments and avoid approval seeking. Flattery based conversations and feedback leads us nowhere. Rob Fitzpatrick’s book The Mom Test makes clear how compliments, fluff and ideas sometimes give us false negatives (making us think the idea is dead when it’s not) and more dangerously- false positives (convincing yourself you’re right when you are not).
Ask questions that help you to find out the importance of a problem for test users. What are their efforts to solve it, and if no efforts are made, why not. Asking questions based on past experiences of users helps to better understand whether the tested product or service is likely to be used and in which contexts. Again think about recruiting the right users to know about past experiences that might be relevant for the user test you are conducting.

A rule of thumb when evaluating user feedback – Anything involving the future is an over-optimistic lie. For more, have a look at ‘The Mom Test’.
 
 
 

#6 BE PRECISE AND PERSONALISE THE EXPERIENCE

Placeholders like ‘Lorem Ipsum’ and arbitrary information on wireframes or app prototypes are an avoidable source of confusion for users. It also makes the service temporarily redundant for the user and dents the experience as a whole.Two weeks back I was testing an app, which allows to book movie shows via smartphone. When I showed one user the ‘purchased tickets’ functionality the conversation turned like this:

User: So, in my purchased tickets, it shows I have booked a movie ticket for October. I would never do that. I would never book movie tickets so far in the future. I think this is over designed and solves no problem of mine really.
Me: Oh. Yes, you can ignore the date. It’s just a placeholder.

A user test can yield meaningful insights only when test users are in a realistically simulated context. Often contextually meaningful information is unavailable during the design process. However, it’s the designer’s responsibility to put the user in the appropriate context. This includes personalizing the process. For example, a dummy email text could already include the name of the tester to make the process more realistic and personal. For further reading, check this article.

 
 
 
 

#7 TEST HYPOTHESES AROUND THE SERVICE

Often the product you develop is not the only touch point of a service. If you can, leverage user tests to test various other aspects and create experiments around assumptions you are not certain about. Along with my team I developed an experiment in which we would give users a small money box and ask them to secretly leave behind the amount of money that they would be willing to spend for the service. We repeated the exercise twice, not only testing our own service but also already existing competing services. Later we disclosed the results and opened a discussion around them. Exercises like this helped us to zoom out and test secondary assumptions qualitatively while not losing focus on our primary objectives.

 
 
 

GET YOUR HANDS DIRTY

Like many other things user testing is best learned by doing. Since every user is different it is important to figure out common usage patterns and problems to get the basics right. Applying and getting results is enriching and rewarding. So go ahead, get your hands dirty and don’t forget to share your experiences with us. If you want to dig deeper I recommend to have a look at Don’t make me Think Revisited by Steve Krug & The User experience team of One by Leah Buley. Both books cover the topic in more detail and are valuable resources if you want to put your products and services to a test .

Conducting user tests usually is only half the job done. Effectively documenting gained insight, prioritising user needs and estimating necessary resources to meet the needs is often hard and might even collide with underlying business models. Stay tuned to find out how we go about these challenges in our next blog post.

 

 

Comment