July 8, 2015
5 min

Avoiding bias in the oh-so-human world of user testing

Optimal Workshop
"Dear Optimal WorkshopMy question is about biasing users with the wording of questions. It seems that my co-workers and I spend too much time debating the wording of task items in usability tests or questions on surveys. Do you have any 'best practices' for wordings that evoke unbiased feedback from users?" — Dominic

Dear Dominic, Oh I feel your pain! I once sat through a two hour meeting that was dominated by a discussion on the merits of question marks!It's funny how wanting to do right by users and clients can tangle us up like fine chains in an old jewellery box. In my mind, we risk provoking bias when any aspect of our research (from question wording to test environment) influences participants away from an authentic response. So there are important things to consider outside of the wording of questions as well. I'll share my favorite tips, and then follow it up with a must-read resource or two.

Balance your open and closed questions

The right balance of open and closed questions is essential to obtaining unbiased feedback from your users. Ask closed questions only when you want a very specific answer like 'How old are you?' or 'Are you employed?' and ask open questions when you want to gain an understanding of what they think or feel. For example, don’t ask the participant'Would you be pleased with that?' (closed question). Instead, ask 'How do you feel about that?' or even better 'How do you think that might work?' Same advice goes for surveys, and be sure to give participants enough space to respond properly — fifty characters isn’t going to cut it.

Avoid using words that are linked to an emotion

The above questions lead me to my next point — don’t use words like ‘happy’. Don’t ask if they like or dislike something. Planting emotion based words in a survey or usability test is an invite for them to tell you what they think you want to hear . No one wants to be seen as being disagreeable. If you word a question like this, chances are they will end up agreeing with the question itself, not the content or meaning behind it...does that make sense? Emotion based questions only serve to distract from the purpose of the testing — leave them at home.

Keep it simple and avoid jargon

No one wants to look stupid by not understanding the terms used in the question. If it’s too complicated, your user might just agree or tell you what they think you want to hear to avoid embarrassment. Another issue with jargon is that some terms may have multiple meanings which can trigger a biased reaction depending on the user’s understanding of the term. A friend of mine once participated in user testing where they were asked if what they were seeing made them feel ‘aroused’. From a psychology perspective, that means you’re awake and reacting to stimuli.

From the user's perspective? I’ll let you fill in the blanks on that one. Avoid using long, wordy sentences when asking questions or setting tasks in surveys and usability testing. I’ve seen plenty of instances of overly complicated questions that make the user tune out (trust me, you would too!). And because people don't tend to admit their attention has wandered during a task, you risk getting a response that lacks authenticity — maybe even one that aims to please (just a thought...).

Encourage participants to share their experiences (instead of tying them up in hypotheticals)

Instead of asking your user what they think they would do in a given scenario, ask them to share an example of a time when they actually did do it. Try asking questions along the lines of 'Can you tell me about a time when you….?' or 'How many times in the last 12 months have you...?' Asking them to recall an experience they had allows you to gain factual insights from your survey or usability test, not hypothetical maybes that are prone to bias.

Focus the conversation by asking questions in a logical order

If you ask usability testing or survey questions in an order that doesn’t quite follow a logical flow, the user may think that the order holds some sort of significance which in turn may change the way they respond. It’s a good idea to ensure that the questions tell a story and follow a logical progression for example the steps in a process — don’t ask me if I’d be interested in registering for a service if you haven’t introduced the concept yet (you’d be surprised how often this happens!). For further reading on this, be sure to check out this great article from usertesting.com.

More than words — the usability testing experience as a whole

Reducing bias by asking questions the right way is really just one part of the picture. You can also reduce bias by influencing the wider aspects of the user testing process, and ensuring the participant is comfortable and relaxed.

Don’t let the designer facilitate the testing

This isn’t always possible, but it’s a good idea to try to get someone else to facilitate the usability testing on your design (and choose to observe if you like). This will prevent you from bringing your own bias into the room, and participants will be more comfortable being honest when the designer isn't asking the questions. I've seen participants visibly relax when I've told them I'm not the designer of a particular website, when it's apparent they've arrived expecting that to be the case.

Minimize discomfort and give observers a role

The more comfortable your participants are, with both the tester and the observer, the more they can be themselves. There are labs out there with two-way mirrors to hide observers, but in all honesty the police interrogation room isn’t always the greatest look! I prefer to have the observer in the testing room, while being conscious that participants may instinctively be uncomfortable with being observed. I’ve seen observer guidelines that insist observers (in the room) stay completely silent the entire time, but I think that can be pretty creepy for participants! Here's what works best (in my humble opinion).

The facilitator leads the testing session, of course, but the observer is able to pipe up occasionally, mostly for clarification purposes, and certainly join in the welcoming, 'How's the weather?' chit chat before the session begins. In fact, when I observe usability testing, I like to be the one who collects the participant from the foyer. I’m the first person they see and it’s my job to make them feel welcome and comfortable, so when they find out I'll be observing, they know me already. Anything you can do to make the participant feel at home will increase the authenticity of their responses.

A note to finish

At the end of the day the reality is we’re all susceptible to bias. Despite your best efforts you’re never going to eradicate it completely, but just being aware of and understanding it goes a long way to reducing its impacts. Usability testing is, after all, something we design. I’ll leave you with this quote from Jeff Sauro's must-read article on 9 biases to watch out for in usability testing:

"We do the best we can to simulate a scenario that is as close to what users would actually do .... However, no amount of realism in the tasks, data, software or environment can change the fact that the whole thing is contrived. This doesn't mean it's not worth doing."

Publishing date
July 8, 2015
Share this article

Related articles

min read
Kate Keep and Brad Millen: How the relationship between Product Owners and Designers can impact human-centered design

Working in a multi-disciplined product team can be daunting, but how can those relationships be built, and what does that mean for your team, your stakeholders, and the users of the product?

Kate Keep, Product Owner, and Brad Millen, UX Designer, both work in the Digital team at the Accident Compensation Corporation (ACC). They recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, about their experience working on a large project within an organization that was new to continuous improvement and digital product delivery.

In their talk, Kate and Brad discuss how they were able to pull a team together around a common vision, and three key principles they found useful along the way.

Background on Kate

Kate is a Product Owner working in the Digital team at ACC, and her team currently look after ACC’s Injury Prevention websites. Kate is also a Photographer, which keeps her eye for detail sharp and her passion for excellence alive. She comes from a Contact Centre background which drives her dedication to continuously search for the optimal customer experience. Kate and the team are passionate about accessibility and building websites that are inclusive for all of Aotearoa.

Contact Details:

Email address: kate.keep@acc.co.nz

LinkedIn URL: Not provided

Background on Brad

Brad is a Digital UX Designer in Digital team at ACC. Before launching into the world of UX, Brad studied game design which sparked his interest in the way people interact, engage and perceive products. This helped to inform his ethos that you’re always designing with others in mind.

How the relationship between Product Owners and Designers can impact human-centered design 👩🏻💻📓✍🏻💡

Brad and Kate preface their talk by acknowledging that they were both new to their roles and came from different career backgrounds when this project began, which presented a significant challenge. Kate was a Product Owner with no previous delivery experience, while Brad, was a UX designer. To overcome these challenges, they needed to quickly figure out how to work together effectively.

Their talk focuses on three key principles that they believe are essential when building a digital product in a large, multi-disciplined team.

Building Trust-Based Relationships 🤝🏻

The first principle emphasizes the importance of building trust-based relationships. They highlight the need to understand each other's perspectives and work together towards a common vision for the customer. This can only be achieved by building a strong sense of trust with everyone on the team. They stress the value of open and honest communication - both within the team and with stakeholders.

Kate, as Product Owner, identified her role as being one of “setting the vision and getting the hell out of the way”. In this way, she avoided putting Brad and his team of designers in a state of paralysis by critiquing decisions all of the time. Additionally, she was clear from the outset with Brad that she needed “ruthless honesty” in order to build a strong relationship.

Cultivating Psychological Safety and a Flat Hierarchy 🧠

The second principle revolves around creating an environment of psychological safety. Kate explains that team members should feel comfortable challenging the status quo and working through disagreements without fear of ridicule. This type of safety improves communication and fast-tracks the project by allowing the team to raise issues without feeling they need to hide and wait for something to break.

They also advocate for a flat hierarchy where everyone has an equal say in decision-making. This approach empowers team members and encourages autonomy. It also means that decisions don’t need to wait for meetings, where juniors are scheduled to report issues or progress to seniors. Instead, all team members should feel comfortable walking up to a manager and, having built a relationship with them, flag what’s on their mind without having to wait. 

This combination of psychological safety and flat hierarchy, coupled with building trust, means that the team dynamic is efficient and productive.

Continuous Focus on the Customer Voice 🔊

The third principle centers on keeping the customer's voice at the forefront of the product development process. Brad and Kate recommend regularly surfacing customer feedback and involving the entire team in understanding customer needs and goals. They also highlight the importance of making customer feedback tangible and visible to all team members and stakeholders.

Explaining why the topic matters 💡

Kate and Brad’s talk sets a firm foundation for building positive and efficient team dynamics. The principles that they discuss champion empowerment and autonomy, which ultimately help multi-disciplined teams to gel when developing digital products. In practice, these principles set the stage for several key advantages.

They stress that building trust is key, not only for the immediate project team but for organizational stakeholders too. It’s just as crucial for the success of the product that all key stakeholders buy into the same way of thinking i.e. trusting the expertise of the product design and development teams. Kate stresses that sometimes Product Owners need to absorb stakeholder pressure and take failures on the chin so that they to let design teams do what they do best.

That being said, Kate also realizes that sometimes difficult decisions need to be made when disagreements arise within the project team. This is when the value of building trust works both ways. In other words, Kate, as Product Owner, needed to make decisions in the best interest of the team to keep the project moving.

Psychological safety, in practice, means leading by example and providing a safe environment for people to be honest and feel comfortable enough to speak up when necessary. This can even mean being honest about what scares you. People tend to value this type of honesty, and it establishes common ground by encouraging team members (and key stakeholders) to be upfront with each other.

Finally, keeping the customer's voice front and center is important, not just as design best practice, but also as a way of keeping the project team grounded. Whenever the project experiences a bump in the road, or a breakdown in team communication, Kate and Brad suggest always coming back to the question, “What’s most important to the customer?”. Allow user feedback to be accessible to everyone in the team. This means that the customer's voice can be present throughout the whole project, and everyone, including key stakeholders, never lose sight of the real-life application of the product. In this way, teams are consistently able to work with facts and insights rather than making assumptions that they think are best for the product.

What is UX New Zealand? 🤷

UX New Zealand is a leading UX and IA conference hosted by Optimal Workshop, that brings together industry professionals for three days of thought leadership, meaningful networking and immersive workshops. 

At UX New Zealand 2023, we featured some of the best and brightest in the fields of user experience, research and design. A raft of local and international speakers touched on the most important aspects of UX in today’s climate for service designers, marketers, UX writers and user researchers.

These speakers are some of the pioneers leading the way and pushing the standard for user experience today. Their experience and perspectives are invaluable for those working at the coalface of UX, and together, there’s a tonne of valuable insight on offer. 

min read
How to sell human-centered design

Picture this scenario: You're in your local coffee shop and hear a new song. You want to listen to it when you get back to the office. How do you obtain it? If you’re one of the 232 million Spotify users, you’ll simply open the app, search for the song and add it to your playlist. Within seconds, you’ll have the song ready to play whenever and wherever you want.

This new norm of music streaming wasn’t always the status quo. In the early days of the internet, the process of finding music was easy but nowhere nearly as easy as it is now. You’d often still be able to find any song you wanted, but you would need to purchase it individually or as part of an album, download it to your computer and then sync it across to a portable music player like the iPod.

Spotify is a prime example of successful human-centered design. The music service directly addresses the primary pain points with accessing music and building music collections by allowing users to pay a monthly fee and immediately gain access to a significant catalog of the world’s music.

It’s also far from the only example. Take HelloFresh, for example. Founded by Dominik Richter, Thomas Griesel and Jessica Nilsson in 2011, this company delivers a box of ingredients and recipes to your door each week, meaning there’s no need for grocery shopping or thinking about what to cook. It’s a service that addresses a fairly common problem: People struggle to find the time to go out and buy groceries and also create tasty, healthy meals, so the founders addressed both issues.

Both HelloFresh and Spotify are solutions to real user problems. They weren’t born as a result of people sitting in a black box and trying to come up with new products or services. This is the core of human-centered design – identifying something that people have trouble with and then building an appropriate answer.

The origins of human-centered design

But, someone is likely to ask, what’s even the point of human-centered design? Shouldn’t all products and services be designed for the people using them? Well, yes.

Interestingly, while terms like human-centered design and design thinking have become much more popular in recent years, they’re not entirely new methods of design. Designers have been doing this same work for decades, just under a different name: design. Just take one look at some of the products put together by famed industrial designer Dieter Rams (who famously influenced ex-Apple design lead Jony Ive). You can’t look at the product below and say it was designed without the end user in mind.

A radio by industrial designer Dieter Rams.
A radio by industrial designer Dieter Rams.

Why did human-centered design even gain traction as a term? David Howell (a UX designer from Australia) explains that designers often follow Parkinson’s Law, where “work expands so as to fill the time available for its completion”. He notes that designers could always do more (more user research, more ideation, more testing, etc), and that by wrapping everything under a single umbrella (like human-centered design) designers can “speak to their counterparts in business as a process and elevate their standing, getting the coveted seat at the table”.

Human-centered design, for all intents and purposes, is really just a way for designers to package up the important principles intrinsic to good design and sell them to those who may not be sympathetic to exactly why they’re important. At a broader level, the same thinking can be applied to UX as a whole. Good user experience should naturally fall under design, but occasionally a different way of looking at something is necessary to drive the necessary progress.


So human-centered design can really just be thought of as a vehicle to sell the importance of a user-first approach to organizations – that’s useful, but how exactly are you supposed to start? How do you sell something that’s both easily understandable but at the same time quite nebulous? Well, you sell it in the same way you’d sell user research.

How to sell human-centered design

Focus on the product

In the simplest terms, a product designed and built based on user input is going to perform better than one that was assembled based on internal organizational thinking.

When utilized in the right way, taking a human-centered approach to product design leads to products that resonate much more effectively with people. We looked at Spotify at the beginning of this article for a company that’s continuously adopted this practice, but there are countless others. AirBnB, Uber, Pinterest and more all jump to mind. Google and LinkedIn, meanwhile, serve as good examples of the ‘old guard’ that are starting to invest more in the user experience.

Understand the cost-benefit

In 2013, Microsoft was set to unveil the latest version of its Xbox video game console. Up until that point, the company had found significant success in the videogame market. Past versions of the Xbox console had largely performed very well both critically and commercially. With the newest version, however, the company quickly ran into problems.

The new ‘Xbox One’ was announced with several features that attracted scorn from both the target market and the gaming press. The console would, for example, tie both physical and digital purchases to users’ accounts, meaning they wouldn’t be able to sell them on (a popular practice). The console would also need to remain connected to the internet to check these game licenses, likely leading to significant problems for those without reliable internet access. Lastly, Microsoft also stated that users would have to keep an included camera system plugged in at all times otherwise the console wouldn’t function. This led to privacy advocates arguing that the camera system’s data could be used for things like targeted advertising and user surveillance.

Needless to say, after seeing the response from the press and the console’s target market, Microsoft backtracked and eventually released the Xbox One without the always-on requirement, game licensing system or camera connection requirement.

Think of the costs Microsoft likely incurred having to roll back every one of these decisions so late into the product’s development. If you’re able to identify an issue in the research or prototype phase, it’s going to be significantly cheaper to fix here as opposed to 3 years into development with a release on the horizon.

Wrap-up

As the Spotify founders discovered back in back in 2008, taking a human-centered approach to product design can lead to revolutionary products and experiences. It’s not surprising. After all, how can you be expected to build something that people want to use without understanding said people?

Seeing is believing

Dive into our platform, explore our tools, and discover how easy it can be to conduct effective UX research.