August 8, 2022
4 min

Usability Testing: what, how and why?

Knowing and understanding why and how your users use your product can be invaluable for getting to the nitty gritty of usability. Where they get stuck and where they fly through. Delving deep with probing questions into motivation or skimming over looking for issues can equally be informative.

Usability testing can be done in several ways, each way has its benefits. Put super simply, usability testing literally is testing how useable your product is for your users. If your product isn't useable users will not stick around or very often complete their task, let alone come back for more.

What is usability testing? 🔦

Usability testing is a research method used to evaluate how easy something is to use by testing it with representative users.

These tests typically involve observing a participant as they work through a series of tasks involving the product being tested. Having conducted several usability tests, you can analyze your observations to identify the most common issues.

We go into the three main methods of usability testing:

  1. Moderated and unmoderated
  2. Remote or in person
  3. Explorative, assessment or comparative

1. Moderated or unmoderated usability testing 👉👩🏻💻

Moderated usability testing is done in-person or remotely by a researcher who introduces the test to participants, answers their queries, and asks follow-up questions. Often these tests are done in real time with participants and can involve other research stakeholders. Moderated testing usually produces more in-depth results thanks to the direct interaction between researchers and test participants. However, this can be expensive to organize and run.

Top tip: Use moderated testing to investigate the reasoning behind user behavior.

Unmoderated usability testing is done without direct supervision; likely participants are in their own homes and/or using their own devices to browse the website that is being tested. And often at their own pace.  The cost of unmoderated testing is lower, though participant answers can remain superficial and making follow-up questions can be difficult.

Top tip: Use unmoderated testing to test a very specific question or observe and measure behavior patterns.

2. Research or in-person usability testing 🕵

Remote usability testing is done over the internet or by phone. Allowing the participants to have the time and space to work in their own environment and at their own pace. This however doesn’t give the researcher much in the way of contextual data because you’re unable to ask questions around intention or probe deeper if the participant makes a particular decision. Remote testing doesn’t go as deep into a participant’s reasoning, but it allows you to test large numbers of people in different geographical areas using fewer resources.

Top tip: Use remote testing when a large group of participants are needed and the questions asked can be direct and unambiguous.

In-person usability testing, as the name suggests, is done in the presence of a researcher. In-person testing does provide contextual data as researchers can observe and analyze body language and facial expressions. You’re also often able to converse with participants and find out more about why they do something. However, in-person testing can be expensive and time-consuming: you have to find a suitable space, block out a specific date, and recruit (and often pay) participants.

Top tip: In-person testing gives researchers more time and insight into motivation for decisions.

3. Explorative, Assessment or comparative testing 🔍

These three usability testing methods generate different types of information:

Explorative testing is open-ended. Participants are asked to brainstorm, give opinions, and express emotional impressions about ideas and concepts. The information is typically collected in the early stages of product development and helps researchers pinpoint gaps in the market, identify potential new features, and workshop new ideas.

Assessment research is used to test a user's satisfaction with a product and how well they are able to use it. It's used to evaluate general functionality.

Comparative research methods involve asking users to choose which of two solutions they prefer, and they may be used to compare a product with its competitors.

Top tip: Depending on what research is being done, and how much qualitative or quantitative data is wanted.

Which method is right for you? 🧐

Whether the testing is done in-person, remote, moderated or unmoderated will depend on your purpose, what you want out of the testing, and to some extent your budget. 

Depending on what you are testing, each of the usability testing methods we explored here can offer an answer. If you are at the development stage of a product it can be useful to conduct a usability test on the entire product. Checking the intuitive usability of your website, to ensure users can make the best decisions, quickly. Or adding, changing or upgrading a product can also be the moment to check on a specific question around usability. Planning and understanding your objectives are key to selecting the right usability testing option for your project.

Let's take a look at a couple of examples of usability testing.

1. Lab based, in-person moderated testing - mid-life website

Imagine you have a website that sells sports equipment. Over time your site has become cluttered and disorganized, much like a bricks and mortar store may. You’ve noticed a drop in sales in certain areas. How do you find out what is going wrong or where users are getting lost? Having an in-person, lab (or other controlled environment), moderated usability test with users you can set tasks, watch (and record) what they do.

The researcher can literally be standing or sitting next to the participant throughout, recording contextual information such as how they interacted with the mouse, laptop or even the seat. Watching for cues as to the comfort of the participant and asking questions about why they make decisions can provide richer insights. Maybe they wanted purple yoga pants, but couldn’t find the ‘yoga’ section which was listed under gym rather than a clothing section.

Meaning you can look at how your stock is organised, or even investigate undertaking a card sort. This provides robust and fully rounded feedback on users behaviours, expectations and experiences. Providing data that can directly be turned into actionable directives when redeveloping the website. 

2. Remote, moderated assessment testing - app product development

You are looking at launching an app for parents to access for information and updates for the school. It’s still in development stage and at this point you want to know how easy the app is to use. Setting some very specific set tasks for participants to complete the app can be sent to them and they can be left to complete (or not). Providing feedback and comments around the usability.

The next step may be to use first click testing to see how and where the interface is clicked and where participants may be spending time, or becoming lost. Whilst the feedback and data gathered from this testing can be light, it will be very direct to the questions asked. And will provide data to back up (or possibly not) what assumptions were made.

3. Moderated, In-person, explorative testing - new product development

You’re right at the start of the development process. The idea is new and fresh and the basics are being considered. What better way to get an understanding of what your users’ truly want than an explorative study.

Open-ended questions with participants in a one-on-one environment (or possibly in groups) can provide rich data and insights for the development team. Imagine you have an exciting new promotional app that you are developing for a client. There are similar apps on the market but none as exciting as what your team has dreamt up. By putting it (and possibly the competitors) to participants they can give direct feedback on what they like, love and loathe.

They can also help brainstorm ideas or better ways to make the app work, or improve the interface. All of this done, before there is money sunk in development.

Wrap up 🌯

Key objectives will dictate which usability testing method will deliver the answers to your questions.

Whether it’s in-person, remote, moderated or comparative with a bit of planning you can gather data around your users very real experience of your product. Identify issues, successes and failures. Addressing your user experience with real data, and knowledge can but lead to a more intuitive product.

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

Building Trust Through Design for Financial Services

When it comes to financial services, user experience goes way beyond just making things easy to use. It’s about creating a seamless journey and establishing trust at every touchpoint. Think about it: as we rely more and more on digital banking and financial apps in our everyday lives, we need to feel absolutely confident that our personal information is safe and that the companies managing our money actually know what they're doing. Without that trust foundation, even the most competitive brands will struggle with customer adoption.

Why Trust Matters More Than Ever

The stakes are uniquely high in financial UX. Unlike other digital products where a poor experience might result in minor frustration, financial applications handle our life savings, investment portfolios, and sensitive personal data. A single misstep in design can trigger alarm bells for users, potentially leading to lost customers.

Using UX Research to Measure and Build Trust

Building high trust experiences requires deep insights into user perceptions, behaviors, and pain points. The best UX platforms can help financial companies spot trust issues and test whether their solutions actually work.

Identify Trust Issues with Tree Testing

Tree testing helps financial institutions understand how easily users can find critical information and features:

  • Test information architecture to ensure security features and privacy information are easily discoverable
  • Identify confusing terminology that may undermine user confidence
  • Compare findability metrics for trust-related content across different user segments

Optimize for Trustworthy First Impressions with First-Click Testing

First-click testing helps identify where users naturally look for visual symbols and cues that are associated with security:

  • Test where users instinctively look for security indicators like references to security certifications
  • Compare the effectiveness of different visual trust symbols (locks, shields, badges)
  • Identify the optimal placement for security messaging across key screens

Map User Journeys with Card Sorting

Card sorting helps brands understand how users organize concepts. Reducing confusion, helps your financial brand appear more trustworthy, quickly:

  • Use open card sorts to understand how users naturally categorize security and privacy features
  • Identify terminology that resonates with users' perceptions around security

Qualitative Insights Through Targeted Questions

Gathering qualitative data through strategically placed questions allows financial institutions to collect rich, timely insights about how much their customers trust their brand:

  • Ask open ended questions about trust concerns at key moments in the testing process
  • Gather specific feedback on security terminology understanding and recognition
  • Capture emotional responses to different trust indicators

What Makes a Financial Brand Look Trustworthy?

Visual Consistency and Professional Polish

When someone opens your financial app or website, they're making snap judgments about whether they can trust you with their money. It happens in milliseconds, and a lot of that decision comes down to how polished and consistent everything looks.Clean, consistent design sends that signal of stability and attention to detail that people expect when money's involved.

To achieve this, develop and rigorously apply a solid design system across all digital touchpoints including fonts, colors, button styles, and spacing, it all needs to be consistent across every page and interaction. Even small inconsistencies can make people subconsciously lose confidence.

Making Security Visible

Unlike walking into a bank where you can see the vault and security cameras, digital security happens behind the scenes. Users can't see all the protection you've built in unless you make a point of showing them.

Highlighting your security measures in ways that feel reassuring rather than overwhelming gives people that same sense of "my money is safe here" that they'd get from seeing a bank's physical security.

From a design perspective, apply this thinking to elements like:

  • Real time login notifications
  • Transaction verification steps
  • Clear encryption indicators
  • Transparent data usage explanations
  • Session timeout warnings

You can test the success of these design elements through preference testing, where you can compare different approaches to security visualization to determine which elements most effectively communicate trust without creating anxiety.

Making Complex Language Simple

Financial terminology is naturally complex, but your interface content doesn't have to be. Clear, straightforward language builds trust so it’s important to develop a content strategy that:

  • Explains unavoidable complex terms contextually
  • Replaces jargon with plain language
  • Provides proactive guidance before errors occur
  • Uses positive, confident messaging around security features

You can test your language and navigation elements by using tree testing to evaluate user understanding of different terminology, measuring success rates for finding information using different labeling options.

Create an Ongoing Trust Measurement Program

A user research platform enables financial institutions to implement ongoing trust measurement across the product lifecycle:

Establish Trust Benchmarks

Use UX research tools to establish baseline metrics for measuring user trust:

  • Findability scores for security features
  • User reported confidence ratings
  • Success rates for security related tasks
  • Terminology comprehension levels

Validate Design Updates

Before implementing changes to critical elements, use quick tests to validate designs:

  • Compare current vs. proposed designs with prototype testing
  • Measure findability improvements with tree testing
  • Evaluate usability through first-click testing

Monitor Trust Metrics Over Time

Create a dashboard of trust metrics that can be tracked regularly:

  • Task success rates for security related activities
  • Time-to-completion for verification processes
  • Confidence ratings at key security touchpoints

Cross-Functional Collaboration to Improve Trust

While UX designers can significantly impact brand credibility, remember that trust is earned across the entire customer experience:

  • Product teams ensure feature promises align with actual capabilities
  • Security teams translate complex security measures into user-friendly experiences
  • Marketing ensures brand promises align with the actual user experience
  • Customer service supports customers when trust issues arise

Trust as a Competitive Advantage

In an industry where products and services can often seem interchangeable to consumers, trust becomes a powerful differentiator. By placing trust at the center of your design philosophy and using comprehensive user research to measure and improve trust metrics, you're not just improving user experience, you're creating a foundation for lasting customer relationships in an industry where loyalty is increasingly rare.

The most successful financial institutions of the future won't necessarily be those with the most features or the slickest interfaces, but those that have earned and maintained user trust through thoughtful UX design built on a foundation of deep user research and continuous improvement.

Learn more
1 min read

The ultimate reading list for new user researchers

Having a library of user research books is invaluable. Whether you’re an old hand in the field of UX research or just dipping your toes in the water, being able to reference detailed information on methods, techniques and tools will make your life much easier.

There’s really no shortage of user research/UX reading lists online, so we wanted to do something a little different. We’ve broken our list up into sections to make finding the right book for a particular topic as easy as possible.

General user research guides

These books cover everything you need to know about a number of UX/user research topics. They’re great to have on your desk to refer back to – we certainly have them on the bookshelf here at Optimal Workshop.

Observing the User Experience: A Practitioner's Guide to User Research

Mike Kuniavsky

Observing the User Experience: A Practitioner’s Guide to User Research

This book covers 13 UX research techniques in a reference format. There’s a lot of detail, making it a useful resource for people new to the field and those who just need more clarification around a certain topic. There’s also a lot of practical information that you’ll find applicable in the real world. For example, information about how to work around research budgets and tight time constraints.

Just Enough Research

Erika Hall

Just Enough Research

In Just Enough Research, author Erika Hall explains that user research is something everyone can and should do. She covers several research methods, as well as things like how to identify your biases and make use of your findings. Designers are also likely to find this one quite useful, as she clearly covers the relationship between research and design.

Research Methods in Human-Computer Interaction

Harry Hochheiser, Jonathan Lazar, Jinjuan Heidi Feng

Research Methods in Human-Computer Interaction

Like Observing the User Experience, this is a dense guide – but it’s another essential one. Here, experts on human-computer interaction and usability explain different qualitative and quantitative research methods in an easily understandable format. There are also plenty of real examples to help frame your thinking around the usefulness of different research methods.

Information architecture

If you’re new to information architecture (IA), understanding why it’s such an important concept is a great place to start. There’s plenty of information online, but there are also several well-regarded books that make great starting points.

Information Architecture for the World Wide Web: Designing Large-Scale Web Sites

Peter Morville, Louis Rosenfeld

Information Architecture for the World Wide Web: Designing Large-Scale Web Site

You’ll probably hear this book referred to as “the polar bear book”, just because the cover features a polar bear. But beyond featuring a nice illustration of a bear, this book clearly covers the process of creating large websites that are both easy to navigate and appealing to use. It’s a useful book for designers, information architects and user researchers.

How to Make Sense of Any Mess

Abby Covert

How to Make Sense of Any Mess

This is a great introduction to information architecture and serves as a nice counter to the polar bear book, being much shorter and more easily digestible. Author Abby Covert explains complex concepts in a way anyone can understand and also includes a set of lessons and exercises with each chapter.

User interviews

For those new to the task, the prospect of interviewing users is always daunting. That makes having a useful guide that much more of a necessity!

Interviewing Users: How to Uncover Compelling Insights

Steve Portigal

Interviewing Users: How to Uncover Compelling Insights

While interviewing users may seem like something that doesn’t require a guide, an understanding of different interview techniques can go a long way. This book is essentially a practical guide to the art of interviewing users. Author Steve Portigal covers how to build rapport with your participants and the art of immersing yourself in how other people see the world – both key skills for interviewers!

Usability testing

Web usability is basically the ease of use of a website. It’s a broad topic, but there are a number of useful books that explain why it’s important and outline some of the key principles.

Don't Make Me Think: A Common Sense Approach to Web Usability

Steve Krug

Don't Make Me Think: A Common Sense Approach to Web Usability

Don’t Make Me Think is the first introduction to the world of UX and usability for many people, and for good reason – it’s a concise introduction to the topics and is easy to digest. Steve Krug explains some of the key principles of intuitive navigation and information architecture clearly and without overly technical language. In the latest edition, he’s updated the book to include mobile usability considerations.

As a testament to just how popular this book is, it was released in 2000 and has since had 2 editions and sold 400,000 copies.

Design

The design–research relationship is an important one, even if it’s often misunderstood. Thankfully, authors like Don Norman and Vijay Kumar are here to explain everything.

The Design of Everyday Things

Don Norman

The Design of Everyday Things

This book, by cognitive scientist and usability engineer Don Norman, explains how design is the communication between an object and its user, and how to improve this communication as a way of improving the user experience. If nothing else, this book will force you to take another look at the design of everyday objects and assess whether or not they’re truly user-friendly.

101 Design Methods: A Structured Approach for Driving Innovation in Your Organization

Vijay Kumar

101 Design Methods: A Structured Approach for Driving Innovation in Your Organization

A guidebook for innovation in the context of product development, this book approaches the subject in a slightly different way to many other books on the same subject. The focus here is that the practice of creating new products is actually a science – not an art. Vijay Kumar outlines practical methods and useful tools that researchers and designers can use to drive innovation, making this book useful for anyone involved in product development.

See our list on Goodreads

We've put together a list of all of the above books on Goodreads, which you can access here.

Further reading

For experienced practitioners and newcomers alike, user research can often seem like a minefield to navigate. It can be tricky to figure out which method to use when, whether you bring a stakeholder into your usability test (you should) and how much you should pay participants. Take a look at some of the other articles on our blog if you’d like to learn more.

Learn more
1 min read

Using paper prototypes in UX

In UX research we are told again and again that to ensure truly user-centered design, it’s important to test ideas with real users as early as possible. There are many benefits that come from introducing the voice of the people you are designing for in the early stages of the design process. The more feedback you have to work with, the more you can inform your design to align with real needs and expectations. In turn, this leads to better experiences that are more likely to succeed in the real world.It is not surprising then that paper prototypes have become a popular tool used among researchers. They allow ideas to be tested as they emerge, and can inform initial designs before putting in the hard yards of building the real thing. It would seem that they’re almost a no-brainer for researchers, but just like anything out there, along with all the praise, they have also received a fair share of criticism, so let’s explore paper prototypes a little further.

What’s a paper prototype anyway? 🧐📖

Paper prototyping is a simple usability testing technique designed to test interfaces quickly and cheaply. A paper prototype is nothing more than a visual representation of what an interface could look like on a piece of paper (or even a whiteboard or chalkboard). Unlike high-fidelity prototypes that allow for digital interactions to take place, paper prototypes are considered to be low-fidelity, in that they don’t allow direct user interaction. They can also range in sophistication, from a simple sketch using a pen and paper to simulate an interface, through to using designing or publishing software to create a more polished experience with additional visual elements.

Screen Shot 2016-04-15 at 9.26.30 AM
Different ways of designing paper prototypes, using OptimalSort as an example

Showing a research participant a paper prototype is far from the real deal, but it can provide useful insights into how users may expect to interact with specific features and what makes sense to them from a basic, user-centered perspective. There are some mixed attitudes towards paper prototypes among the UX community, so before we make any distinct judgements, let's weigh up their pros and cons.

Advantages 🏆

  • They’re cheap and fastPen and paper, a basic word document, Photoshop. With a paper prototype, you can take an idea and transform it into a low-fidelity (but workable) testing solution very quickly, without having to write code or use sophisticated tools. This is especially beneficial to researchers who work with tight budgets, and don’t have the time or resources to design an elaborate user testing plan.
  • Anyone can do itPaper prototypes allow you to test designs without having to involve multiple roles in building them. Developers can take a back seat as you test initial ideas, before any code work begins.
  • They encourage creativityFrom both the product teams participating in their design, but also from the users. They require the user to employ their imagination, and give them the opportunity express their thoughts and ideas on what improvements can be made. Because they look unfinished, they naturally invite constructive criticism and feedback.
  • They help minimize your chances of failurePaper prototypes and user-centered design go hand in hand. Introducing real people into your design as early as possible can help verify whether you are on the right track, and generate feedback that may give you a good idea of whether your idea is likely to succeed or not.

Disadvantages 😬

  • They’re not as polished as interactive prototypesIf executed poorly, paper prototypes can appear unprofessional and haphazard. They lack the richness of an interactive experience, and if our users are not well informed when coming in for a testing session, they may be surprised to be testing digital experiences on pieces of paper.
  • The interaction is limitedDigital experiences can contain animations and interactions that can’t be replicated on paper. It can be difficult for a user to fully understand an interface when these elements are absent, and of course, the closer the interaction mimics the final product, the more reliable our findings will be.
  • They require facilitationWith an interactive prototype you can assign your user tasks to complete and observe how they interact with the interface. Paper prototypes, however, require continuous guidance from a moderator in communicating next steps and ensuring participants understand the task at hand.
  • Their results have to be interpreted carefullyPaper prototypes can’t emulate the final experience entirely. It is important to interpret their findings while keeping their limitations in mind. Although they can help minimize your chances of failure, they can’t guarantee that your final product will be a success. There are factors that determine success that cannot be captured on a piece of paper, and positive feedback at the prototyping stage does not necessarily equate to a well-received product further down the track.

Improving the interface of card sorting, one prototype at a time 💡

We recently embarked on a research project looking at the user interface of our card-sorting tool, OptimalSort. Our research has two main objectives — first of all to benchmark the current experience on laptops and tablets and identify ways in which we can improve the current interface. The second objective is to look at how we can improve the experience of card sorting on a mobile phone.

Rather than replicating the desktop experience on a smaller screen, we want to create an intuitive experience for mobiles, ensuring we maintain the quality of data collected across devices.Our current mobile experience is a scaled down version of the desktop and still has room for improvement, but despite that, 9 per cent of our users utilize the app. We decided to start from the ground up and test an entirely new design using paper prototypes. In the spirit of testing early and often, we decided to jump right into testing sessions with real users. In our first testing sprint, we asked participants to take part in two tasks. The first was to perform an open or closed card sort on a laptop or tablet. The second task involved using paper prototypes to see how people would respond to the same experience on a mobile phone.

blog_artwork_01-03

Context is everything 🎯

What did we find? In the context of our research project, paper prototypes worked remarkably well. We were somewhat apprehensive at first, trying to figure out the exact flow of the experience and whether the people coming into our office would get it. As it turns out, people are clever, and even those with limited experience using a smartphone were able to navigate and identify areas for improvement just as easily as anyone else. Some participants even said they prefered the experience of testing paper prototypes over a laptop. In an effort to make our prototype-based tasks easy to understand and easy to explain to our participants, we reduced the full card sort to a few key interactions, minimizing the number of branches in the UI flow.

This could explain a preference for the mobile task, where we only asked participants to sort through a handful of cards, as opposed to a whole set.The main thing we found was that no matter how well you plan your test, paper prototypes require you to be flexible in adapting the flow of your session to however your user responds. We accepted that deviating from our original plan was something we had to embrace, and in the end these additional conversations with our participants helped us generate insights above and beyond the basics we aimed to address. We now have a whole range of feedback that we can utilize in making more sophisticated, interactive prototypes.

Whether our success with using paper prototypes was determined by the specific setup of our testing sessions, or simply by their pure usefulness as a research technique is hard to tell. By first performing a card sorting task on a laptop or tablet, our participants approached the paper prototype with an understanding of what exactly a card sort required. Therefore there is no guarantee that we would have achieved the same level of success in testing paper prototypes on their own. What this does demonstrate, however, is that paper prototyping is heavily dependent on the context of your assessment.

Final thoughts 💬

Paper prototypes are not guaranteed to work for everybody. If you’re designing an entirely new experience and trying to describe something complex in an abstracted form on paper, people may struggle to comprehend your idea. Even a careful explanation doesn’t guarantee that it will be fully understood by the user. Should this stop you from testing out the usefulness of paper prototypes in the context of your project? Absolutely not.

In a perfect world we’d test high fidelity interactive prototypes that resemble the real deal as closely as possible, every step of the way. However, if we look at testing from a practical perspective, before we can fully test sophisticated designs, paper prototypes provide a great solution for generating initial feedback.In his article criticizing the use of paper prototypes, Jake Knapp makes the point that when we show customers a paper prototype we’re inviting feedback, not reactions. What we found in our research however, was quite the opposite.

In our sessions, participants voiced their expectations and understanding of what actions were possible at each stage, without us having to probe specifically for feedback. Sure we also received general comments on icon or colour preferences, but for the most part our users gave us insights into what they felt throughout the experience, in addition to what they thought.

Further reading 🧠

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.