In-person moderated user testing is a valuable part of any research project. Meaning you can see first-hand how your users interact with your prototypes and products. But in-person isn’t always a viable option. What to do if your project needs user testing but it’s just not possible to get in front of your users personally? 

‍

Let’s talk unmoderated user testing. This approach sidesteps the need to meet your participants face-to-face as it’s done entirely remotely, over the internet. By it’s very nature there are also considerable benefits to unmoderated user testing.

‍

What is unmoderated user testing? 💻👀

‍

In the most basic sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants complete the testing activity by themselves and in their own time. For the most part, everything else stays the same.

‍

The key differences are:

‍

• You can’t ask follow-up questions
• You can’t use low-fidelity prototypes
• You can’t support participants (beyond the initial instructions you send them).

‍

Is unmoderated user testing right for your research project?

‍

By nature, unmoderated user research does not include any direct interaction between the researcher and the study participants. This is really the biggest benefit and also the biggest drawback. 

‍

Benefits of unmoderated usability testing 👩🏻💻

‍

• Speed and turnaround  - As there is no need to schedule meetings with each participant, unmoderated testing is usually much faster to initiate and complete. Depending on the study, it may be possible to launch a study and receive results in just a few hours.

• Size of study (participant numbers) - Unmoderated user testing also allows you to collect feedback from dozens or even hundreds of users at the same time.

• Location (local and/or international) -Testing online removes reliance on participants being physically present for the testing which broadens the ability to make contact with participants within your country or across the globe. 

‍

If you’d like to know more about the benefits of unmoderated usability testing, take a look at our article five reasons you should consider unmoderated user testing.

‍

Limitations of unmoderated usability testing 🚧

‍

• Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype.

• Participant behavior - Without a moderator, participants tend to be less engaged and behave less realistically in tasks that depend on imagination, decision-making, or emotional responses.

• Inability to ask follow-up questions - by not being in the testing with the participant, the facilitator can’t ask further questions to get a deeper understanding of the participant’s reasoning. As you can’t rely on human judgment through a moderator being in the room with the participants and the ability to adjust the test in the moment, unmoderated usability testing needs thorough up front planning.

‍

Because of these limitations unmoderated usability testing usually works best for evaluating live websites and apps or highly functional prototypes.  It’s great for testing activities that don’t require a lot of imagination or emotion from participants. Such as testing functionality or answering direct queries to do with your product.

‍

What’s involved when setting up unmoderated usability testing? 🤔💭

‍

1. Define testing goals

With any usability testing, it pays to define your goals before getting underway with setting up the software. What do you want to know from the participants? Goals vary from test to test. Understanding your goals upfront will help you to make the correct tool choice.

‍

2. Define your demographic

With a clear understanding of your goal, now it’s time to consider which participants are right for your study. Think about who they are, their demographic, and where they live. Are they new users or existing? Are they experts or novices?

‍

3. Selecting testing software

As unmoderated studies, are done remotely, the software used to faciliate the study plays a key role in ensuring you get useful results. Without a facilitator, the software must guide the participants through the session and record what happens. Take the time to test software and select one that is right for your study.

‍

4. Write your own tasks and questions

Think through your goals and what you want to achieve from the testing. Many of the unmoderated testing services include study templates with generic example tasks. Remember they are templates, and your tasks and questions should be specific to your particular study. Any task instructions guiding the participants should be clear and directive.

‍

5. Trial session

You’ve done all of the upfront work, now it’s time to test that it works, the software does what you expect and the instructions you have written can be followed. Doing a test run is crucial, especially with unmoderated usability testing, as there won’t be a facilitator in the testing to fix any problems.

‍

6. Recruit participants

Having defined your target audience and demographic, now is the time to recruit participants. Ensuring you have some control over the recruitment process is important, either through screening questions or recruiting your own. There are services that  recruit from a pool of willing participants. Thiscan be a great way to get a wide range of users.

‍

7. Analyze results

You are likely to accumulate a lot of data from your unmoderated testing. You’ll need a way to organize and analyze the data to derive insights that are valuable. Depending on the type of usability testing you do will vary the type of results. Quantitative testing gives data-driven results and direct answers. Whereas qualitative testing through audio or video recordings of participants’ actions or comments will need time to analyze and look at behavioral observations. 

‍

Wrap Up 🌯

‍

Unmoderated usability testing can be a good option for your study. It may not be right for all of your studies all of the time. While it can be quick to implement and often cheaper than moderated usability testing, it still requires time and planning to ensure you get the data insights you are looking for. Following a checklist can be a great way to ensure you approach your research methodically.

‍

The term ‘usability’ captures sentiments of how usable, useful, enjoyable, and intuitive a website or app is perceived by users. By its very nature, usability is somewhat subjective. But what we’re really looking for when we talk about usability is how well a website can be used to achieve a specific task or goal. Using this definition we can analyze usability metrics (standard units of measurement) to understand how well user experience design is performing.

‍

Usability metrics provide helpful insights before and after any digital product is launched. They help us form a deeper understanding of how we can design with the user front of mind. This user-centered design approach is considered the best-practice in building effective information architecture and user experiences that help websites, apps, and software meet and exceed users' needs.

‍

In this article, we’ll highlight key usability metrics, how to measure and understand them, and how you can apply them to improve user experience.

‍

Understanding Usability Metrics

‍

Usability metrics aim to understand three core elements of usability, namely: effectiveness, efficiency, and satisfaction. A variety of research techniques offer designers an avenue for quantifying usability. Quantifying usability is key because we want to measure and understand it objectively, rather than making assumptions.

‍

Types of Usability Metrics
‍

There are a few key metrics that we can measure directly if we’re looking to quantify effectiveness, efficiency, and satisfaction. Here are four common examples:

‍

• Success rate: Also known as ‘completion rate’, success rate is the percentage of users who were able to successfully complete the tasks.
• Time-based efficiency: Also known as ‘time on task’, time-based efficiency measures how much time a user needs to complete a certain task.
• Number of errors: Sounds like what it is! It measures the average number of times where an error occurred per user when performing a given task.
• Post-task satisfaction: Measures a user's general impression or satisfaction after completing (or not completing) a given task.

‍

How to Collect Usability Metrics

‍
Usability metrics are outputs from research techniques deployed when conducting usability testing. Usability testing in web design, for example, involves assessing how a user interacts with the website by observing (and listening to) users completing defined tasks, such as purchasing a product or signing up for newsletters.

‍

Conducting usability testing and collecting usability metrics usually involves:

‍

• Defining a set of tasks that you want to test
• Recruitment of test participants
• Observing participants (remotely or in-person)
• Recording detailed observations
• Follow-up satisfaction survey or questionnaire

‍

Tools such Reframer are helpful in conducting usability tests remotely, and they enable live collaboration of multiple team members. It is extremely handy when trying to record and organize those insightful observations! Using paper prototypes is an inexpensive way to test usability early in the design process.

‍

The Importance of Usability Metrics in User-Centered Design

‍

User-centered design challenges designers to put user needs first. This means in order to deploy user-centered design, you need to understand your user. This is where usability testing and metrics add value to website and app performance; they provide direct, objective insight into user behavior, needs, and frustrations. If your user isn’t getting what they want or expect, they’ll simply leave and look elsewhere.

‍

Usability metrics identify which parts of your design aren’t hitting the mark. Recognizing where users might be having trouble completing certain actions, or where users are regularly making errors, are vital insights when implementing user-centered design. In short, user-centered design relies on data-driven user insight.

‍

But why hark on about usability metrics and user-centered design? Because at the heart of most successful businesses is a well-solved user problem. Take Spotify, for example, which solved the problem of dodgy, pirated digital files being so unreliable. People liked access to free digital music, but they had to battle viruses and fake files to get it. With Spotify, for a small monthly fee, or the cost of listening to a few ads, users have the best of both worlds. The same principle applies to user experience - identify recurring problems, then solve them.

‍

Best Practices for Using Usability Metrics

‍

Usability metrics should be analyzed by design teams of every size. However, there are some things to bear in mind when using usability metrics to inform design decisions:

‍

• Defining success: Usability metrics are only valuable if they are being measured against clearly defined benchmarks. Many tasks and processes are unique to each business, so use appropriate comparisons and targets; usually in the form of an ‘optimized’ user (a user with high efficiency).
• Real user metrics: Be sure to test with participants that represent your final user base. For example, there’s little point in testing your team, who will likely be intimately aware of your business structure, terminology, and internal workflows.
• Test early: Usability testing and subsequent usability metrics provide the most impact early on in the design process. This usually means testing an early prototype or even a paper prototype. Early testing helps to avoid any significant, unforeseen challenges that could be difficult to rewind in your information architecture.
• Regular testing: Usability metrics can change over time as user behavior and familiarity with digital products evolve. You should also test and analyze the usability of new feature releases on your website or app.

‍

Remember, data analysis is only as good as the data itself. Give yourself the best chance of designing exceptional user experiences by collecting, researching, and analyzing meaningful and accurate usability metrics.

‍

Conclusion

‍

Usability metrics are a guiding light when it comes to user experience. As the old saying goes, “you can’t manage what you can’t measure”. By including usability metrics in your design process, you invite direct user feedback into your product. This is ideal because we want to leave any assumptions or guesswork about user experience at the door.

‍

User-centered design inherently relies on constant user research. Usability metrics such as success rate, time-based efficiency, number of errors, and post-task satisfaction will highlight potential shortcomings in your design. Subsequently, they identify where improvements can be made, AND they lay down a benchmark to check whether any resulting updates addressed the issues.

‍

Ready to start collecting and analyzing usability metrics? Check out our guide to planning and running effective usability tests to get a head start!

The Age of the Customer is well and truly here. In every industry and vertical across the globe, UX professionals now dictate the terms, placing customers at the heart of every design decision. Or at least, this is the new reality that’s unfolding in the organizations and businesses that don’t want to be left behind.

Make no mistake; simply claiming to be the best is no longer enough. To survive and thrive, people need to be placed at the heart. The golden key that will allow organizations to pivot to this new reality lies with that of the user researcher.

But it’s not enough to simply “do user research”. Sure, some customer insight is obviously better than none at all, but to really be useful it needs to be effective research. That’s what this article is all about.

Get comfortable, because this is going to be a long one – for good reason.

Why (effective) user research is so important

You are not your user. As much as you may like to think that you are, you’re not. It can be a tricky proposition to get your head around, especially when we regularly assume that everyone thinks like us. There are 8 billion people out there who have a vastly different set of experiences and perspectives than you. With that in mind, when we start to generalize based on our own personal experiences, this is what’s known as availability bias.

Unfortunately, solving this is issue not as easy as getting into a room with customers and having a chat. People don’t always tell the truth! This isn’t to say that the participant in your last user interview was flat out lying to you, but the things that people say are different from the things that people do. It;’s your job (as a user researcher) to intuit the actual behaviors and actions, and identify their needs based on this data.

When you’re doing your job correctly, you’ve given your organization the best possible chance of success. Everything  – and I mean everything – starts with a solid understanding of your users. Doors will open, paths will reveal themselves – you get the idea.

The qualities of an effective user researcher

Let me preface this section by saying that you don’t have to have all of these qualities in spades, the list below is really just a way for you to better understand some of the traits of an effective user researcher, to get you thinking and on the right path.

• Curious: User research can be quite repetitive, especially when you get to the 6th user interview and need to ask the same questions. A genuine curiosity about people, the challenges they face and their behaviors will go a long way in helping you to push through.
• Pragmatic: Being an idealist has its uses, but it’s also important to be pragmatic. As a researcher, you need to operate on a fine line and balance your capacity to do research with business goals, finances and the desires of your stakeholders. Do the most with what you’ve got.
• Organized: It takes a lot to plan a research project, from scheduling testing sessions to assembling large slide decks for presentations. You’ve got to manage a large number of complex components, so it’s important that you can organize and prioritize.
• Collaborative: User research is most effective when it’s carried out collaboratively. This means working with your team, with the organization and with other disciplines. Think outside the box: Who stands to benefit from your research and how can you involve them?
• Empathetic: Real, natural empathy is a rare trait, but adopting an empathetic mindset is something everyone can (and should) learn. Beyond just uncovering insights from your participants, consider what these insights mean and how they all connect. This will truly enable you to understand your users.
• Sociable: You don’t have to suddenly adopt an extroverted persona, but being actively interested in other people will help you build relationships both inside your organization and with customers.
• Perceptive: User research means listening and observing. During a user interview or usability test, you need to be able to filter all of the data entering your mind and extract the most relevant insights.
• Analytical: In a similar vein to perceptiveness, being analytical is also key if you want to understand all of the data that your research will produce. Filter, examine, extract and move on.

How to run user research effectively (and at a low cost)

There are innumerable methods for user research, but many are resource- and time-intensive. What’s more, certain research methods come attached with significant costs.

But, research doesn’t have to be the time and money sink that it can often first appear to be. Certain actions before you ever step into the room with a participant can make a world of difference.

Conduct research at the start

User research is obviously valuable whenever you do it, but you’ll see the biggest impact when you carry it out right the start of a project. Conduct research to get the lay of the land; to learn how and why customers make certain decisions, and where the biggest opportunities lie.

Note: Don’t research in a silo, involve your team, stakeholders and other interested parties.

Have clear goals – and a plan

Every research project needs a clear objective, and that comes from a detailed UX research plan, which includes well-formulated research questions. Every project will have a different question, but they’re the best starting point to ensure research success.

Choose the right methods

There’s no shortage of research methods to choose from, but being an effective user researcher is all about being able to pick the right methods for each project, and use them correctly. Nearly every research project will benefit from using a combination of qualitative and quantitative methods in order to generate the most useful insights.

To understand which method to use, it’s a good idea to view them using the following framework:

Involve stakeholders

Bring stakeholders into your research project as early as possible. These are the people that will end up utilizing the results of your work, and chances are they’re the ones who’ll have the most questions at the end. Involve them through consultation, regular updates, the all-too-important presentation at the end of the project and by letting them take notes for you during research sessions.

Wrap up

It’s not enough to simply run a card sort now (although that’s still a very useful exercise). You need to think cohesively about the role of your research in your organization and make sure that you’re as aware of your bias as you are of the various methods and tools available to you. Happy researching!