October 3, 2022

Unmoderated usability testing: a checklist

In-person moderated user testing is a valuable part of any research project. Meaning you can see first-hand how your users interact with your prototypes and products. But in-person isn’t always a viable option. What to do if your project needs user testing but it’s just not possible to get in front of your users personally? 

Let’s talk unmoderated user testing. This approach sidesteps the need to meet your participants face-to-face as it’s done entirely remotely, over the internet. By it’s very nature there are also considerable benefits to unmoderated user testing.

What is unmoderated user testing? 💻👀

In the most basic sense, unmoderated user testing removes the ‘moderated’ part of the equation. Instead of having a facilitator guide participants through the test, participants complete the testing activity by themselves and in their own time. For the most part, everything else stays the same.

The key differences are:

  • You can’t ask follow-up questions
  • You can’t use low-fidelity prototypes
  • You can’t support participants (beyond the initial instructions you send them).

Is unmoderated user testing right for your research project?

By nature, unmoderated user research does not include any direct interaction between the researcher and the study participants. This is really the biggest benefit and also the biggest drawback. 

Benefits of unmoderated usability testing 👩🏻💻

  • Speed and turnaround  - As there is no need to schedule meetings with each participant, unmoderated testing is usually much faster to initiate and complete. Depending on the study, it may be possible to launch a study and receive results in just a few hours.
  • Size of study (participant numbers) - Unmoderated user testing also allows you to collect feedback from dozens or even hundreds of users at the same time.
  • Location (local and/or international) -Testing online removes reliance on participants being physically present for the testing which broadens the ability to make contact with participants within your country or across the globe. 

If you’d like to know more about the benefits of unmoderated usability testing, take a look at our article five reasons you should consider unmoderated user testing.

Limitations of unmoderated usability testing 🚧

  • Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype.
  • Participant behavior - Without a moderator, participants tend to be less engaged and behave less realistically in tasks that depend on imagination, decision-making, or emotional responses.
  • Inability to ask follow-up questions - by not being in the testing with the participant, the facilitator can’t ask further questions to get a deeper understanding of the participant’s reasoning. As you can’t rely on human judgment through a moderator being in the room with the participants and the ability to adjust the test in the moment, unmoderated usability testing needs thorough up front planning.

Because of these limitations unmoderated usability testing usually works best for evaluating live websites and apps or highly functional prototypes.  It’s great for testing activities that don’t require a lot of imagination or emotion from participants. Such as testing functionality or answering direct queries to do with your product.

What’s involved when setting up unmoderated usability testing? 🤔💭

  1. Define testing goals

With any usability testing, it pays to define your goals before getting underway with setting up the software. What do you want to know from the participants? Goals vary from test to test. Understanding your goals upfront will help you to make the correct tool choice.


  1. Define your demographic

With a clear understanding of your goal, now it’s time to consider which participants are right for your study. Think about who they are, their demographic, and where they live. Are they new users or existing? Are they experts or novices?

  1. Selecting testing software

As unmoderated studies, are done remotely, the software used to faciliate the study plays a key role in ensuring you get useful results. Without a facilitator, the software must guide the participants through the session and record what happens. Take the time to test software and select one that is right for your study.

  1. Write your own tasks and questions

Think through your goals and what you want to achieve from the testing. Many of the unmoderated testing services include study templates with generic example tasks. Remember they are templates, and your tasks and questions should be specific to your particular study. Any task instructions guiding the participants should be clear and directive.

  1. Trial session

You’ve done all of the upfront work, now it’s time to test that it works, the software does what you expect and the instructions you have written can be followed. Doing a test run is crucial, especially with unmoderated usability testing, as there won’t be a facilitator in the testing to fix any problems.

  1. Recruit participants

Having defined your target audience and demographic, now is the time to recruit participants. Ensuring you have some control over the recruitment process is important, either through screening questions or recruiting your own. There are services that  recruit from a pool of willing participants. Thiscan be a great way to get a wide range of users.

  1. Analyze results

You are likely to accumulate a lot of data from your unmoderated testing. You’ll need a way to organize and analyze the data to derive insights that are valuable. Depending on the type of usability testing you do will vary the type of results. Quantitative testing gives data-driven results and direct answers. Whereas qualitative testing through audio or video recordings of participants’ actions or comments will need time to analyze and look at behavioral observations. 

Wrap Up 🌯

Unmoderated usability testing can be a good option for your study. It may not be right for all of your studies all of the time. While it can be quick to implement and often cheaper than moderated usability testing, it still requires time and planning to ensure you get the data insights you are looking for. Following a checklist can be a great way to ensure you approach your research methodically.

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

Why information architecture is important for designers

Sitting inside any beautifully crafted and designed digital product, there must be a fully functional and considered information architecture.

As much as information architecture shouldn’t be developed in a vacuum. Neither should the design and look of digital products. In fact, a large proportion of the function of digital designers is devoted to supporting users locating content they need and driving them towards content that the product owners want them to find.

Incorporating visual markers to make sure that certain content is distinct from the rest or creating layers that demonstrate the diverse content on a product.

If you do not have quality content, it is impossible to design a quality digital product. It all comes back to creating a user experience that makes sense and is designed to make task completion simple. And this relates back to designing the product with the content planned for it in mind.

8 Principles of information architecture, according to Dan Brown 🏗️

As a designer, the more you know about information architecture, the better the products you design will meet your user requirements and deliver what they need. If you work with an information architect, even better. If you’re still learning about information architecture the 8 Principles according to Dan Brown is a great place to begin.

If you haven’t come across Dan Brown yet, you have more than likely come across his 8 principles. Dan Brown is one of the UX world's most prolific experts with a career that spans most areas of UX designs. He’s written 3 books on the subject and experience across a multitude of high profile projects. Aiding large organizations to make the most of their user experience.

  1. The principle of objects: Content should be treated as a living, breathing thing. It has lifecycles, behaviors, and attributes.
  2. The principle of choices: Less is more. Keep the number of choices to a minimum.
  3. The principle of disclosure: Show a preview of information that will help users understand what kind of information is hidden if they dig deeper.
  4. The principle of examples: Show examples of content when describing the content of the categories.
  5. The principle of front doors: Assume that at least 50% of users will use a different entry point than the home page.
  6. The principle of multiple classifications: Offer users several different classification schemes to browse the site’s content.
  7. The principle of focused navigation: Keep navigation simple and never mix different things.
  8. The principle of growth: Assume that the content on the website will grow. Make sure the website is scalable.

It’s highly likely that you’ve already used some, or all, of these IA principles in your designs. Don’t be shy about mastering them, or at the very least be familiar. They can only help you become a better user experience designer.

Wrap up 🌯

Mastering the 8 principles, according to IA expert Dan Brown will see you mastering the complex tasks of information architecture. Understanding IA is key to creating digital designs with a content structure that is functional, logical and just what your users need to navigate your product. Design without good IA doesn’t work as well, just as a content structure without a well designed interface will not engage users.

Learn more
1 min read

Affinity mapping - an introduction

User research is key to discovering the inner workings of your users’ minds – their emotional, organizational, informative needs and desires. These are all super important to creating a user experience that is intuitive and meeting your users’ needs in a way that means they feel loved, cared for and considered. All the deep understanding stuff that keeps them coming back!

Qualitative research allows you to collect verbatim data from participants that give insights into why they do or feel things. You can even get into whether ‘Dee’ understood how the website worked or why ‘Andrew’ would (or wouldn’t) revisit the app outside of testing.

Gathering these awesome insights is one step. Analyzing and organizing these is a skill and talent in its own right. And armed with the right tools or methods it can be immersive, interesting and a great way to get under the skin of your users. Let’s take a look at affinity mapping as a method of analyzing this data - as a tool it can help researchers visualize and easily group and theme data.

Affinity mapping is used outside of the UX world and can be done independently, however is a great analysis method to use collaboratively. For researchers, it can be a great tool to collaborate and engage the team and potentially stakeholders. Bringing people together to identify, discuss and resolve user experience issues. 

Here we’ll lay out what affinity mapping is, specifically why it’s useful for user research and set out key steps to get you underway. 

What is Affinity Mapping? 🗺️

By definition, affinity mapping is the process of collecting, organizing, and grouping qualitative data to create an affinity diagram.

Put simply it is a tool to group, map, sort and categorize information. A tool where you’ll look at the information and patterns of your qualitative user research and work to group these together to make sense of them. It helps you to find patterns, similar outcomes and insights that allow you to draw conclusions and collate results in a cohesive manner, then report to the wider team in a way that makes sense and provides a clear road to applicable and achievable outcomes.

What is an Affinity Diagram? 🖼️

An affinity diagram is what you have once you have gone through the affinity mapping process. It is the final ‘diagram’ of your grouping, sorting and categorizing. An ordered visual sorting of insights and information from your user research. And the place to filter or funnel observations and information into patterns and reach final outcomes. 

Allowing you to see where the key outtakes are and where there may need to be improvements, changes or updates. And from here a roadmap can be decided.

An affinity map using Reframer by Optimal Workshop

Essentially the mapping part is the process of creating the diagram, a visual sorting of insights and information from your user research. So how do you make affinity mapping work for you?

1. Start with a large space

This could be a table, desk, pinboard or even a whiteboard. Somewhere that you can stick, pin or attach your insights to in a collaborative space. Becoming more common recently is the use of shared digital and online whiteboard tools.  allowing people to access and participate remotely.

2. Record all notes

Write observations, thoughts, research insights on individual cards or sticky notes.

3. Look for patterns

As a group read, comment and write notes or observations. Stick each of the notes onto the board, desk or whiteboard. Add, and shuffle into groups as you go. You can keep adding or moving as you go.

4. Create a group/theme

This will start to make sense as more sticky notes are added to the map. Creating groups for similar observations or insights, or for each pattern or theme.

Create a group/theme using affinity mapping

5. Give each theme or group a name

As more notes are added there will be natural groups formed. Openly discuss if there are notes that are more difficult to categorize or themes to be decided. (We’ve outlined some ideas for UX research themes in another section below.)

6. Determine priorities

You’ve tidied everything into themes and groups, now what? How do you decide which of these are priorities for your organization? Discussion and voting can be the best way to decide what outcomes make the most sense and may have the biggest impact on your business.

7. Report on your findings

Pulling together and reporting on the findings through your affinity diagram process should be key to putting actionable outcomes in place.

How to define research themes 🔬

Commonly, user research is digested through thematic analysis. During thematic analysis, you aim to make sense of all the notes, observations, and discoveries you’ve documented across all your information sources, by creating themes to organize the information. 

Depending on your role and the type of research you conduct, the themes you create for your affinity diagram can vary. Here are some examples of affinity groups that you could form from your UX research:

  • User sentiment and facial expressions when completing certain tasks
  • Frequently used words or phrases when describing a product or experience
  • Suggestions for improving your product or experience

Wrap up 🌯

Qualitative user testing and the resulting observations can be some of the best insights you get into your users’ minds. Filtering, organizing and ordering these disparate and very individual observations can be tricky. Especially if done in silo.

So, draw a team together, bring in stakeholders from throughout your organization and work collaboratively to sort, organize and categorize through affinity mapping. This opens the doors to discussion, buy-in and ultimately a collective understanding of user research. Its importance and its role within the organization. And most importantly the real-world implications UX research and its insights have on organizational products and output.

Learn more
1 min read

Why information architecture is the foundation of UX

Ever wondered what the relationship is between information architecture (IA) and UX? Simply put, IA is the foundation of UX. We outline why.

What is Information Architecture? 🛠️

According to Abby Covert, a leader in the field of information architecture, IA is ‘the way we arrange the parts to make sense of the whole.’ This can relate to a website, a retail store or an app. And you could even consider the way a library is sorted to be information architecture. For the purposes of this article, we will focus on digital products (apps or websites).

Well-organized information architecture is fundamentally important to the success of your product. As a designer, knowing the content you are delivering and how, is fundamental to creating a UX that performs. Working with the needs of the organization and meeting the requirements of the users in a meaningful and delightful way. Organizing and structuring the information so that navigating, searching, and understanding your product is seamless is ultimately what UX design is all about. Arranging the parts to make sense of the whole, you could say.

While design is about creating visual pointers for users to find their way, information architecture can be broken down into 3 main areas to consider when building a great user experience:

  • Navigation: How people make their way through information 
  • Labels: How information is named and represented.
  • Search: How people will look for information (keywords, categories)

When put like this it does seem pretty straightforward. Maybe even simple? But these tasks need to be straightforward for your users. Putting thought, time, and research at the front of your design and build can increase your chances of delivering an intuitive product. In fact, at any point in your product’s life cycle, it’s worth testing and reviewing these 3 areas.  

Key things to consider to build an effective IA for UX 🏗

Developing a well-thought-out and researched information architecture for your product could be considered a foundation step to creating a great UX product. To help you on your way, here are 6 key things to consider when building effective information architecture for a great user experience. 

  1. Define the goals of your organization: Before starting your IA plan, uncover what is the purpose of your product and how this will align with the goals of your stakeholders.
  2. Figure out your user’s goals: Who do you want to use your product? Create scenarios, discuss with probable users and find out what they’ll use your product for and how they’ll use it.  
  3. Study your competitors: Take note of websites, apps and other digital products that are similar to yours and look at their information architecture from a UX point of view. How does the design work with the IA. Is it simple to navigate? Easy to find what to do next?  Look at how key information is designed and displayed.
  4. Draw a site map: Once the IA is planned and developed and the content is ready, it’s time to figure out how users are going to access all of your information. Spend time planning navigation that is not too complex that will help users to browse your product easily. 
  5. Cross browser testing: Your information architecture behavior may vary from one browser so it’s worth doing some cross-browser compatibility testing. It would be very disappointing to work so hard to get the best UX with your product, only to be let down because of browser variances.
  6. Usability testing: End users are the perfect people to let you know how your product is performing. Set up a testing/staging environment and test on external users. Observing your participants while they move their way through your product uninterrupted and listening to their opinions can shed light on the successes (and failures) in a very insightful way. 

Wrap Up 🌯

Information architecture is the foundation of designing a great product that meets (or even exceeds) your users’ needs, wants, and desires. By balancing an organization’s needs with insight into what users actually want, you’re well equipped to design an information architecture  that helps build a product that delivers a positive user experience. Research, test, research, and test again should be the mantra throughout the development, design, and implementation of your product and beyond.

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.