January 13, 2016
5 min

A comprehensive look at usability testing

Usability testing has an important role in UX and if you’re new to it, this article gives you a solid introduction to it with practical tips, a checklist for success and a look at our remote testing tool, Treejack.

Concepts Of usability testing 👩🏾💻

Usability testing is the process of evaluating a product or service with users prior to implementation. The goal of usability testing is to identify any usability issues before the product or service is released into the world for use. Usability testing is a research activity that results in both quantitative and qualitative insights and can be used to gauge user satisfaction.A typical usability testing session is moderated and involves a participant, a facilitator and an observer. The facilitator leads the session and the observer takes notes while the participant completes the task based scenario.

While this is common, usability testing is scalable and the possible approaches are endless giving you the flexibility to work with the resources you have available—sometimes one person performs the role of facilitator and observer!Location also varies for usability testing. For example, you might conduct your testing in a lab environment or you might talk to users in a specific environment. It also worth noting that not all usability testing sessions are moderated—more about this later.Usability testing usually occurs multiple times during the design process and can be conducted anytime you have a design you would like to test.

User research activities like a focus group for example, are conducted early in the design process to explore and gain understanding before ideas are generated. Usability testing is about testing those ideas as early and as often as possible. From a fully functioning digital prototype to a simple hand drawn wireframe on paper, nothing is too unrefined or too rough to be tested.

Developing a usability test plan 🛠️

Before you start a round of usability testing, you need to develop a usability test plan. The usability test plan will keep you organised and is an opportunity to define roles and set clear expectations upfront. The first step in developing this is to hold a meeting with your team and stakeholders to discuss what you are going to do and how you plan to achieve it. Following this meeting, a document outlining the usability test plan as it was discussed is created and shared with the group for review. Any changes suggested by the group are then added to the final document for approval from the relevant stakeholders.

What to include in your usability test plan:

  • The goal, scope and intent of the usability testing
  • Constraints impacting upon testing
  • Details on what will be tested eg wireframes
  • Schedule and testing location
  • Associated costs eg participant recruitment
  • Facilitator and observer details for each session
  • Session details
  • Participant recruitment approach
  • Equipment
  • Details of any documentation to be produced eg a report

Usability testing questions 🤔

Once you have developed your test plan, you need to create a list of questions and task based scenarios for the testing session. These form the structure for your testing and provide a framework of consistency across all testing sessions in the study.The questions serve as a warm up to ease the participant into the session and can also provide insights on the user that you may not have had before. These questions can be a combination of open and closed questions and are especially useful if you are also developing personas for example. Some examples of what you might ask include:

  • Tell me about a recent experience you had with this product/service
  • Do you currently use this product/service?
  • Do you own a tablet device?

The purpose of the task based scenarios is to simulate a real life experience as closely as possible. They provide a contextual setting for the participant to frame their approach and they need to be realistic—your participant needs an actionable starting point to work from. A good starting point for task based scenario development would be to look at a use case.It is also important that you avoid using language that provides clues to the solution or leads your participant as this can produce inaccurate results. An example of a task based scenario would be:You’re planning a Christmas vacation to New Zealand for your family of two adults and 4 children. Find the lowest priced airfares for your trip.

Usability testing software: Tree testing 🌲🌳🌿

Treejack is a remote information architecture (IA) validation tool that shows you exactly where users are getting lost in your content. Knowing this will enable you to design a structure for your website that makes sense to users before moving on to the user interface (UI) design.Treejack works like a card sort in reverse. Imagine you have just completed a card sort with your users to determine your IA and you are now working backwards to test that thinking against real world task based scenarios. Treejack does this using a text-based version of your IA that is free from distracting visual aids such as navigation and colour allowing you to determine if your structure is usable from the ground up. A Treejack study is structured around task based scenarios and comes with the option to include pre and post study questionnaires.

treejack task image
Usability testing with Treejack

As a remote testing tool, Treejack is unmoderated and provides the opportunity to reach a much larger audience because all you have to do is share a link to the study with your participants to gain insights. You also have the option of handing the task of targeted participant recruitment over to Optimal Workshop.Once launched and shared with participants, Treejack takes care of itself by recording the results as they come in giving you the freedom to multitask while you wait for the testing to finish.

The results produced by Treejack are not only detailed and comprehensive but are also quite beautiful. The story of your participants’ journey through your testing activity is told through pietrees. A pietree is a detailed pathway map that shows where your participants went at each fork in the road and their destinations. They allow you to pinpoint exactly where the issues lie and are a powerful way to communicate the results to your team and stakeholders.

bananacom pie tree
Treejack presents your results using pietrees

Treejack also provides insights into where your participants landed their first click and records detailed information on pathways followed by each individual participant.

bananacom paths
Treejack records full details of the paths followed by every participant

Usability testing checklist 📋

The following checklist will help ensure your usability testing process runs smoothly:

  • Meet with team and stakeholders
  • Determine goals, scope and intent of usability testing
  • Decide how many sessions will be conducted
  • Create usability testing schedule
  • Select facilitators and observers for each session if applicable
  • Develop and complete a usability test plan
  • Determine test questions and scenarios
  • Recruit participants for testing
  • Gather equipment required for testing if applicable
  • Book testing location if applicable
  • Keep a list of useful contact details close by in case you need to contact anyone during testing
  • Complete a dry run of a testing session with a team member to ensure everything works before actual testing begins
  • Organise debrief meetings with observers after each testing session
  • Set aside time to analyse the findings
  • Document and present findings to team and relevant stakeholders

Related reading 📚🎧☕

Share this article
Author
Optimal
Workshop

Related articles

View all blog articles
Learn more
1 min read

A comprehensive look at usability testing

Usability testing has an important role in UX and if you’re new to it, this article gives you a solid introduction to it with practical tips, a checklist for success and a look at our remote testing tool, Treejack.

Concepts Of usability testing 👩🏾💻

Usability testing is the process of evaluating a product or service with users prior to implementation. The goal of usability testing is to identify any usability issues before the product or service is released into the world for use. Usability testing is a research activity that results in both quantitative and qualitative insights and can be used to gauge user satisfaction.A typical usability testing session is moderated and involves a participant, a facilitator and an observer. The facilitator leads the session and the observer takes notes while the participant completes the task based scenario.

While this is common, usability testing is scalable and the possible approaches are endless giving you the flexibility to work with the resources you have available—sometimes one person performs the role of facilitator and observer!Location also varies for usability testing. For example, you might conduct your testing in a lab environment or you might talk to users in a specific environment. It also worth noting that not all usability testing sessions are moderated—more about this later.Usability testing usually occurs multiple times during the design process and can be conducted anytime you have a design you would like to test.

User research activities like a focus group for example, are conducted early in the design process to explore and gain understanding before ideas are generated. Usability testing is about testing those ideas as early and as often as possible. From a fully functioning digital prototype to a simple hand drawn wireframe on paper, nothing is too unrefined or too rough to be tested.

Developing a usability test plan 🛠️

Before you start a round of usability testing, you need to develop a usability test plan. The usability test plan will keep you organised and is an opportunity to define roles and set clear expectations upfront. The first step in developing this is to hold a meeting with your team and stakeholders to discuss what you are going to do and how you plan to achieve it. Following this meeting, a document outlining the usability test plan as it was discussed is created and shared with the group for review. Any changes suggested by the group are then added to the final document for approval from the relevant stakeholders.

What to include in your usability test plan:

  • The goal, scope and intent of the usability testing
  • Constraints impacting upon testing
  • Details on what will be tested eg wireframes
  • Schedule and testing location
  • Associated costs eg participant recruitment
  • Facilitator and observer details for each session
  • Session details
  • Participant recruitment approach
  • Equipment
  • Details of any documentation to be produced eg a report

Usability testing questions 🤔

Once you have developed your test plan, you need to create a list of questions and task based scenarios for the testing session. These form the structure for your testing and provide a framework of consistency across all testing sessions in the study.The questions serve as a warm up to ease the participant into the session and can also provide insights on the user that you may not have had before. These questions can be a combination of open and closed questions and are especially useful if you are also developing personas for example. Some examples of what you might ask include:

  • Tell me about a recent experience you had with this product/service
  • Do you currently use this product/service?
  • Do you own a tablet device?

The purpose of the task based scenarios is to simulate a real life experience as closely as possible. They provide a contextual setting for the participant to frame their approach and they need to be realistic—your participant needs an actionable starting point to work from. A good starting point for task based scenario development would be to look at a use case.It is also important that you avoid using language that provides clues to the solution or leads your participant as this can produce inaccurate results. An example of a task based scenario would be:You’re planning a Christmas vacation to New Zealand for your family of two adults and 4 children. Find the lowest priced airfares for your trip.

Usability testing software: Tree testing 🌲🌳🌿

Treejack is a remote information architecture (IA) validation tool that shows you exactly where users are getting lost in your content. Knowing this will enable you to design a structure for your website that makes sense to users before moving on to the user interface (UI) design.Treejack works like a card sort in reverse. Imagine you have just completed a card sort with your users to determine your IA and you are now working backwards to test that thinking against real world task based scenarios. Treejack does this using a text-based version of your IA that is free from distracting visual aids such as navigation and colour allowing you to determine if your structure is usable from the ground up. A Treejack study is structured around task based scenarios and comes with the option to include pre and post study questionnaires.

treejack task image
Usability testing with Treejack

As a remote testing tool, Treejack is unmoderated and provides the opportunity to reach a much larger audience because all you have to do is share a link to the study with your participants to gain insights. You also have the option of handing the task of targeted participant recruitment over to Optimal Workshop.Once launched and shared with participants, Treejack takes care of itself by recording the results as they come in giving you the freedom to multitask while you wait for the testing to finish.

The results produced by Treejack are not only detailed and comprehensive but are also quite beautiful. The story of your participants’ journey through your testing activity is told through pietrees. A pietree is a detailed pathway map that shows where your participants went at each fork in the road and their destinations. They allow you to pinpoint exactly where the issues lie and are a powerful way to communicate the results to your team and stakeholders.

bananacom pie tree
Treejack presents your results using pietrees

Treejack also provides insights into where your participants landed their first click and records detailed information on pathways followed by each individual participant.

bananacom paths
Treejack records full details of the paths followed by every participant

Usability testing checklist 📋

The following checklist will help ensure your usability testing process runs smoothly:

  • Meet with team and stakeholders
  • Determine goals, scope and intent of usability testing
  • Decide how many sessions will be conducted
  • Create usability testing schedule
  • Select facilitators and observers for each session if applicable
  • Develop and complete a usability test plan
  • Determine test questions and scenarios
  • Recruit participants for testing
  • Gather equipment required for testing if applicable
  • Book testing location if applicable
  • Keep a list of useful contact details close by in case you need to contact anyone during testing
  • Complete a dry run of a testing session with a team member to ensure everything works before actual testing begins
  • Organise debrief meetings with observers after each testing session
  • Set aside time to analyse the findings
  • Document and present findings to team and relevant stakeholders

Related reading 📚🎧☕

Learn more
1 min read

Using User Engagement Metrics to Improve Your Website's User Experience

Are your users engaged in your website? The success of your website will largely depend on your answer. After all, engaged users are valuable users; they keep coming back and will recommend your site to colleagues, friends, and family. So, if you’re not sure if your users are engaged or not, consider looking into your user engagement metrics.

User engagement can be measured using a number of key metrics provided by website analytics platforms. Metrics such as bounce rate, time on page, and click-through rate all provide clues to user engagement and therefore overall website user experience.

This article will help you understand user engagement and why it’s important to measure. We’ll also discuss how to apply user engagement insights to improve website success. Combining a little bit of data with some user research is a powerful thing, so let’s get into it.

Understanding User Engagement Metrics 📐

User engagement metrics provide valuable insight for both new and existing websites. They should be checked regularly as a sort of ‘pulse check’ for website user experience and performance. So, what metrics should you be looking at? Website metrics can be overwhelming; there are hundreds if not thousands to analyze, so let’s focus on three:

Bounce rate


Measures the percentage of users that visit just one page on your site before leaving. If your bounce rate is high it suggests that users aren’t finding the content relevant, engaging, or useful. It points to a poor initial reaction to your site and means that users are arriving, making a judgment about your design or content, and then leaving.

Time on page


Calculated by the time difference between the point when a person lands on the page and when they move on to the next one. It indicates how engaging or relevant individual pages on your website are. Low time on page figures suggest that users aren’t getting what they need from a certain page, either in terms of the content, the aesthetics, or both.

Click-through rate


Click-through rate compares the number of times someone clicks on your content, to the number of impressions you get (how many times an internal link or ad was viewed). The higher the rate, the better the engagement and performance of that element. User experience design can influence click-through rates through copywriting, button contrasts, heading structure, navigation, etc.

Conversion rate


Conversion rates are perhaps the pinnacle of user engagement metrics. Conversion rate is the percentage of users that perform specific tasks you define. They are therefore dictated by your goals, which could include form submissions, transactions, etc. If your website has high conversion rates, you can be fairly confident that your website is matching your users’ needs, requirements, and expectations.

But how do these metrics help? Well, they don’t give you an answer directly. The metrics point to potential issues with website user experience. They guide further research and subsequent updates that lead to website improvement. In the next section, we’ll discuss how these and others can support better website user experiences.

Identifying Areas for Improvement 💡

So, you’ve looked at your website’s user engagement metrics and discovered some good, and some bad. The good news is, there’s value in discovering both! The catch? You just need to find it. Remember, the metrics on their own don’t give you answers; they provide you direction.

The ‘clues’ that user engagement metrics provide are the starting point for further research. Remember, we want to make data-driven decisions. We want to avoid making assumptions and jumping to conclusions about why our website is reporting certain metrics. Fortunately, there are a bunch of different ways to do this.

User research data can be gathered by using both qualitative and quantitative research techniques. Insights into user behavior and needs can reveal why your website might be performing in certain ways. Research can include both qualitative and quantitative techniques.

Qualitative research techniques

  • Usability test – Test a product with people by observing them as they attempt to complete various tasks.
  • User interview – Sit down with a user to learn more about their background, motivations and pain points.
  • Contextual inquiry – Learn more about your users in their own environment by asking them questions before moving onto an observation activity.
  • Focus group – Gather 6 to 10 people for a forum-like session to get feedback on a product.

Quantitate research techniques

  • Card sorts – Find out how people categorize and sort information on your website.
  • First-click tests – See where people click first when tasked with completing an action.
  • A/B tests – Compare 2 versions of a design in order to work out which is more effective.
  • Clickstream analysis – Analyze aggregate data about website visits.
  • Tree-testing - Test your site structure using text-only categorization and labels

The type of research depends on what question you want to answer. Being specific about your question will help you identify what research technique(s) to deploy and ultimately the quality of your answer. If you’re serious about website improvement; identify problem areas with user engagement metrics, and investigate how to fix them with user research.

Optimizing Content and Design

If you have conducted user research and found weak areas on your website, there are many things to consider. Three good places to start are navigation, content, and website layout. Combined, these have a huge impact on user experience and can be leveraged to address disappointing engagement metrics.

Navigation


Navigation is a crucial aspect of creating a good user experience since it fundamentally connects pages and content which allows users to find what they need. Navigation should be simple and easy to follow, with important information/actions at the top of menus. Observing the results of card sorting, tree testing, and user testing can be particularly useful in website optimization efforts. You may find that search bars, breadcrumb trails, and internal links can also help overcome navigation issues.

Content


Are users seeing compelling or relevant content when they arrive on your site? Is your content organized in a way that encourages further exploration? Card sorting and content audits are useful in answering these questions and can help provide you with the insights required to optimize your content. You should identify what content might be redundant, out of date, or repetitive, as well as any gaps that may need filling.

Layout


A well-designed layout can improve the overall usability of a website, making it easier for users to find what they're looking for, understand the content, and engage with it. Consider how consistent your heading structures are and be sure to use consistent styling throughout the site, such as similar font sizes and colors. Don’t be afraid to use white space; it’s great at breaking up sections and making content more readable.

An additional factor related to layout is mobile optimization. Mobile-first design is necessary for apps, but it should also factor into your website design. How responsive is your website? How easy is it to navigate on mobile? Is your font size appropriate? You might find that poor mobile experience is negatively impacting user engagement metrics.

Measuring Success 🔎

User experience design is an iterative, ongoing process, so it’s important to keep a record of your website’s user experience metrics at various points of development. Fortunately, website analytics platforms will provide you with historic user data and key metrics; but be sure to keep a separate record of what improvements you make along the way. This will help you pinpoint what changes impacted different metrics.

Define your goals and create a website optimization checklist that monitors key metrics on your site. For example, whenever you make an update, ensure bounce rates don’t exceed a certain number during the days following; check that your conversion rates are performing as they should be; check your time on sites hasn’t dropped. Be sure to compare metrics between desktop and mobile too.

User’s needs and expectations change over time, so keep an eye on how new content is performing. For example, which new blog posts have attracted the most attention? What pages or topics have had the most page views compared to the previous period? Tracking such changes can help to inform what your users are currently engaged in, and will help guide your user experience improvements.

Conclusion 🤗

User engagement metrics allow you to put clear parameters around user experience. They allow you to measure where your website is performing well, and where your website might need improving. Their main strength is in how accessible they are; you can access key metrics on website analytics platforms in moments. However, user engagement metrics on their own may not reveal how and why certain website improvements should be made. In order to understand what’s going on, you often need to dig a little deeper.

Time on page, bounce rate, click-through rate, and conversion rates are all great starting points to understand your next steps toward website improvement. Use them to define where further research may be needed. Not sure why your average pages per session is two? Try conducting first-click testing; where are they heading that seems to be a dead end? Is your bounce rate too high? Conduct a content audit to find out if your information is still relevant, or look into navigation roadblocks. Whatever the question; keep searching for the answer.

User engagement metrics will keep you on your toes, but that’s a good thing. They empower you to make ongoing website improvements and ensure that users are at the heart of your website design. 

Learn more
1 min read

The Role of Usability Metrics in User-Centered Design

The term ‘usability’ captures sentiments of how usable, useful, enjoyable, and intuitive a website or app is perceived by users. By its very nature, usability is somewhat subjective. But what we’re really looking for when we talk about usability is how well a website can be used to achieve a specific task or goal. Using this definition we can analyze usability metrics (standard units of measurement) to understand how well user experience design is performing.

Usability metrics provide helpful insights before and after any digital product is launched. They help us form a deeper understanding of how we can design with the user front of mind. This user-centered design approach is considered the best-practice in building effective information architecture and user experiences that help websites, apps, and software meet and exceed users' needs.

In this article, we’ll highlight key usability metrics, how to measure and understand them, and how you can apply them to improve user experience.

Understanding Usability Metrics

Usability metrics aim to understand three core elements of usability, namely: effectiveness, efficiency, and satisfaction. A variety of research techniques offer designers an avenue for quantifying usability. Quantifying usability is key because we want to measure and understand it objectively, rather than making assumptions.

Types of Usability Metrics

There are a few key metrics that we can measure directly if we’re looking to quantify effectiveness, efficiency, and satisfaction. Here are four common examples:

  • Success rate: Also known as ‘completion rate’, success rate is the percentage of users who were able to successfully complete the tasks.
  • Time-based efficiency: Also known as ‘time on task’, time-based efficiency measures how much time a user needs to complete a certain task.
  • Number of errors: Sounds like what it is! It measures the average number of times where an error occurred per user when performing a given task.
  • Post-task satisfaction: Measures a user's general impression or satisfaction after completing (or not completing) a given task.

How to Collect Usability Metrics


Usability metrics are outputs from research techniques deployed when conducting usability testing. Usability testing in web design, for example, involves assessing how a user interacts with the website by observing (and listening to) users completing defined tasks, such as purchasing a product or signing up for newsletters.

Conducting usability testing and collecting usability metrics usually involves:

  • Defining a set of tasks that you want to test
  • Recruitment of test participants
  • Observing participants (remotely or in-person)
  • Recording detailed observations
  • Follow-up satisfaction survey or questionnaire

Tools such Reframer are helpful in conducting usability tests remotely, and they enable live collaboration of multiple team members. It is extremely handy when trying to record and organize those insightful observations! Using paper prototypes is an inexpensive way to test usability early in the design process.

The Importance of Usability Metrics in User-Centered Design

User-centered design challenges designers to put user needs first. This means in order to deploy user-centered design, you need to understand your user. This is where usability testing and metrics add value to website and app performance; they provide direct, objective insight into user behavior, needs, and frustrations. If your user isn’t getting what they want or expect, they’ll simply leave and look elsewhere.

Usability metrics identify which parts of your design aren’t hitting the mark. Recognizing where users might be having trouble completing certain actions, or where users are regularly making errors, are vital insights when implementing user-centered design. In short, user-centered design relies on data-driven user insight.

But why hark on about usability metrics and user-centered design? Because at the heart of most successful businesses is a well-solved user problem. Take Spotify, for example, which solved the problem of dodgy, pirated digital files being so unreliable. People liked access to free digital music, but they had to battle viruses and fake files to get it. With Spotify, for a small monthly fee, or the cost of listening to a few ads, users have the best of both worlds. The same principle applies to user experience - identify recurring problems, then solve them.

Best Practices for Using Usability Metrics

Usability metrics should be analyzed by design teams of every size. However, there are some things to bear in mind when using usability metrics to inform design decisions:

  • Defining success: Usability metrics are only valuable if they are being measured against clearly defined benchmarks. Many tasks and processes are unique to each business, so use appropriate comparisons and targets; usually in the form of an ‘optimized’ user (a user with high efficiency).
  • Real user metrics: Be sure to test with participants that represent your final user base. For example, there’s little point in testing your team, who will likely be intimately aware of your business structure, terminology, and internal workflows.
  • Test early: Usability testing and subsequent usability metrics provide the most impact early on in the design process. This usually means testing an early prototype or even a paper prototype. Early testing helps to avoid any significant, unforeseen challenges that could be difficult to rewind in your information architecture.
  • Regular testing: Usability metrics can change over time as user behavior and familiarity with digital products evolve. You should also test and analyze the usability of new feature releases on your website or app.

Remember, data analysis is only as good as the data itself. Give yourself the best chance of designing exceptional user experiences by collecting, researching, and analyzing meaningful and accurate usability metrics.

Conclusion

Usability metrics are a guiding light when it comes to user experience. As the old saying goes, “you can’t manage what you can’t measure”. By including usability metrics in your design process, you invite direct user feedback into your product. This is ideal because we want to leave any assumptions or guesswork about user experience at the door.

User-centered design inherently relies on constant user research. Usability metrics such as success rate, time-based efficiency, number of errors, and post-task satisfaction will highlight potential shortcomings in your design. Subsequently, they identify where improvements can be made, AND they lay down a benchmark to check whether any resulting updates addressed the issues.

Ready to start collecting and analyzing usability metrics? Check out our guide to planning and running effective usability tests to get a head start!

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.