A short guide to personas

"Dear Optimal Workshop

How do I quantify the ROI [return on investment] of investing in user experience?"

— Brian

‍

Dear Brian,

I'm going to answer your question with a resounding 'It depends'. I believe we all differ in what we're willing to invest, and what we expect to receive in return. So to start with, and if  you haven’t already, it's worth grabbing your stationery tools of choice and brainstorming your way to a definition of ROI that works for you, or for the people you work for.

‍

I personally define investment in UX as time given, money spent, and people utilized. And I define return on UX as time saved, money made, and people engaged. Oh, would you look at that — they’re the same! All three (time, money, and humans) exist on both sides of the ROI fence and are intrinsically linked. You can’t engage people if you don’t first devote time and money to utilizing your people in the best possible way! Does that make sense?

‍

That’s just my definition — you might have a completely different way of counting those beans, and the organizations you work for may think differently again.

‍

I'll share my thoughts on the things that are worth quantifying (that you could start measuring today if you were so inclined) and a few tips for doing so. And I'll point you towards useful resources to help with the nitty-gritty, dollars-and-cents calculations.

‍

5 things worth quantifying for digital design projects

‍

Here are five things I think are worthy of your attention when it comes to measuring the ROI of user experience, but there are plenty of others. And different projects will most likely call for different things.

‍

(A quick note: There's a lot more to UX than just digital experiences, but because I don't know your specifics Brian, the ideas I share below apply mainly to digital products.)

‍

1. What’s happening in the call centre?

‍

A surefire way to get a feel for the lay of the land is to look at customer support — and if measuring support metrics isn't on your UX table yet, it's time to invite it to dinner. These general metrics are an important part of an ongoing, iterative design process, but getting specific about the best data to gather for individual projects will give you the most usable data.

‍

Improving an application process on your website? Get hard numbers from the previous month on how many customers are asking for help with it, go away and do your magic, get the same numbers a month after launch, and you've got yourself compelling ROI data.

‍

Are your support teams bombarded with calls and emails? Has the volume of requests increased or decreased since you released that new tool, product, or feature? Are there patterns within those requests — multiple people with the same issues? These are just a few questions you can get answers to.

‍

You'll find a few great resources on this topic online, including this piece by Marko Nemberg that gives you an idea of the effects a big change in your product can have on support activity.

‍

2. Navigation vs. Search

‍

This is a good one: check your analytics to see if your users are searching or navigating. I’ve heard plenty of users say to me upfront that they'll always just type in the search bar and that they’d never ever navigate. Funny thing is, ten minutes later I see the same users naturally navigating their way to those gorgeous red patent leather pumps. Why?

‍

Because as Zoltán Gócza explains in UX Myth #16, people do tend to scan for trigger words to help them navigate, and resort to problem solving behaviour (like searching) when they can’t find what they need. Cue frustration, and the potential for a pretty poor user experience that might just send customers running for the hills — or to your competitors. This research is worth exploring in more depth, so check out this article by Jared Spool, and this one by Jakob Nielsen (you know you can't go wrong with those two).

‍

3. Are people actually completing tasks?

‍

Task completion really is a fundamental UX metric, otherwise why are we sitting here?! We definitely need to find out if people who visit our website are able to do what they came for.

‍

For ideas on measuring this, I've found the Government Service Design Manual by GOV.UK to be an excellent resource regardless of where you are or where you work, and in relation to task completion they say:

‍

"When users are unable to complete a digital transaction, they can be pushed to use other channels. This leads to low levels of digital take-up and customer satisfaction, and a higher cost per transaction."

‍

That 'higher cost per transaction' is your kicker when it comes to ROI.

‍

So, how does GOV.UK suggest we quantify task completion? They offer a simple (ish) recommendation to measure the completion rate of the end-to-end process by going into your analytics and dividing the number of completed processes by the number of started processes.

‍

While you're at it, check the time it takes for people to complete tasks as well. It could help you to uncover a whole host of other issues that may have gone unnoticed. To quantify this, start looking into what Kim Oslob on UXMatters calls 'Effectiveness and Efficiency ratios'. Effectiveness ratios can be determined by looking at success, error, abandonment, and timeout rates. And Efficiency ratios can be determined by looking at average clicks per task, average time taken per task, and unique page views per task.

‍

You do need to be careful not to make assumptions based on this kind of data— it can't tell you what people were intending to do. If a task is taking people too long, it may be because it’s too complicated ... or because a few people made themselves coffee in between clicks. So supplement these metrics with other research that does tell you about intentions.

‍

4. Where are they clicking first?

‍

A good user experience is one that gets out of bed on the right side. First clicks matter for a good user experience.

‍

A 2009 study showed that in task-based user tests, people who got their first click right were around twice as likely to complete the task successfully than if they got their first click wrong. This year, researchers at Optimal Workshop followed this up by analyzing data from millions of completed Treejack tasks, and found that people who got their first click right were around three times as likely to get the task right.

‍

That's data worth paying attention to.

‍

So, how to measure? You can use software that records mouse clicks first clicks from analytics on your page, but it difficult to measure a visitor's intention without asking them outright, so I'd say task-based user tests are your best bet.

‍

For in-person research sessions, make gathering first-click data a priority, and come up with a consistent way to measure it (a column on a spreadsheet, for example). For remote research, check out Chalkmark (a tool devoted exclusively to gathering quantitative, first-click data on screenshots and wireframes of your designs) and UserTesting.com (for videos of people completing tasks on your live website).

‍

5. Resources to help you with the number crunching

‍

Here's a great piece on uxmastery.com about calculating the ROI of UX.

‍

Here's Jakob Nielsen in 1999 with a simple 'Assumptions for Productivity Calculation', and here's an overview of what's in the 4th edition of NN/G's Return on Investment for Usability report (worth the money for sure).

‍

Here's a calculator from Write Limited on measuring the cost of unclear communication within organizations (which could quite easily be applied to UX).

‍

And here's a unique take on what numbers to crunch from Harvard Business Review.

‍

I hope you find this as a helpful starting point Brian, and please do have a think about what I said about defining ROI. I’m curious to know how everyone else defines and measures ROI — let me know!

In the fast-paced world of product development, risk is an ever-present factor. From potential user dissatisfaction to costly redesigns, the stakes are high. User Experience Research (UXR) plays a crucial role in identifying and mitigating these risks, but quantifying its preventive value can be challenging. Let's explore how UXR contributes to risk mitigation and how we can measure its impact.

‍

Understanding risk in product development 😬

‍

Product development is an exciting yet challenging journey that requires careful navigation of inherent risks. Teams invest significant time and resources into creating solutions they hope will resonate with users, but this process is far from a guaranteed success. When embarking on a new product venture, teams are essentially making an educated guess about what users want and need. This inherent uncertainty brings several considerations, including substantial time investments, allocation of financial and human resources, and the need to adapt to constantly evolving user preferences and competitive landscapes.

‍

The challenge lies in aligning all these elements to create a successful product. Getting it wrong can have significant consequences that extend beyond mere disappointment. Wasted development efforts can result in resources being spent on features or products that don't meet market needs. There's also the potential for negative impact on brand perception if a product misses the mark, potentially affecting how customers view the company as a whole. Furthermore, missed opportunities in the fast-paced world of product development can allow competitors to gain an advantage, affecting a company's market position.

‍

However, there's a powerful tool that can help mitigate these risks: user research. As one industry leader noted in our research, "In periods of change, those who maintain a deep connection with their customers' evolving needs are best positioned to adapt and thrive." This insight highlights a crucial strategy for navigating the uncertain waters of product development.

‍

By prioritizing user research, teams can gain valuable insights that guide their decision-making process. This approach allows them to identify genuine user needs and pain points, potentially uncovering issues that might have been overlooked. It also provides an opportunity to spot potential problems early in the development process, when changes are less costly and easier to implement. Moreover, deep user understanding can uncover opportunities for innovation and differentiation that might not be apparent without this research.

‍

While user research doesn't eliminate all risks associated with product development, it provides a compass that can guide teams through the process with greater confidence. In the dynamic world of product creation, the biggest risk often comes from operating without these user insights. By integrating user research into the development process, teams can navigate uncertainties more effectively and increase their odds of creating products that truly resonate with their target audience.

‍

Successful product development is ultimately about finding the right balance between innovation, user needs, and calculated risk-taking. It's a complex dance of creativity, market understanding, and strategic decision-making. By maintaining a strong connection to user needs and preferences throughout the development process, teams can mitigate risks and increase their chances of success. This user-centric approach not only helps in creating products that meet market demands but also positions companies to adapt and thrive in periods of change and uncertainty.

‍

UXR's role in identifying and mitigating risks 🔎

‍

User experience research plays a crucial role in identifying and mitigating risks throughout the product development process. Acting as an early warning system, UX research helps teams pinpoint potential issues before they evolve into costly problems. This proactive approach allows organizations to make informed decisions and adjustments early in the development cycle, potentially saving significant time and resources.

‍

By engaging with users throughout the development process, researchers gain invaluable insights that can shape the direction of a product. These interactions enable teams to validate product concepts and designs, ensuring that the final output aligns with user expectations and needs. Through various research methodologies, UX researchers can identify usability issues and pain points that might otherwise go unnoticed until after launch. This early detection allows for timely refinements, resulting in a more polished and user-friendly final product.

‍

Our survey findings underscore the value of integrating UX research into the product development process. Organizations that have fully embedded UXR into their workflows demonstrate a superior ability to navigate uncertainties and make user-centered decisions. This integration allows for a more agile and responsive approach to product development, where user feedback and insights directly inform strategic choices.

‍

Download the report

‍

Methodologies for quantifying prevented issues 🧮

‍

In the space of user experience research, one of the most significant yet often overlooked benefits is its ability to prevent issues before they arise. This preemptive approach can save organizations substantial time, resources, and potential reputational damage. However, quantifying the value of something that didn't happen presents a unique challenge. How do you measure the impact of problems avoided? This question has led to the development of various methodologies aimed at quantifying the value of UX research in issue prevention.

‍

1. Issue tracking: Keep a detailed log of potential issues identified through research. Categorize them by severity and potential impact.

‍

2. Cost estimation: Work with product and engineering teams to estimate the cost of addressing issues at different stages of development. Compare this to the cost of conducting research.

‍

3. A/B Testing: Use controlled experiments to compare the performance of research-informed designs against alternatives.

‍

4. Predictive modeling: Develop models that estimate the potential impact of issues on key metrics like user retention or conversion rates.

‍

5. Historical comparison: Analyze past projects where research was not conducted and compare their outcomes to research-informed projects.

‍

One effective approach is to use a research ROI calculator that estimates potential cost savings and revenue increases associated with research-driven improvements. This provides a clear financial justification for UXR investments.

‍

Communicating preventive value to stakeholders 📢

‍

To effectively communicate the value of risk mitigation through UXR, consider these strategies:

‍

1. Speak the language of business: Frame research findings in terms of business outcomes, such as potential cost savings, revenue impact, or risk reduction.

‍

2. Use visualizations: Create compelling visual representations of prevented issues and their potential impact.

‍

3. Share success stories: Highlight case studies where research prevented significant issues or led to successful outcomes.

‍

4. Involve stakeholders: Engage key decision-makers in the research process to build understanding and buy-in.

‍

5. Provide ongoing updates: Regularly communicate how research insights are influencing decisions and mitigating risks throughout the development process.

‍

Remember, as one research manager in our study observed, "When I hear that a company is downsizing, I immediately wonder how it will affect their research capabilities."

‍

This highlights the importance of consistently demonstrating the value of UXR in risk mitigation.

‍

By quantifying and communicating the preventive value of UX research, we can shift the perception of UXR from a cost center to a critical investment in risk mitigation and product success. As the field continues to evolve, developing robust methodologies for measuring this preventive value will be key to securing resources and support for UXR initiatives.

‍

Ultimately, the goal is to create a culture where user research is seen as an essential safeguard against costly mistakes and a driver of informed, user-centered decision-making. By doing so, organizations can navigate the uncertainties of product development with greater confidence and success.

‍

DOWNLOAD THE FULL REPORT

‍

Maximize your risk mitigation efforts with Optimal 👩🏻🚒

‍

Ready to elevate your UX research and risk mitigation strategies? Optimal Workshop's comprehensive platform offers powerful tools to streamline your research process, from participant recruitment to data analysis. Our suite of user-friendly solutions enables you to conduct more efficient studies, uncover deeper insights, and effectively communicate the preventive value of your research to stakeholders. 

‍

With Optimal, you can quantify your risk mitigation efforts more accurately and demonstrate the ROI of UXR with greater clarity. Don't let potential risks threaten your product's success. 

‍

Try Optimal Workshop today and transform your approach to UX research and risk prevention. 

‍

Qualitative user research is about exploration. Exploration is about the journey, not only the destination (or outcome). Gaining information and insights about your users through interviews, usability testing, contextual, observations and diary entries. Using these qualitative research methods to not only answer your direct queries, but to uncover and unravel your users ‘why’.

‍

It can be important to use qualitative research to really dig deep, get to know your users and get inside their heads, and their reasons. Creating intuitive and engaging products that deliver the best user experience. 

‍

What is qualitative research? 🔎

‍

The term ‘qualitative’ refers to things that cannot be measured numerically and qualitative user research is no exception. Qualitative research is primarily an exploratory research method that is typically done early in the design process and is useful for uncovering insights into people’s thoughts, opinions, and motivations. It allows us to gain a deeper understanding of problems and provides answers to questions we didn’t know we needed to ask. 

‍

Qualitative research could be considered the ‘why’. Where quantitative user research uncovers the how or the what users want. Qualitative user research will uncover why they make decisions (and possibly much more).

‍

Priorities ⚡⚡⚡⚡

‍

When undertaking user research it is great to do a mix of quantitative and qualitative research. Which will round out the numbers with human driven insights.

‍

Quantitative user research methods, such as card sorting or tree testing, will answer the ‘what’ your users want, and provide data to support this. These insights are number driven and are based on testing direct interaction with your product. This is super valuable to report to stakeholders. Hard data is difficult to argue what changes need to be made to how your information architecture (IA) is ordered, sorted or designed. To find out more about the quantitative research options, take a read.

‍

Qualitative user research, on the other hand, may uncover a deeper understanding of ‘why’ your users want the IA ordered, sorted or designed a certain way.  The devil is in the detail afterall and great user insights are discoverable. 

‍

Priorities for your qualitative research needs to be less about the numbers, and more on discovering your users ‘why’. Observing, listening, questioning and looking at reasons for users decisions will provide valuable insights for product design and ultimately improve user experience.

‍

Usability Testing - this research method is used to evaluate how easy and intuitive a product is to use.  Observing, noting and watching the participant complete tasks without interference or questions can uncover a lot of insights that data alone can’t give. This method can be done in a couple of ways, moderated or unmoderated. While it can be quicker to do unmoderated and easier to arrange, the deep insights will come out of moderated testing. 

‍

Observational - with this qualitative research method your insights will be uncovered from observing and noting what the participant is doing, paying particular attention to their non-verbal communication. Where do they demonstrate frustration, or turn away from the task, or change their approach? Factual note taking, meaning there shouldn’t be any opinions attached to what is being observed, is important to keep the insights unbiased.

‍

Contextual - paying attention to the context in which the interview or testing is done is important. Is it hot, loud, cold or is the screen of their laptop covered in post-its that make it difficult to see? Or do they struggle with navigating using the laptop tracker? All of this noted, in a factual manner, without personal inferring or added opinion based observations can give a window into why the participant struggled or was frustrated at any point.

‍

These research methods can be done as purely observational research (you don’t interview or converse with your participant) and noting how they interact (more interested in the process than the outcome of their product interaction). Or, these qualitative research methods can be coupled with an

‍

Interview - a series of questions asked around a particular task or product. Careful note taking around what the participant says as well as noting any observations. This method should allow a conversation to flow. Whilst the interviewer should be prepared with a list of questions around their topic, remain flexible enough to dig deeper where there might be details or insights of interest. An interviewer that is comfortable in getting to know their participants unpicks reservations and allows a flow of conversation, and generates amazing insights.

‍

With an interview it can be of use to have a second person in the room to act as the note taker. This can free up the interviewer to engage with the participant and unpick the insights.

‍

Using a great note taking side kick, like our Reframer, can take the pain out of recording all these juicy and deep insights. Time-stamping, audio or video recordings and notes all stored in one place. Easily accessed by the team, reviewed, reports generated and stored for later.

‍

Let’s consider 🤔

‍

You’re creating a new app to support your gym and it’s website. You’re looking to generate personal training bookings, allow members to book classes or have updates and personalise communication for your members. But before investing in final development it needs to be tested. How do your users interact with it? Why would they want to? Does it behave in a way that improves the user experience? Or does it simply not deliver? But why?

‍

First off, using quantitative research like Chalkmark would show how the interface is working. Where are users clicking, where do they go after that. Is it simple to use? You now have direct data that supports your questions, or possibly suggests a change of design to support quicker task completion, or further engagement.

‍

While all of this is great data for the design, does it dig deep enough to really get an understanding of why your users are frustrated? Do they find what they need quickly? Or get completely lost? Finding out these insights and improving on them can make the most of your users’ experience.

‍

When quantitative research is coupled with robust qualitative research that prioritizes an in-depth understanding of what your users need, ultimately the app can make the most of your users’ experience.

‍

Using moderated usability testing for your gym app, observations can be made about how the participant interacts with the interface. Where do they struggle, get lost, or where do they complete a task quickly and simply. This type of research enhances the quantitative data and gives insight into where and why the app is or isn't performing.

‍

Then interviewing participants about why they make decisions on the app, how they use it and why they would use it. These focussed questions, with some free flow conversation will round out your research. Giving valuable insights that can be reviewed, analyzed and reported to the product team and key stakeholders. Focussing the outcome, and designing a product that delivers on not just what users need, but in-depth understand of why. 

‍

Wrap Up 🥙

‍

Quantitative and qualitative user research do work hand in hand, each offering a side to the same coin. Hard number driven data with quantitative user research will deliver the what needs to be addressed. With focussed quantitative research it is possible to really get a handle on why your users interact with your product in a certain way, and how. 

‍

The Optimal Workshop platform has all the tools, research methods and even the note taking tools you need to get started with your user research, now, not next week! See you soon.