Event Recap: Measuring the Value of UX Research at UXDX
Last week Optimal Workshop was delighted to sponsor UXDX USA 2024 in New York. The User Experience event brings together Product, Design, UX, CX, and Engineering professionals and our team had an amazing time meeting with customers, industry experts, and colleagues throughout the conference. This year, we also had the privilege of sharing some of our industry expertise by running an interactive forum on “Measuring the Value of UX Research” - a topic very close to our hearts.
Our forum, hosted by Optimal Workshop CEO Alex Burke and Product Lead Ella Fielding, was focused on exploring the value of User Experience Research (UXR) from both an industry-wide perspective and within the diverse ecosystem of individual companies and teams conducting this type of research today.
The session brought together a global mix of UX professionals for a rich discussion on measuring and demonstrating the effectiveness of and the challenges facing organizations who are trying to tie UXR to tangible business value today.
The main topics for the discuss were:
Metrics that Matter: How do you measure UXR's impact on sales, customer satisfaction, and design influence?
Challenges & Strategies: What are the roadblocks to measuring UXR impact, and how can we overcome them?
Beyond ROI: UXR's value beyond just financial metrics
Some of the key takeaways from our discussions during the session were:
The current state of UX maturity and value
Many UX teams don’t measure the impact of UXR on core business metrics and there were more attendees who are not measuring the impact of their work than those that are measuring it.
Alex & Ella discussed with the attendees the current state of UX research maturity and the ability to prove value across different organizations represented in the room. Most organizations were still early in their UX research maturity with only 5% considering themselves advanced in having research culturally embedded.
Defining and proving the value of UX research
The industry doesn’t have clear alignment or understanding of what good measurement looks like. Many teams don’t know how to accurately measure UXR impact or don’t have the tools or platforms to measure it, which serve as core roadblocks for measuring UXRs’ impact.
Alex and Ella discussed challenges in defining and proving the value of UX research, with common values being getting closer to customers, innovating faster, de-risking product decisions, and saving time and money. However, the value of research is hard to quantify compared to other product metrics like lines of code or features shipped.
Measuring and advocating for UX research
When teams are measuring UXR today there is a strong bias for customer feedback, but little ability or understanding about how to measure impact on business metrics like revenue.
The most commonly used metrics for measuring UXR are quantitative and qualitative feedback from customers as opposed to internal metrics like stakeholder involvement or tieing UXR to business performance metrics (including financial performance).
Attendees felt that in organizations where research is more embedded, researchers spend significant time advocating for research and proving its value to stakeholders rather than just conducting studies. This included tactics like research repositories and pointing to past study impacts as well as ongoing battles to shape decision making processes.
One of our attendees highlighted that engaging stakeholders in the process of defining key research metrics prior to running research was a key for them in proving value internally.
Relating user research to financial impact
Alex and Ella asked the audience if anyone had examples of demonstrating financial impact of research to justify investment in the team and we got some excellent examples from the audience proving that there are tangible ways to tie research outcomes to core business metrics including:
Calculating time savings for employees from internal tools as a financial impact metric.
Measuring a reduction in calls to service desks as a way to quantify financial savings from research.
Most attendees recognise the value in embedding UXR more deeply in all levels of their organization - but feel like they’re not succeeding at this today.
Most attendees feel that UXR is not fully embedded in their orgnaization or culture, but that if it was - they would be more successful in proving its overall value.
Stakeholder buy-in and engagement with UXR, particularly from senior leadership varied enormously across organizations, and wasn’t regularly measured as an indicator of UXR value
In organizations where research was more successfully embedded, researchers had to spend significant time and effort building relationships with internal stakeholders before and after running studies. This took time and effort away from actual research, but ended up making the research more valuable to the business in the long run.
With the large range of UX maturity and the democratization of research across teams, we know there’s a lot of opportunity for our customers to improve their ability to tie their user research to tangible business outcomes and embed UX more deeply in all levels of their organizations. To help fill this gap, Optimal Workshop is currently running a large research project on Measuring the Value of UX which will be released in a few weeks.
At Optimal Workshop, we're dedicated to building the best user research platform to empower you with the tools to better understand your customers and create intuitive digital experiences. We're thrilled to announce some game-changing updates and new products that are on the horizon to help elevate the way you gather insights and keep customers at the heart of everything you do.
What’s new…
Integration with Figma 🚀
Last month, we joined forces with design powerhouse Figma to launch our integration. You can import images from Figma into Chalkmark (our click-testing tool) in just a few clicks, streamlining your workflows and getting insights to make decisions based on data not hunches and opinions.
What’s coming next…
Session Replays 🧑💻
With session replay you can focus on other tasks while Optimal Workshop automatically captures card sort sessions for you to watch in your own time. Gain valuable insights into how participants engage and interpret a card sort without the hassle of running moderated sessions. The first iteration of session replays captures the study interactions, and will not include audio or face recording, but this is something we are exploring for future iterations. Session replays will be available in tree testing and click-testing later in 2024.
Reframer Transcripts 🔍
Say goodbye to juggling note-taking and hello to more efficient ways of working with Transcripts! We're continuing to add more capability to Reframer, our qualitative research tool, to now include the importing of interview transcripts. Save time, reduce human errors and oversights by importing transcripts, tagging and analyzing observations all within Reframer. We’re committed to build on transcripts with video and audio transcription capability in the future, we’ll keep you in the loop and when to expect those releases.
Prototype testing 🧪
The team is fizzing to be working on a new Prototype testing product designed to expand your research methods and help test prototypes easily from the Optimal Workshop platform. Testing prototypes early and often is an important step in the design process, saving you time and money before you invest too heavily in the build. We are working with customers and on delivering the first iteration of this exciting new product. Stay tuned for Prototypes coming in the second quarter of 2024.
Workspaces 🎉
Making Optimal Workshop easier for large organizations to manage teams and collaborate more effectively on projects is a big focus for 2024. Workspaces are the first step towards empowering organizations to better manage multiple teams with projects. Projects will allow greater flexibility on who can see what, encouraging working in the open and collaboration alongside the ability to make projects private. The privacy feature is available on Enterprise plans.
Questions upgrade❓
Our survey product Questions is in for a glow up in 2024 💅. The team are enjoying working with customers, collecting and reviewing feedback on how to improve Questions and will be sharing more on this in the coming months.
Help us build a better Optimal Workshop
We are looking for new customers to join our research panel to help influence product development. From time to time, you’ll be invited to join us for interviews or surveys, and you’ll be rewarded for your time with a thank-you gift. If you’d like to join the team, email product@optimalworkshop.com
It’s no secret - New Zealand has a diversity problem in design and technology.
Throughout her career, Sachi often felt like the odd one out - the only woman, the only Pasifika person, the one who laughed too loud, the one who looked different and sounded different. But as a leader, Sachi has been able to create change.
Sachi Taulelei, Head of Design, ANZ, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, on how she is building a diverse team of designers at New Zealand’s largest bank.
In her talk, Sachi shares the challenges she’s faced as a Pasifika woman in design and technology; and how this has shaped her approach to leadership and her drive to create inclusive environments where individuals and teams thrive.
Background on Sachi Taulelei
Sachi is a creative strategist, a design leader, and a recovering people pleaser. She has worked in digital and design for over 25 years, spending most of her career creating and designing digital experiences centered on people.
As a proud Pasifika woman, she has a particular interest in diversity, equity, and inclusion. She has spoken out about the need for more diversity within design and technology and the impact it can have on the technology we create.
Sachi is passionate about giving back - when she's not running after her two kids, you'll find her mentoring Pasifika youth, cheering on young leaders through the Young Enterprise Scheme, judging awards for Women in AI, or volunteering at the local hospice.
Odd one out: embracing diversity in design and technology ✨
Looking and sounding different from her peers, Sachi always felt like she was trying to find her place in the office. She always felt like she didn’t belong.
Sachi has experienced all forms of racism and discrimination as a result of her heritage. These experiences aren’t spoken about and often go unnoticed by the majority. She has held equivalent jobs to male counterparts but received lower pay, and was advised to change her name from Sachi to Sacha on her job applications to improve her chances.
Sachi’s response was to work hard and become great at what she does, which was recognized over time. Slowly, she began to rise through the ranks. However, having reached leadership roles, she struggled to be heard and participate, without knowing why. The advice was given freely by managers to “stick at it”, to “grow thicker skin”, and to grow through the “school of hard knocks”. Although this advice worked at face value and she flourished, Sachi began to feel like a fraud and constantly second-guessing herself. She began to “edit” herself to fit into an acceptable mold and, in doing so, felt like she lost part of who she was.
What is success? 🏆🎯💎
Success often comes in the form of our leaders who have already climbed the mountains of achievement. When you see success in this way, as someone who doesn’t fit the mold, there is pressure to conform to get ahead. Using the same tools and advice given to these leaders, she realized, would actually hold her back.
Realizing true value through our uniqueness 🪐🦋
Sachi recounts the treatment of Japanese-American citizens in the U.S. in the years following Pearl Harbour, where Japanese-American citizens were moved to concentration camps. This happened despite an official report finding conclusively that there was no threat from this population. Even though Germany and Italy were also at war with the U.S., for example, citizens with Italian and German heritage were not treated this way. This caused immeasurable pain, shame, and fear for the victims, and fostered a head-down, work-hard mentality in order to try and forget the treatment they received. This attitude, Sachi believes, was passed down to her from her ancestors who experienced that reality. Sachi explains that while there are many things that can hold someone back in life, creating meaningful change starts with introspection. Often, that requires us to work through fear and shame.
Reflecting on her heritage, which is part Samoan and part Japanese, Sachi started to embrace her unique traits. In her case, she embraced the deep empathy and human compassion from her Japanese side and the deep sense of community and connection from her Samoan side. Her uniqueness is something to celebrate, not to hide behind.
Becoming a leader and realizing this, Sachi wanted to create a team culture based on equity, openness, and a sense of belonging – all things that Sachi wished for herself on her journey.
Why it matters 💫
Once she understood herself and what she wanted for her team, Sachi set to work on building a new team culture. Sachi breaks down key learnings from how she turned this vision into reality.
Define
Define what diversity means for your team. You need to clearly understand what it is you want to achieve before you can achieve it. For Sachi’s team, they knew that they wanted to create a team that was representative of New Zealand. Sachi knew, for example, that she had a lack of Māori and Pacific representation within the team. Māori and Pasifika represent 25% of the population. So, an effort was made to increase ranks by hiring talent from these cultures.
Additionally, Sachi focused on creating new role levels - from intern right through to graduates, juniors, and intermediate-level positions. This helped to acknowledge age differences within her team and also helped to manage career progression opportunities.
Effort
It can be difficult to achieve diversity and inclusion and it requires a lot of work. For example, Sachi learned that posting an ad on job boards and expecting to receive hundreds of Māori and Pasifika applicants wasn’t realistic. Instead, partnerships were built with local design schools, and networking events were consistently attended. Job referrals from within the team were also leveraged, as well as establishing a strong direction for recruitment specialists within the organization.
Sachi also recognized that, as a leader, she needed to be more visible and more vocal about sharing her views of the world and what she was trying to achieve. It was important to be clear about the type of culture she was building within her team so that she could promote it.
In less than a year her team grew (from 11 to 40!) which meant a focus on building an inclusive team culture was required. The central theme throughout this time was, “You have to connect to yourself and your strengths first and foremost, before you can connect with others and as a team”. This meant that the team used tools like the Clifton Strength Finder, in order to learn about themselves and each other. Each designer was then encouraged to delve into their own natural working styles and were taught how to amplify their own strengths through various workshops. This approach also becomes handy when recruiting and strengthening potential weak spots.
Integrity
It’s important to have leaders who care - you can’t do it on your own. There can be pain points on the journey to creating diversity and inclusion, so it’s necessary to have leaders who listen, support, and work through some of the challenges that can arise.
Benefits of diversity and inclusion in design teams 👩🏼🤝👨🏿
Why push for diversity and inclusion? Sachi argues that the benefits are evident in the way that her team designs.
For example, her team:
Insist that research is done with diverse customer groups
Advocates for accessibility when no one else will
Understand problems from different perspectives before diving into a project
Most importantly, the benefits show up in the way that each other is treated, and the relationships that are built with key stakeholders. Diversity and inclusion are wins for everyone - the team, the organization, and the customer.
Natalie and Lulu have forged a unique team culture that focuses on positive outputs (and outcomes) for their app’s growing user base. In doing so, they turned the traditional design approach on its head and created a dynamic and supportive team.
Natalie, Director of Design at Hatch, and Lulu, UX Design Specialist, recently spoke at UX New Zealand, the leading UX and IA conference in New Zealand hosted by Optimal Workshop, on their concept of “radical collaboration”.
In their talk, Nat and Lulu share their experience of growing a small app into a big player in the finance sector, and their unique approach to teamwork and culture which helped achieve it.
Background on Natalie Ferguson and Lulu Pachuau
Over the last two decades, Lulu and Nat have delivered exceptional customer experiences for too many organizations to count. After Nat co-founded Hatch, she begged Lulu to join her on their audacious mission: To supercharge wealth building in NZ. Together, they created a design and product culture that inspired 180,000 Kiwi investors to join in just 4 years.
Radical Collaboration - How teamwork makes the dream work 💪💪💪
Nat and Lulu discuss how they nurtured a team culture of “radical collaboration” when growing the hugely popular app Hatch, based in New Zealand. Hatch allows everyday New Zealanders to quickly and easily trade in the U.S. share market.
The beginning of the COVID pandemic spelled huge growth for Hatch and caused significant design challenges for the product. This growth meant that the app had to grow from a baby startup to one that could operate at scale - virtually overnight.
In navigating this challenge, Nat and Lulu coined the term radical collaboration, which aims to “dismantle organizational walls and supercharge what teams achieve”. Radical collaboration has six key pillars, which they discuss alongside their experience at Hatch.
Pillar #1: When you live and breathe your North star
Listening to hundreds of their customers’ stories, combined with their own personal experiences with money, compelled Lulu and Nat to change how their users view money. And so, “Grow the wealth of New Zealanders” became a powerful mission statement, or North Star, for Hatch. The mission was to give people the confidence and the ability to live their own lives with financial freedom and control. Nat and Lulu express the importance of truly believing in the mission of your product, and how this can become a guiding light for any team.
Pillar #2: When you trust each other so much, you’re happy to give up control
As Hatch grew rapidly, trusting each other became more and more important. Nat and Lulu state that sometimes you need to take a step back and stop fueling growth for growth’s sake. It was at this point that Nat asked Lulu to join the team, and Nat’s first request was for Lulu to be super critical about the product design to date - no feedback was out of bounds. Letting go, feeling uncomfortable, and trusting your team can be difficult, but sometimes it’s what you need in order to drag yourself out of status quo design. This resulted in a brief hiatus from frantic delivery to take stock and reprioritize what was important - something that can be difficult without heavy doses of trust!
Pillar #3: When everyone wears all the hats
During their journey, the team at Hatch heard lots of stories from their users. Many of these stories were heard during “Hatcheversery Calls”, where team members would call users on their sign-up anniversary to chat about their experience with the app. Some of these calls were inspiring, insightful, and heartwarming.
Everyone at Hatch made these calls – designers, writers, customer support, engineers, and even the CEO. Speaking to strangers in this way was a challenge for some, especially since it was common to field technical questions about the business. Nevertheless, asking staff to wear many hats like this turned the entire team into researchers and analysts. By forcing ourselves and our team outside of our comfort zone, we forced each other to see the whole picture of the business, not just our own little piece.
Pillar #4: When you do what’s right, not what’s glam
In an increasingly competitive industry, designers and developers are often tempted to consistently deliver new and exciting features. In response to rapid growth, rather than adding more features to the app, Lulu and Nat made a conscious effort to really listen to their customers to understand what problems they needed solving.
As it turned out, filing overseas tax returns was a significant and common problem for their customers - it was difficult and expensive. So, the team at Hatch devised a tax solution. This solution was developed by the entire team, with almost no tax specialists involved until the very end! This process was far from glamorous and it often fell outside of standard job descriptions. However, the team eventually succeeded in simplifying a notoriously difficult process and saved their customers a massive headache.
Pillar #5: When you own the outcome, not your output.
Over time Hatch’s user base changed from being primarily confident, seasoned investors, to being first-time investors. This new user group was typically scared of investing and often felt that it was only a thing wealthy people did.
At this point, Hatch felt it was necessary to take a step back from delivering updates to take stock of their new position. This meant deeply understanding their customers’ journey from signing up, to making their first trade. Once this was intimately understood, the team delivered a comprehensive onboarding process which increased the sign-up conversion rate by 10%!
Pillar #6: When you’re relentlessly committed to making it work
Nat and Lulu describe a moment when Allbirds wanted to work with Hatch to allow ordinary New Zealanders to be involved in their IPO launch on the New York stock exchange. Again, this task faced numerous tax and trade law challenges, and offering the service seemed like yet another insurmountable task. The team at Hatch nearly gave up several times during this project, but everyone was determined to get this feature across the line – and they did. As a result, New Zealanders were some of the few regular investors from outside the U.S that were able to take part in Albirds IPO.
Why it matters 💥
Over four years, Hatch grew to 180,000 users who collectively invested over $1bn. Nat and Lulu’s success underscores the critical role of teamwork and collaboration in achieving exceptional user experiences. Product teams should remember that in the rapidly evolving tech industry, it's not just about delivering the latest features; it's about fostering a positive and supportive team culture that buys into the bigger picture.
The Hatch team grew to be more than team members and technical experts. They grew in confidence and appreciated every moving part of the business. Product teams can draw inspiration from Hatch's journey, where designers, writers, engineers, and even the CEO actively engaged with users, challenged traditional design decisions, and prioritized solving actual user problems. This approach led to better, more user-centric outcomes and a deep understanding of the end-to-end user experience.
Most importantly, through the good times and tough, the team grew to trust each other. The mission weaved its way through each member of the team, which ultimately manifested in positive outcomes for the user and the business.
Nat and Lulu’s concept of radical collaboration led to several positive outcomes for Hatch:
It changed the way they did business. Information was no longer held in the minds of a few individuals – instead, it was shared. People were able to step into other people's roles seamlessly.
Hatch achieved better results faster by focusing on the end-to-end experience of the app, rather than by adding successive features.
The team became more nimble – potential design/development issues were anticipated earlier because everyone knew what the downstream impacts of a decision would be.
Over the next week, Lulu and Nat encourage designers and researchers to get outside of their comfort zone and:
Visit customer support team
Pick up the phone and call a customer
Challenge status quo design decisions. Ask, does this thing solve an end-user problem?
