Which comes first: card sorting or tree testing?

Recently, Optimal Workshop partnered with Eniola Abioye, Lead UX Researcher at Meta and UXR Career Coach at UX Outloud to host a career masterclass featuring practical advice and guidance on how to: revamp and build a portfolio, emphasize the impact of your projects and showcase valuable collaborations. It also included panel discussions with experts from a variety of roles (UX, product management, engineering, career coaching and content design) talking about their journeys to becoming UX leaders. 

‍

Keep reading to get key takeaways from the discussion on:

‍

• How to show the impact of your UX work
• Common blockers in UX work
• How to collaborate with cross-functional UX stakeholders 
• How to build a resume and portfolio that uses industry language to present your experience

‍

How to show the impact of your UX 💥

‍

At a time when businesses are reducing costs to focus on profitability - proving the value of your work is more important than ever. Unfortunately, measuring the impact of UX isn’t as straightforward as tracking sales or marketing metrics. With this in mind, Eniola asked the panelists - how do you show the impact of UX in your work?Providing insights is simply not enough. “As a product manager, what I really care about is insights plus recommendations, because recommendations make my life easier,” said Kwame Odame. 

‍

Auset Parris added her perspective on this topic as a Growth Content Designer, “the biggest thing for me to be impactful in my space [Content Design] is to consistently document the changes that I’ve made and share them with the team along with recommendations.” Auset also offered her perspective regarding recommendations, “recommendations are not always going to lead to the actual product executions, but recommendations are meant to guide us.” When it comes to deciding which recommendation to proceed with (if any) it's important to consider whether or not they are aligned with the overarching goal. 

‍

Blockers in UX work 🚧

‍

As UXR becomes more democratized in many businesses and the number of stakeholders increases, the ability to gain cross-functional buy-in for the role and outcomes of UXR is a key way to help keep research a priority. 

‍

In his past experience, Kwame has realized that the role of a user experience researcher is just as important as that of a product manager, data scientist, engineer, or designer. However, one of the biggest blockers for him as a product manager is how the role of a UX researcher is often overlooked. “Just because I’m the product manager doesn’t mean that I’m owning every aspect of the product. I don’t have a magic wand right? We all work as a team.” Furthermore, Kwame notes that a user researcher is an incredibly hard role and a very important one, and I think we need to invest more in the UX space.

‍

Auset also shared her perspective on the topic, “I wouldn’t say this is a blocker, but I do think this is a challenging piece of working in a team - there are so many stakeholders.” Although it would be ideal for each of the different departments to work seamlessly together at all times, that’s not always the case. Auset spoke about a time where the data scientists and user researchers were in disagreement. Her role as a Growth Content Designer is to create content that enhances the user experience. “But if I’m seeing two different experiences, how do I move forward? That’s when I have to ask everyone - come on let’s dig deeper. Are we looking at the right things?” If team members are seeing different results, or having different opinions, then maybe they are not asking the right questions and it's time to dig deeper. 

‍

How to collaborate with cross-functional UX stakeholders 🫱🏽🫲🏻

‍

The number and type of roles that now engage with research are increasing. As they do, the ability to collaborate and manage stakeholders in research projects has become essential. 

‍

Kwame discussed how he sets up a meeting for the team to discuss their goals for the next 6 months. Then, he meets with the team on a weekly basis to ensure alignment. The main point of the meeting is to ensure everyone is leaving with their questions answered and blockers addressed. It's important to ensure everyone is moving in the right direction. 

‍

Auset added that she thinks documentation is key to ensuring alignment. “One thing that has been helpful for me is having the documentation in the form of a product brief or content brief.” The brief can include the overarching goal, strategy, and indicate what each member of the team is working on. Team members can always look back at this document to ensure they are on the right track. 

‍

Career advice: documenting the value you bring 💼

‍

One of the participants asked the panel, “how do you secure the stability of your UX career?” 

‍

Eniola took this opportunity to share some invaluable advice as a career coach, “I think the biggest thing that comes to mind is value proposition. It's important to be very clear about the value and impact you bring to the team. It used to be enough to just be really, really good at research and just do research and provide recommendations. Now that’s not enough. Now you have to take your teams through the process, integrate your recommendations into the product, and focus on driving impact.” 

‍

Companies aren’t looking to hire someone who can perform a laundry list of tasks, they’re looking for UX professionals who can drive results. Think about the metrics you can track, to help showcase the impact of your work. For example, if you’re a UX designer - how much less time did the user spend on the task with your new design? Did the abandonment or error rate decrease significantly as a result of your work? How much did the overall customer satisfaction score rise, after you implemented your project? Before starting your project, decide on several metrics to track (make sure they align with your organization’s goals), and reflect on these after each project. 

‍

Fatimah Richmond offered another piece of golden career advice. She encourages UX researchers to create an ongoing impact tracker. She’ll create a document where she lists the company's objectives, the projects she worked on, and the specific impact she made on the companies objectives. It's much easier to keep track of the wins as they happen, and jot a few notes about the impact you’ve made with each project, then scrambling to think of all the impact you’ve made when writing your resume. It's also important to note the impact your work has made on the different departments - product, marketing, sales, etc.

‍

She also advises UX researchers to frequently share their science insights with their colleagues as the project progresses. Instead of waiting until the very end of the project and providing a “perfectly polished” deck, be transparent with the team about what you are working on and the impact it's having throughout the duration of the project.

‍

Another participant asked - what if you need help determining the value you bring? Auset recommends asking for actionable feedback from coworkers. These people work with you every single day, so they know your contributions you are making to the team. 

‍

Documenting the tangible impact you make as a UX professional is crucial - not only will it help create greater stability for your career, but it will also help organizations recognize the importance of a UX research. As Kwame discussed in the “blockers” section, one of the biggest challenges he faces as a product manager is the perception of the UX role as less important than the more traditional product manager, Engineer, and Designer roles. 

‍

About Eniola Abioye

‍

Eniola helps UX researchers improve their research practice. Whether you’re seasoned and looking to level up or a new researcher looking to get your bearings in UX, Eniola can help you focus and apply your skillset. She is a UX Researcher and Founder of UX Outloud. As a career coach, she guides her clients through short and long term SMART goals and then works with them to build a strategic plan of attack. She is innately curious, a self-starter, adaptable, and communicative with a knack for storytelling.

‍

Learn more about UX Outloud.

‍

Connect with Eniola on Linkedin.

‍

About the panelists 🧑🏽🤝🧑🏽

‍

The panel was comprised of talented professionals from a variety of fields including UX research, content strategy, product management & engineering, and career coaching. Their diverse perspectives led to an insightful and informative panel session. Keep reading to get to know each of the amazing panelists: 

‍

Growth Content Designer: Auset Parris is a growth content designer at Meta. She has spent 7 years navigating the ever-evolving landscape of content strategy. She is passionate about the role of user research in shaping content strategies. Furthermore, Auset believes that understanding user behavior and preferences is fundamental to creating content that not only meets but exceeds user expectations. 

‍

Senior UX Researcher: Jasmine Williams, Ph.D. is a senior researcher with over a decade of experience conducting youth-focused research. She has deep expertise in qualitative methods, child and adolescent development, and social and emotional well-being. Jasmine is currently a user experience researcher at Meta and her work focuses on teen safety and wellbeing. 

‍

Product Manager: Kwame Odame has over 7 years of high-tech experience working in product management and software engineering. At Meta, Kwame is currently responsible for building the product management direction for Fan Engagement on Facebook. Kwame has also helped build Mastercard’s SaaS authentication platform, enabling cardholders to quickly confirm their identity when a suspicious transaction occurred, leveraging biometric technology. 

‍

UX Researcher (UXR): Fatimah Richmond is a well-rounded UX researcher with over 15 years of experience, having influenced enterprise products across leading tech giants like Google, SAP, Linkedin, and Microsoft. Fatimah has led strategy for research, programs and operations that have significantly impacted the UXR landscape, from clinician engagement strategist to reshaping Linkedin Recruiter and Jobs. As a forward thinker, she’s here to challenge our assumptions and the status quo on how research gets planned, communicated, and measured.

‍

Career Coach: An Xia spent the first decade of her professional life in consulting and Big Tech data science (Netflix, Meta). As a career coach, An has supported clients in gaining clarity on their career goals, navigating challenges of career growth, and making successful transitions. As a somatic coach, An has helped clients tap into the wisdom of their soma to reconnect with what truly matters to them. 

‍

UX Strategist: Natalie Gauvin is an experienced professional with a demonstrated history of purpose-driven work in agile software development industries and higher education. Skilled in various research methodologies. Doctor of Philosophy (Ph.D.) Candidate in Learning Design and Technology from the University of Hawaii at Manoa, focused on empathy in user experience through personas

‍

Level up your UXR capabilities (for free!) with the Optimal Academy 📚

‍

Here at Optimal we really care about helping UX researchers level up their career. This is why we’ve developed the Optimal Academy, to help you master your Optimal Workshop skills and learn more about user research and information architecture.

‍

Check out some of our free courses here: https://academy.optimalworkshop.com/

Hello, my name is Rick and I’m a sociologist. All together, “Hi, Rick!” Now that we’ve got that out of the way, let me tell you about how I use card sorting in my research. I'll soon be running a series of in-person, moderated card sorting sessions. This article covers why card sorting is an integral part of my research, and how I've designed the study toanswer specific questions about two distinct parts of society.

‍

Card sorting to establish how different people comprehend their worlds

‍

Card sorting,or pile sorting as it’s sometimes called, has a long history in anthropology, psychology and sociology. Anthropologists, in particular, have used it to study how different cultures think about various categories. Researchers in the 1970s conducted card sorts to understand how different cultures categorize things like plants and animals. Sociologists of that era also used card sorts to examine how people think about different professions and careers. And since then, scholars have continued to use card sorts to learn about similar categorization questions.

‍

In my own research, I study how different groups of people in the United States imagine the category of 'religion'. Asthose crazy 1970s anthropologists showed, card sorting is a great way to understand how people cognitively understand particular social categories. So, in particular,I’m using card sorting in my research to better understand how groups of people with dramatically different views understand 'religion' — namely, evangelical Christians and self-identified atheists. Thinkof it like this. Some people say that religion is the bedrock of American society.

‍

Others say that too much religion in public life is exactly what’s wrong with this country. What's not often considered is these two groups oftenunderstand the concept of 'religion' in very different ways. It’s like the group of blind men and the elephant: one touches the trunk, one touches the ears, and one touches the tail. All three come away with very different ideas of what an elephant is. So you could say that I study how different people experience the 'elephant' of religion in their daily lives. I’m doing so using primarily in-person moderated sorts on an iPad, which I’ll describe below.

‍

How I generated the words on the cards

‍

The first step in the process was to generate lists of relevant terms for my subjects to sort. Unlike in UX testing, where cards for sorting might come from an existing website, in my world these concepts first have to be mined from the group of people being studied. So the first thing I did was have members of both atheist and evangelical groups complete a free listing task. In a free listing task, participants simply list as many words as they can that meet the criteria given. Sets of both atheist and evangelical respondents were given the instructions: "What words best describe 'religion?' Please list as many as you can.” They were then also asked to list words that describe 'atheism', 'spirituality', and 'Christianity'.

‍

I took the lists generated and standardizedthem by combining synonyms. For example, some of my atheists used words like 'ancient', 'antiquated', and 'archaic' to describe religion. SoI combined all of these words into the one that was mentioned most: 'antiquated'. By doing this, I created a list of the most common words each group used to describe each category. Doing this also gave my research another useful dimension, ideal for exploring alongside my card sorting results. Free lists can beanalyzed themselves using statistical techniques likemulti-dimensional scaling, so I used this technique for apreliminary analysis of the words evangelicals used to describe 'atheism':

‍

‍

Now that I’m armed with these lists of words that atheist and evangelicals used to describe religion, atheism etc., I’m about to embark on phase two of the project: the card sort.

‍

Why using card sorting software is a no-brainer for my research

‍

I’ll be conducting my card sorts in person, for various reasons. I have relatively easy access to the specific population that I’m interested in, and for the kind of academic research I’m conducting, in-person activities are preferred. In theory, I could just print the words on some index cards and conduct a manual card sort, but I quickly realized that a software solution would be far preferable, for a bunch of reasons.

‍

First of all, it's important for me to conductinterviews in coffee shops and restaurants, and an iPad on the table is, to put it mildly, more practical than a table covered in cards — no space for the teapot after all.

‍

Second, usingsoftwareeliminates the need for manual data entry on my part. Not only is manual data entry a time consuming process, but it also introduces the possibly of data entry errors which may compromise my research results.

‍

Third, while the bulk of the card sorts are going to be done in person, having an online version will enable meto scale the project up after the initial in-person sorts are complete. The atheist community, in particular, has a significant online presence, making a web solution ideal for additional data collection.

‍

Fourth, OptimalSort gives the option to re-direct respondents after they complete a sort to any webpage, which allows multiple card sorts to be daisy-chained together. It also enables card sorts to be easily combined with complex survey instruments from other providers (e.g. Qualtrics or Survey Monkey), so card sorting data can be gathered in conjunction with other methodologies.

‍

Finally, and just as important, doing card sorts on a tablet is more fun for participants. After all, who doesn’t like to play with an iPad? If respondents enjoy the unique process of the experiment, this is likely to actually improve the quality of the data, andrespondents are more likely to reflect positively on the experience, making recruitment easier. And a fun experience also makes it more likely that respondents will complete the exercise.

‍

What my in-person, on-tablet card sorting research will look like

‍

Respondents will be handed an iPad Air with 4G data capability. While the venues where the card sorts will take place usually have public Wi-Fi networks available, these networks are not always reliable, so the cellular data capabilities are needed as a back-up (and my pre-testing has shown that OptimalSort works on cellular networks too).

‍

The iPad’s screen orientation will be locked to landscape and multi-touch functions will be disabled to prevent respondents from accidentally leaving the testing environment. In addition, respondents will have the option of using a rubber tipped stylus for ease of sorting the cards. While I personally prefer to use a microfiber tipped stylus in other applications, pre-testing revealed that an old fashioned rubber tipped stylus was easier for sorting activities.

‍

‍

When the respondent receives the iPad, the card sort first page with general instructions will already be open on the tablet in the third party browser Perfect Web. A third party browser is necessary because it is best to run OptimalSort locked in a full screen mode, both for aesthetic reasons and to keep the screen simple and uncluttered for respondents. Perfect Web is currently the best choice in the ever shifting app landscape.

‍

‍

I'll give respondents their instructions and then go to another table to give them privacy (because who wants the creepy feeling of some guy hanging over you as you do stuff?). Altogether, respondents will complete two open card sorts and a fewsurvey-style questions, all chained together by redirect URLs. First, they'll sort 30 cards into groups based on how they perceive 'religion', and name the categories they create. Then, they'll complete a similar card sort, this time based on how they perceive 'atheism'.

‍

Both atheist and evangelicals will receive a mixture of some of the top words that both groups generated in the earlier free listing tasks. To finish, they'll answer a few questions that will provide further data on how they think about 'religion'. After I’ve conducted these card sorts with both of my target populations, I’ll analyze the resulting data on its own and also in conjunction with qualitative data I’ve already collected via ethnographic research and in-depth interviews. I can't wait, actually. In a few months I’ll report back and let you know what I’ve found.

In 2009, Bob Bailey and Cari Wolfson published apaper entitled “FirstClick Usability Testing: A new methodology for predicting users’ success on tasks”. They’d analyzed 12 scenario-based user tests and concluded that the first click people make is a strong leading indicator of their ultimate success on a given task. Their results were so compelling that we got all excited and created Chalkmark, a tool especially for first click usability testing. It occurred to me recently that we’ve never revisited the original premise for ourselves in any meaningful way.

‍

And then one day I realized that, as if by magic, we’re sitting on quite possibly the world’s biggest database of tree test results. I wondered: can we use these results to back up Bob and Cari’s findings (and thus the relevanceof Chalkmark)?Hell yes we can.So we’ve analyzed tree testing data from millions of responses in Treejack, and we're thrilled (relieved) that it confirmed the findings from the 2009 paper — convincingly.

‍

What the original study found

‍

Bob and Cari analyzed data from twelve usability studies on websites and products ‘with varying amounts and types of content, a range of subject matter complexity, and distinct user interfaces’. They found that people were about twice as likely to complete a task successfully if they got their first click right, than if they got it wrong:

‍

If the first click was correct, the chances of getting the entire scenario correct was 87%If the first click was incorrect, the chances of eventually getting the scenario correct was only 46%

‍

What our analysis of tree testing data has found

We analyzed millions of tree testing responses in our database. We've found that people who get the first click correct are almost three times as likely to complete a task successfully:

‍

If the first click was correct, the chances of getting the entire scenario correct was 70%If the first click was incorrect, the chances of eventually getting the scenario correct was 24%

‍

To give you another perspective on the same data, here's the inverse:

‍

If the first click was correct, the chances of getting the entire scenario incorrect was 30%If the first click was incorrect, the chances of getting the whole scenario incorrect was 76%

‍

How Treejack measures first clicks and task success

‍

Bob and Cari proved the usefulness of the methodology by linking two key metrics in scenario-based usability studies: first clicks and task success. Chalkmark doesn't measure task success — it's up to the researcher to determine as they're setting up the study what constitutes 'success', and then to interpret the results accordingly. Treejack does measure task success — and first clicks.

‍

In a tree test, participants are asked to complete a task by clicking though a text-only version of a website hierarchy, and then clicking 'I'd find it here' when they've chosen an answer. Each task in a tree test has a pre-determined correct answer — as was the case in Bob and Cari's usability studies — and every click is recorded, so we can see participant paths in detail.

‍

Thus, every single time a person completes an individual Treejack task, we record both their first click and whether they are successful or not. When we came to test the 'correct first click leads to task success' hypothesis, we could therefore mine data from millions of task.

‍

To illustrate this, have a look at the results for one task.The overall Task result, you see a score for success and directness, and a breakdown of whether each Success, Fail, or Skip was direct (they went straight to an answer), or indirect (they went back up the tree before they selected an answer):

‍

‍

In the pietree for the same task, you can look in more detail at how many people went the wrong way froma label (each label representing one page of your website):

‍

‍

In the First Click tab, you get a percentage breakdown of which label people clicked first to complete the task:

‍

‍

And in the Paths tab, you can view individual participant paths in detail (including first clicks), and can filter the table by direct and indirect success, fails, and skips (this table is only displaying direct success and direct fail paths):

‍

‍

How to get busy with first click testing

‍

This analysis reinforces something we already knew that firstclicks matter. It is worth your time to get that first impression right.You have plenty of options for measuring the link between first clicks and task success in your scenario-based usability tests. From simply noting where your participants go during observations, to gathering quantitative first click data via online tools, you'll win either way. And if you want to add the latter to your research, Chalkmark can give you first click data on wireframes and landing pages,and Treejack on your information architecture.

‍

To finish, here's a few invaluable insights from other researchers ongetting the most from first click testing:

‍

• Jeff Sauro details a useful approach to running a first click test, and shares the findings from a test he ran on 13 people.
• An article on Neoinsight describes three common usability problems that first click testing can solve.
• Gianna LaPin describes a first click test she ran on Netflix, VUDU, and Hulu Plus.