December 5, 2022

Live training: How to benchmark an existing site structure using Treejack

If you missed our live training, don’t worry, we’ve got you covered! In this session, our product experts Katie and Aidan discuss why, how and when to benchmark an existing structure using Treejack.

They also talk through some benchmarking use cases, demo how to compare tasks between different studies, and which results are most helpful.

Share this article
Author
Sarah
Flutey

Related articles

View all blog articles
Learn more
1 min read

Collating your user testing notes

It’s been a long day. Scratch that - it’s been a long week! Admit it. You loved every second of it.

Twelve hour days, the mad scramble to get the prototype ready in time, the stakeholders poking their heads in occasionally, dealing with no-show participants and the excitement around the opportunity to speak to real life human beings about product or service XYZ. Your mind is exhausted but you are buzzing with ideas and processing what you just saw. You find yourself sitting in your war room with several pages of handwritten notes and with your fellow observers you start popping open individually wrapped lollies leftover from the day’s sessions. Someone starts a conversation around what their favourite flavour is and then the real fun begins. Sound familiar? Welcome to the post user testing debrief meeting.

How do you turn those scribbled notes and everything rushing through your mind into a meaningful picture of the user experience you just witnessed? And then when you have that picture, what do you do next? Pull up a bean bag, grab another handful of those lollies we feed our participants and get comfy because I’m going to share my idiot-proof, step by step guide for turning your user testing notes into something useful.

Let’s talk

Get the ball rolling by holding a post session debrief meeting while it’s all still fresh your collective minds. This can be done as one meeting at the end of the day’s testing or you could have multiple quick debriefs in between testing sessions. Choose whichever options works best for you but keep in mind this needs to be done at least once and before everyone goes home and forgets everything. Get all observers and facilitators together in any meeting space that has a wall like surface that you can stick post its to - you can even use a window! And make sure you use real post its - the fake ones fall off!

Mark your findings (Tagging)

Before you put sharpie to post it, it’s essential to agree as a group on how you will tag your observations. Tagging the observations now will make the analysis work much easier and help you to spot patterns and themes. Colour coding the post its is by far the simplest and most effective option and how you assign the colours is entirely up to you. You could have a different colour for each participant or testing session, you could have different colours to denote participant attributes that are relevant to your study eg senior staff and junior staff, or you could use different colours to denote specific testing scenarios that were used. There’s many ways you could carve this up and there’s no right or wrong way. Just choose the option that suits you and your team best because you’re the ones who have to look at it and understand it. If you only have one colour post it eg yellow, you could colour code the pen colours you use to write on the notes or include some kind of symbol to help you track them.

Processing the paper (Collating)

That pile of paper is not going to process itself! Your next job as a group is to work through the task of transposing your observations to post it notes. For now, just stick them to the wall in any old way that suits you. If you’re the organising type, you could group them by screen or testing scenario. The positioning will all change further down the process, so at this stage it’s important to just keep it simple. For issues that occur repeatedly across sessions, just write them down on their own post its- doubles will be useful to see further down the track.In addition to  holding a debrief meetings, you also need to round up everything that was used to capture the testing session/s. And I mean EVERYTHING.

Handwritten notes, typed notes, video footage and any audio recordings need to be reviewed just in case something was missed. Any handwritten notes should be typed to assist you with the completion of the report. Don’t feel that you have to wait until the testing is completed before you start typing up your notes because you will find they pile up very quickly and if your handwriting is anything like mine…. Well let’s just say my short term memory is often required to pick up the slack and even that has it’s limits. Type them up in between sessions where possible and save each session as it’s own document. I’ll often use the testing questions or scenario based tasks to structure my typed notes and I find that makes it really easy to refer back to.Now that you’ve processed all the observations, it’s time to start sorting your observations to surface behavioural patterns and make sense of it all.

Spotting patterns and themes through affinity diagramming

Affinity diagramming is a fantastic tool for making sense of user testing observations. In fact it’s just about my favourite way to make sense of any large mass of information. It’s an engaging and visual process that grows and evolves like a living creature taking on a life of its own. It also builds on the work you’ve just done which is a real plus!By now, testing is over and all of your observations should all be stuck to a wall somewhere. Get everyone together again as a group and step back and take it all in. Just let it sit with you for a moment before you dive in. Just let it breathe. Have you done that? Ok now as individuals working at the same time, start by grouping things that you think belong together. It’s important to just focus on the content of the labels and try to ignore the colour coded tagging at this stage, so if session one was blue post its don’t group all the blue ones together just because they’re all blue! If you get stuck, try grouping by topic or create two groups eg issues and wins and then chunk the information up from there.

You will find that the groups will change several times over the course of the process  and that’s ok because that’s what it needs to do.While you do this, everyone else will be doing the same thing - grouping things that make sense to them.  Trust me, it’s nowhere near as chaotic as it sounds! You may start working as individuals but it won’t be long before curiosity kicks in and the room is buzzing with naturally occurring conversation.Make sure you take a step back regularly and observe what everyone else is doing and don’t be afraid to ask questions and move other people’s post its around- no one owns it! No matter how silly something may seem just put it there because it can be moved again. Have a look at where your tagged observations have ended up. Are there clusters of colour? Or is it more spread out? What that means will depend largely on how you decided to tag your findings. For example if you assigned each testing session its own colour and you have groups with lot’s of different colours in them you’ll find that the same issue was experienced by multiple people.Next, start looking at each group and see if you can break them down into smaller groups and at the same time consider the overall picture for bigger groups eg can the wall be split into say three high level groups.Remember, you can still change your groups at anytime.

Thinning the herd (Merging)

Once you and your team are happy with the groups, it’s time to start condensing the size of this beast. Look for doubled up findings and stack those post its on top of each other to cut the groups down- just make sure you can still see how many there were. The point of merging is to condense without losing anything so don’t remove something just because it only happened once. That one issue could be incredibly serious. Continue to evaluate and discuss as a group until you are happy. By now clear and distinct groups of your observations should have emerged and at a glance you should be able to identify the key findings from your study.

A catastrophe or a cosmetic flaw? (Scoring)

Scoring relates to how serious the issues are and how bad the consequences of not fixing them are. There are arguments for and against the use of scoring and it’s important to recognise that it is just one way to communicate your findings.I personally rarely use scoring systems. It’s not really something I think about when I’m analysing the observations. I rarely rank one problem or finding over another. Why? Because all data is good data and it all adds to the overall picture.I’ve always been a huge advocate for presenting the whole story and I will never diminish the significance of a finding by boosting another. That said, I do understand the perspective of those who place metrics around their findings. Other designers have told me they feel that it allows them to quantify the seriousness of each issue and help their client/designer/boss make decisions about what to do next.We’ve all got our own way of doing things, so I’ll leave it up to you to choose whether or not you score the issues. If you decide to score your findings there are a number of scoring systems you can use and if I had to choose one, I quite like Jakob Nielsen’s methodology for the simple way it takes into consideration multiple factors. Ultimately you should choose the one that suits your working style best.

Let’s say you did decide to score the issues. Start by writing down each key finding on it’s own post it and move to a clean wall/ window. Leave your affinity diagram where it is. Divide the new wall in half: one side for wins eg findings that indicate things that tested well and the other for issues. You don’t need to score the wins but you do need to acknowledge what went well because knowing what you’re doing well is just as important as knowing where you need to improve. As a group (wow you must be getting sick of each other! Make sure you go out for air from time to time!) score the issues based on your chosen methodology.Once you have completed this entire process you will have everything you need to write a kick ass report.

What could possibly go wrong? (and how to deal with it)

No process is perfect and there are a few potential dramas to be aware of:

People jumping into solution mode too early

In the middle of the debrief meeting, someone has an epiphany. Shouts of We should move the help button! or We should make the yellow button smaller! ring out and the meeting goes off the rails.I’m not going to point fingers and blame any particular role because we’ve all done it, but it’s important to recognise that’s not why we’re sitting here. The debrief meeting is about digesting and sharing what you and the other observers just saw. Observing and facilitating user testing is a privilege. It’s a precious thing that deserves respect and if you jump into solution mode too soon, you may miss something. Keep the conversation on track by appointing a team member to facilitate the debrief meeting.

Storage problems

Handwritten notes taken by multiple observers over several days of testing adds up to an enormous pile of paper. Not only is it a ridiculous waste of paper but they have to be securely stored for three months following the release of the report. It’s not pretty. Typing them up can solve that issue but it comes with it’s own set of storage related hurdles. Just like the handwritten notes, they need to be stored securely. They don’t belong on SharePoint or in the share drive or any other shared storage environment that can be accessed by people outside your observer group. User testing notes are confidential and are not light reading for anyone and everyone no matter how much they complain. Store any typed notes in a limited access storage solution that only the observers have access to and if anyone who shouldn’t be reading them asks, tell them that they are confidential and the integrity of the research must be preserved and respected.

Time issues

Before the storage dramas begin, you have to actually pick through the mountain of paper. Not to mention the video footage, and the audio and you have to chase up that sneaky observer who disappeared when the clock struck 5. All of this takes up a lot of time. Another time related issue comes in the form of too much time passing in between testing sessions and debrief meetings. The best way to deal with both of these issues  is to be super organised and hold multiple smaller debriefs in between sessions where possible. As a group, work out your time commitments before testing begins and have a clear plan in place for when you will meet.  This will prevent everything piling up and overwhelming you at the end.

Disagreements over scoring

At the end of that long day/week we’re all tired and discussions around scoring the issues can get a little heated. One person’s showstopper may be another person’s mild issue. Many of the ranking systems use words as well as numbers to measure the level of severity and it’s easy to get caught up in the meaning of the words and ultimately get sidetracked from the task at hand. Be proactive and as a group set ground rules upfront for all discussions. Determine how long you’ll spend discussing an issue and what you will do in the event that agreement cannot be reached. People want to feel heard and they want to feel like their contributions are valued. Given that we are talking about an iterative process, sometimes it’s best just to write everything down to keep people happy and merge and cull the list in the next iteration. By then they’ve likely had time to reevaluate their own thinking.

And finally...

We all have our own ways of making sense of our user testing observations and there really is no right or wrong way to go about it. The one thing I would like to reiterate is the importance of collaboration and teamwork. You cannot do this alone, so please don’t try. If you’re a UX team of one, you probably already have a trusted person that you bounce ideas off. They would be a fantastic person to do this with. How do you approach this process? What sort of challenges have you faced? Let me know in the comments below.

Learn more
1 min read

67 ways to use Optimal for user research

User research and design can be tough in this fast-moving world. Sometimes we can get so wrapped up in what we’re doing, or what we think we’re supposed to be doing, that we don’t take the time to look for other options and other ways to use the tools we already know and love. I’ve compiled this list over last few days (my brain hurts) by talking to a few customers and a few people around the office. I’m sure it's far from comprehensive. I’ve focused on quick wins and unique examples. I’ll start off with some obvious ones, and we’ll get a little more abstract, or niche, as we go. I hope you get some ideas flying as you read through, enjoy!

#1 Benchmark your information architecture (IA)

Without a baseline for your information architecture, you can’t easily tell if any changes you make have a positive effect. If you haven’t done so, benchmark your existing website on Tree testing now. Upload your site structure and get results the same day. Now you’ll have IA scores to beat each month. Easy.

#2 Find out precisely where people get lost

Use Tree testing Pietree to find out exactly where people are getting lost in your website structure and where they go instead. You can also use First-click testing for this if you’re only interested in the first click, and let’s face it, that is where you’ll get the biggest bang for your buck.

#3 Start at the start

If you’re just not sure where to begin then take a screenshot of your homepage, or any page that you think might have some issues and get going with First-click testing. Write up a string of things that people might want to do when they find themselves on this page and use these as your tasks. Surprise all your colleagues with a maddening heatmap showing where people actually clicked in response to your tasks. Now you’ll know have a better idea of which area of your site to focus a tree test or card sort on for your next step.

#4 A/B test your site structure

Tree testing is great for testing more than one content structure. It’s easy to run two separate Tree testing studies — even more than two. It’ll help you decide which structure you and your team should run with, and it won’t take you long to set them up. Learn more.

#5 Make collaborative design decisions

Use Optimal Sort to get your team involved and let their feedback feed your designs — logos, icons, banners, images, the list goes on. By creating a closed image sort with categories where your team can group designs based on their preferences, you can get some quick feedback to help you figure out where you should focus your efforts.

#6 Do your (market) research

Card sorting is a great UX research technique, but it can also be a fun way to involve your users in some market research. Get a better sense of what your users and customers actually want to see on your website, by conducting an image sort of potential products. By providing categories like ‘I would buy this’, ‘I wouldn’t buy this’ to indicate their preferences for each item, you can figure out what types of products appeal to your customers.

#7 Customer satisfaction surveys with surveys

The thoughts and feelings of your users are always important. A simple survey can help you take a deeper look at your checkout process, a recently launched product or service, or even on the packaging your product arrives in — your options are endless.


#8 Crowdsource content ideas

Whether you’re running a blog or a UX conference, Questions can help you generate content ideas and understand any knowledge gaps that might be out there. Figure out what your users and attendees like to read on your blog, or what they want to hear about at your event, and let this feed into what you offer.

#9 Do some sociological research

Using card sorting for sociological research is a great way to deepen your understanding of how different groups may categorize information. Rather than focusing solely on how your users interact with your product or service, consider broadening your research horizons to understand your audience’s mental models. For example, by looking at how young people group popular social media platforms, you can understand the relationships between them, and identify where your product may fit in the mix.

#10 Create tests to fit in your onboarding process

Onboarding new customers is crucial to keeping them engaged with your product, especially if it involves your users learning how to use it. You can set up a quick study to help your users stay on track with onboarding. For example, say your company provided online email marketing software. You can set up a First-click testing study using a photo of your app, with a task asking your participants where they’d click to see the open rates for a particular email that went out.


#11 Quantify the return on investment of UX

Some people, including UX Agony Aunt, define return on UX as time saved, money made, and people engaged. By attaching a value to the time spent completing tasks, or to successful completion of tasks, you can approximate an ROI or at least illustrate the difference between two options.


#12 Collate all your user testing notes using qualitative Insights

Making sense of your notes from qualitative research activities can be simultaneously exciting and overwhelming. It’s fun being out on the field and jotting down observations on a notepad, or sitting in on user interviews and documenting observations into a spreadsheet. You can now easily import all your user research and give it some traceability.


#13 Establish which tags or filters people consider to be the most important

Create a card sort with your search filters or tags as labels, and have participants rank them according to how important they consider them to be. Analytics can tell you half of the story (where people actually click), so the card sort can give another side: a better idea of what people actually think or want.

#14 Reduce content on landing pages to what people access regularly

Before you run an open card sort to generate new category ideas, you can run a closed card sort to find out if you have any redundant content. Say you wanted to simplify the homepage of your intranet. You can ask participants to sort cards (containing homepage links) based on how often they use them. You could compare this card sort data with analytics from your intranet and see if people’s actual behavior and perception are well aligned.

#15 Crowd-source the values you want your team/brand/product to represent

Card sorting is a well-established technique in the ‘company values’ realm, and there are some great resources online to help you and your team brainstorm the values you represent. These ‘in-person’ brainstorm sessions are great, and you can run a remote closed card sort to support your findings. And if you want feedback from more than a small group of people (if your company has, say, more than 15 staff) you can run a remote closed card sort on its own. Use Microsoft’s Reaction Card Method as card inspiration.

#16 Input your learnings and observations from a UX conference with qualitative insights

If you're lucky enough to attend a UX conference, you can now share the experience with your colleagues. You can easily jot down ideas quotes and key takeaways in a Reframer project and keep your notes organized by using a new session for each presenter Bonus — if you’re part of a team, they can watch the live feed rolling into Reframer!


#17 Find out what actions people take across time

Use card sorting to understand when your participants are most likely to perform certain activities over the course of a day, week, or over the space of a year. Create categories that represent time, for example, ‘January to March’, ‘April to June’, ‘July to September’, and ‘October to December’, and ask your participants to sort activities according to the time they are most likely to do them (go on vacation, do their taxes, make big purchases, and so on). While there may be more arduous and more accurate methods for gathering this data, sometimes you need quick insights to help you make the right decisions.


#18 Gather quantitative data on prioritizing project tasks or product features

Closed card sorting can give you data that you might usually gather in team meetings or in Post-its on the wall, or that you might get through support channels. You can model your method on other prioritization techniques, including Eisenhower’s Decision Matrix, for example.

#19 Test your FAQs page with new users

Your support and knowledge base within your website can be just as important as any other core action on your website. If your support site is lacking in navigation and UX, this will no doubt increase support tickets and resources. Make sure your online support section is up to scratch. Here’s an article on how to do it quickly.

#20 Figure out if your icons need labels

Figure out if your icons are doing their job by testing whether your users are understanding them as intended. Uploading icons you currently use, or plan to use in your interface to First-click testing, and ask your users to identify their meaning by making use of post-task questions.

#21 Give your users some handy quick tools

In some cases, users may use your website with very specific goals in mind. Giving your users access to quick toos as soon as they land on your website is a great way to ensure they are able to get what they need done easily. Look at your analytics for things people do often that take several clicks to find, and check whether they can find your ‘quick tool’ in a single click using First-click testing.

#22 Benchmark the IA of your competition

We all have some sort of competitors, and researchers also need to pay attention to what they get up too. Make life easy in your reporting by benchmarking their IA and then reviewing it each quarter for the board and leaders to be wowed with. Also, not a perfect comparison, as users and separate sites have different flows, but compare your success scores with theirs. Makes your work feel like the Olympics with the healthy competition going on.

#23 Improve website conversions

Make the marketing team’s day by doing a fast improvement on some core conversions on your website. Now, there are loads of ways to improve conversions for a check out cart or signup form, but using First-click testing to test out ideas before you start going live A/B test can take mere minutes and give your B version a confidence boost.

#24 Reduce the bounce rates of certain sections of your website

People jumping off your website and not continuing their experience is something (depending on the landing page) everyone tries to improve. The metric ‘time on site’ and ‘average page views’ is a metric that shows the value your whole website has to offer. Again, there are many different ways to do this, but one big reason for people jumping off the website is not being able to find what they’re looking for. That’s where our IA toolkit comes in.

#25 Test your website’s IA in different countries

No, you don’t have to spend thousands of dollars to go to all these countries to test, although that’d be pretty sweet. You can remotely research participants from all over the world, using our integrated recruitment panel. Start seeing how different cultures, languages, and countries interact with your website.

#26 Run an empathy test (card sort)

Empathy – the ability to understand and share the experience of another person – is central to the design process. An empathy test is another great tool to use in the design phase because it enables you to find out if you are creating the right kind of feelings with your user. Take your design and show it to users. Provide them with a variety of words that could represent the design – for example “minimalistic”, “dynamic”, or “professional” – and ask them to pick out which the words which they think are best suited to their experience.

#27 Recall test with first-click testing

This is a technique that is a great way to find out if you have the correct visual hierarchy on the page. For example, a client may come to you and say you need to make the logo bigger as they are worried people won’t spot the branding. A recall test will test if this is correct and check whether people are seeing the things you want them to see on the page. Show your design to a user for a small amount of time (five seconds, for example) and ask them to recall what they see, and the order in which they recall it.

#28 Take Qualitative Insights into the field

Get out of the office or the lab and observe social behaviour in the field. Use Qualitative Insights to input your observations on your field research. Then head back to your office to start making sense of the data in the Theme Builder.

#29 Use heatmaps to get the first impressions of designs

Heatmaps in our First-click testing tool are a great way of getting first impressions of any design. You can see where people clicked (correctly and incorrectly), giving you insights on what works and doesn’t work with your designs. Because it’s so fast to test, you can iterate until your designs start singing.

#30 Multivariate testing

Multivariate testing is when more than two versions of your studies are compared and allows you to understand which version performs better with your audience. Use multivariate testing with Tree testing and First-click testing to find the right design on which to focus and iterate.

#31 Improve your search engine optimization (SEO) with tree testing

Yes, a good IA improves your SEO. Search engines want to know how your users navigate throughout your site. Make sure people can easily find what they’re looking for, and you’ll start to see improvement in your search engine ranking.

#32 Test your mobile information architecture

As more and more people are using their smartphones for apps and to browse sites, you need to ensure its design gives your users a great experience. Test the IA of your mobile site to ensure people aren’t getting lost in the mobile version of your site. If you haven’t got a mobile-friendly design yet, now’s the time to start designing it!

#33 Run an Easter egg hunt using the correct areas in first-click testing

Liven up the workday by creating a fun Easter egg hunt in Chalkmark. Simply upload a photo (like those really hard “spot the X” photos), set the correct area of your target, then send out your study with participant identifiers enabled. You can also send these out as competitions and have closing rules based on time, number of participants, or both.

#34 Keystroke level modeling

When interface efficiency is important you'll want to measure how much a new design can improve task times. You can actually estimate time saved (or lost) using some well-tested approaches that are based on average human performance for typical computer-based operations like clicking, pointing and typing. Read more about measuring task times without users.

#35 Feature prioritization — get some help for your roadmap

Find out what people think are the most important next steps for your team. Set up a card sort and ask people to categorize items and rank them in descending order of importance or impact on their work. This can also help you gauge their thoughts on potential new features for your site, and for bonus points compare team responses with customer responses.

#36 Tame your blog

Get the tags and categories in your blog under control to make life easier for your readers. Set up a card sort and use all your tags and categories as card labels. Either use your existing ones or test a fresh set of new tags and categories.

#37 Test your home button

Would an icon or text link work better for navigating to your home page? Before you go ahead and make changes to your site, you can find out by setting up a first-click testing test.

#38 Validate the designs in your head

As designers, you’ve probably got umpteen designs floating around in your head at any one time. But which of these are really worth pursuing? Figure this out by using The Optimal Workshop Suite to test out wireframes of new designs before putting any more work into them.

#39 ‘Buy now’ button shopping cart visibility

If you’re running an e-commerce site, ease of use and a great user experience are crucial. To see if your shopping cart and checkout processes are as good as they can be, run a first-click test.

#40 IA periodic health checks

Raise the visibility of good IA by running periodic IA health checks using Tree testing and reporting the results. Management loves metrics and catching any issues early is good too!

#41 Focus groups with qualitative insights

Thinking of launching a new product, app or website, or seeking opinions on an existing one? Focus groups can provide you with a lot of candid information that may help get your project off the ground. They’re also dangerous because they’re susceptible to groupthink, design by committee, and tunnel vision. Use with caution, but if you do then use with Qualitative Insights! Compare notes and find patterns across sessions. Pay attention to emotional triggers.

#42 Gather opinions with surveys

Whether you want the opinions of your users or from members of your team, you can set up a quick and simple survey using Surveys. It’s super useful for getting opinions on new ideas (consider it almost like a mini-focus group), or even for brainstorming with teammates.

#43 Design a style guide with card sorting

Style guides (for design and content) can take a lot of time and effort to create, especially when you need to get the guide proofed by various people in your company. To speed this up, simply create a card sort to find out what your guide should consist of. Find out the specifics in this article.

#44 Improve your company's CRM system

As your company grows, oftentimes your CRM can become riddled with outdated information and turn into a giant mess, especially if you deal with a lot of customers every day. To help clear this up, you can use card sorting and tree testing to solve navigational issues and get rid of redundant features. Learn more.

#45 Sort your life out

Let your creativity run wild, and get your team or family involved in organizing or prioritizing the things that matter. And the possibilities really are endless. Organize a long list of DIY projects, or ask the broader team how the functional pods should be re-organized. It’s up to you. How can card sorting help you in your work and daily life?

#46 Create an online diary study

Whether it’s a product, app or website, finding out the long-term behaviour and thoughts of your users is important. That’s where diary studies come in. For those new to this concept, diary studies are a longitudinal research method, aimed at collecting insights about a participant’s needs and behaviors. Participants note down activities as they’re using a particular product, app, or website. Add your participants into a qualitative study and allow them to create their diary study with ease.

#47 Source-specific data with an online survey

Online survey tools can complement your existing research by sourcing specific information from your participants. For example, if you need to find out more about how your participants use social media, which sites they use, and on which devices, you can do it all through a simple survey questionnaire. Additionally, if you need to identify usage patterns, device preferences or get information on what other products/websites your users are aware of/are using, a questionnaire is the ticket.

#48 Guerrilla testing with First-click testing

For really quick first-click testing, take First-click testing on a tablet, mobile device or laptop to a local coffee shop. Ask people standing in line if they’d like to take part in your super quick test in exchange for a cup of joe. Easy!

#50 Ask post-task questions for tree testing and first-click testing

You can now set specific task-related questions for both Tree testing and First-click testing. This is a great way to dive deeper into the mushy minds of your participants. Check out how to use this new(ish) feature here!

#51 Start testing prototypes

Paper prototypes are great, but what happens when your users are scattered around the globe, and you can’t invite them to an in-person test? By scanning (or taking a photo) of your paper prototypes, you can use Chalkmark to test them with your users quickly and easily. Read more about our approach here.

#52 Take better notes for sense making

Qualitative research involves a lot of note-taking. So naturally, to be better at this method, improving how you take notes is important. Reframer is designed to make note-taking easy but it can still be an art. Learn more.

#53 Make sure you get the user's first-click right

Like most things, read a little, and then it’s all about practice.We’ve found that people who get the first click correct are almost three times as likely to complete a task successfully. Get your first clicks right in Treejack and Chalkmark and you’ll start seeing your customers smile.


#54 Run a cat survey. Yep, cats!

We’ve gained some insight into how people intuitively group cats, and so can you (unless you’re a dog person). Honestly, doing something silly can be a useful way to introduce your team to a new method on a Friday afternoon. Remember to distribute the results!


#55 Destroy evil attractors in your tree

Evil attractors are those labels in your IA that attract unjustified clicks across tasks. This usually means the chosen label is ambiguous, or possibly a catch-all phrase like ‘Resources’. Read how to quickly identify evil attractors in the Destinations table of tree test results and how to fix them.

#56 Affinity map using card sorts

We all love our Post-its and sticking things on walls. But sometimes you need something quicker and accessible for people in remote areas. Try out using Card Sorts for a distributed approach to making sense of all the notes. Plus, you can easily import any qualitative insights when creating cards in card sort. Easy.

#57 Preference test with first-click testing

Whether you’re coming up with a new logo design, headline, featured image, or anything, you can preference test it with First-click testing. Create an image that shows the two designs side by side and upload it to First-click testing. From there, you can ask people to click whichever one they prefer!

#58 Add moderated card sort results to your card sort

An excellent way of gathering valuable qualitative insights alongside the results of your remote card sorts is to run a moderated version of the sorts with a smaller group of participants. When you can observe and interact with your participants as they complete the sort, you’ll be able to ask questions and learn more about their mental models and the reasons why they have categorized things in a particular way. Learn more.

#59 Test search box variations with first-click clicking

Case study by Viget: “One of the most heavily used features of the website is its keyword search, so we wanted to make absolutely certain that our redesigned search box didn’t make search harder for users to find and use.”

#60 Run an image card sort to organize products into groups

You can add images to each card that allows you understand how your participants may organize and label particular items. Very useful if you want to organize some retail products and want to find out how other people would organize them given a visual including shape, color, and other potential context.

#61 Test your customers' perceptions of different logo and brand image designs

Understand how customers perceive your brand by creating a closed card sort. Come up with a list of categories, and ask participants to sort images such as logos, and branded images.

#62 Run an open image card sort to classify images into groups based on the emotions they elicit

Are these pictures exhilarating, or terrifying? Are they humoros, or offensive? Relaxing, or boring? Productive, or frantic? Happy memories, or a deep sigh?

#63 Run an image card sort to organize your library

Whether it’s a physical library of books, or a digital drive full of ebooks, you can run a card sort to help organize them in a way that makes sense. Will it be by genre, author name, color or topic? Send out the study to your coworkers to get their input! You can also do this at home for your own personal library, and you can include music/CDs/vinyl records and movies!

#64 HR exercises to determine the motivations of your team

It’s simple to ask your team about their thoughts, feelings, and motivations with a Questions survey. You can choose to leave participant identifiers blank (so responses are anonymous), or you can ask for a name/email address. As a bonus, you can set up a calendar reminder to send out a new survey in the next quarter. Duplicate the survey and send it out again!

#65 Designing physical environments

If your company has a physical environment in which your customers visit, you can research new structures using a mixture of tools in The Optimal Workshop Suite. This especially comes in handy if your customers require certain information within the physical environment in order to make decisions. For example, picture a retail store. Are all the signs clear and communicate the right information? Are people overwhelmed by the physical environment?

#66 Use tree testing to refine an interactive phone menu system

Similar to how you’d design an IA, you can create a tree test to design an automated phone system. Whether you’re designing from the ground up, or improving your existing system, you will be able to find out if people are getting lost.


#67 Have your research team categorize and prioritize all these ideas

Before you dig deeper into more of these ideas, ask the rest of the team to help you decide which one to focus on. Let’s not get in the way of your work. Start your quick wins and log into your account. Here’s a spreadsheet of this list to upload to card sort. Aaaaaaaaaaand that’s a wrap! *Takes out gym towel and wipes sweaty face.
*Got any more suggestions to add to this list? We’d love to hear them in our comments section — we might even add them into this list

Learn more
1 min read

Ready for take-off: Best practices for creating and launching remote user research studies

"Hi Optimal Work,I was wondering if there are some best practices you stick to when creating or sending out different UX research studies (i.e. Card sorts, Prototyye Test studies, etc)? Thank you! Mary"

Indeed I do! Over the years I’ve learned a lot about creating remote research studies and engaging participants. That experience has taught me a lot about what works, what doesn’t and what leaves me refreshing my results screen eagerly anticipating participant responses and getting absolute zip. Here are my top tips for remote research study creation and launch success!

Creating remote research studies

Use screener questions and post-study questions wisely

Screener questions are really useful for eliminating participants who may not fit the criteria you’re looking for but you can’t exactly stop them from being less than truthful in their responses. Now, I’m not saying all participants lie on the screener so they can get to the activity (and potentially claim an incentive) but I am saying it’s something you can’t control. To help manage this, I like to use the post-study questions to provide additional context and structure to the research.

Depending on the study, I might ask questions to which the answers might confirm or exclude specific participants from a specific group. For example, if I’m doing research on people who live in a specific town or area, I’ll include a location based question after the study. Any participant who says they live somewhere else is getting excluded via that handy toggle option in the results section. Post-study questions are also great for capturing additional ideas and feedback after participants complete the activity as remote research limits your capacity to get those — you’re not there with them so you can’t just ask. Post-study questions can really help bridge this gap. Use no more than five post-study questions at a time and consider not making them compulsory.

Do a practice run

No matter how careful I am, I always miss something! A typo, a card with a label in the wrong case, forgetting to update a new version of an information architecture after a change was made — stupid mistakes that we all make. By launching a practice version of your study and sharing it with your team or client, you can stop those errors dead in their tracks. It’s also a great way to get feedback from the team on your work before the real deal goes live. If you find an error, all you have to do is duplicate the study, fix the error and then launch. Just keep an eye on the naming conventions used for your studies to prevent the practice version and the final version from getting mixed up!

Sending out remote research studies

Manage expectations about how long the study will be open for

Something that has come back to bite me more than once is failing to clearly explain when the study will close. Understandably, participants can be left feeling pretty annoyed when they mentally commit to complete a study only to find it’s no longer available. There does come a point when you need to shut the study down to accurately report on quantitative data and you’re not going to be able to prevent every instance of this, but providing that information upfront will go a long way.

Provide contact details and be open to questions

You may think you’re setting yourself up to be bombarded with emails, but I’ve found that isn’t necessarily the case. I’ve noticed I get around 1-3 participants contacting me per study. Sometimes they just want to tell me they completed it and potentially provide additional information and sometimes they have a question about the project itself. I’ve also found that sometimes they have something even more interesting to share such as the contact details of someone I may benefit from connecting with — or something else entirely! You never know what surprises they have up their sleeves and it’s important to be open to it. Providing an email address or social media contact details could open up a world of possibilities.

Don’t forget to include the link!

It might seem really obvious, but I can’t tell you how many emails I received (and have been guilty of sending out) that are missing the damn link to the study. It happens! You’re so focused on getting that delivery right and it becomes really easy to miss that final yet crucial piece of information.

To avoid this irritating mishap, I always complete a checklist before hitting send:

  • Have I checked my spelling and grammar?
  • Have I replaced all the template placeholder content with the correct information?
  • Have I mentioned when the study will close?
  • Have I included contact details?
  • Have I launched my study and received confirmation that it is live?
  • Have I included the link to the study in my communications to participants?
  • Does the link work? (yep, I’ve broken it before)

General tips for both creating and sending out remote research studies

Know your audience

First and foremost, before you create or disseminate a remote research study, you need to understand who it’s going to and how they best receive this type of content. Posting it out when none of your followers are in your user group may not be the best approach. Do a quick brainstorm about the best way to reach them. For example if your users are internal staff, there might be an internal communications channel such as an all-staff newsletter, intranet or social media site that you can share the link and approach content to.

Keep it brief

And by that I’m talking about both the engagement mechanism and the study itself. I learned this one the hard way. Time is everything and no matter your intentions, no one wants to spend more time than they have to. Even more so in situations where you’re unable to provide incentives (yep, I’ve been there). As a rule, I always stick to no more than 10 questions in a remote research study and for card sorts, I’ll never include more than 60 cards. Anything more than that will see a spike in abandonment rates and of course only serve to annoy and frustrate your participants. You need to ensure that you’re balancing your need to gain insights with their time constraints.

As for the accompanying approach content, short and snappy equals happy! In the case of an email, website, other social media post, newsletter, carrier pigeon etc, keep your approach spiel to no more than a paragraph. Use an audience appropriate tone and stick to the basics such as: a high level sentence on what you’re doing, roughly how long the study will take participants to complete, details of any incentives on offer and of course don’t forget to thank them.

Set clear instructions

The default instructions in Optimal Workshop’s suite of tools are really well designed and I’ve learned to borrow from them for my approach content when sending the link out. There’s no need for wheel reinvention and it usually just needs a slight tweak to suit the specific study. This also helps provide participants with a consistent experience and minimizes confusion allowing them to focus on sharing those valuable insights!

Create a template

When you’re on to something that works — turn it into a template! Every time I create a study or send one out, I save it for future use. It still needs minor tweaks each time, but I use them to iterate my template.What are your top tips for creating and sending out remote user research studies? Comment below!

Seeing is believing

Explore our tools and see how Optimal makes gathering insights simple, powerful, and impactful.