Category Archives: OptimalSort

Be saved!

Posted on by

Just before Christmas we rolled out an update for OptimalSort, Treejack and Chalkmark introducing automatic saving of surveys.

This means you no longer have to save your survey manually because it will be saved automatically each time you make a change. You can still hit the save button too if it makes you feel good!

Why autosave? Two reasons:

  1. We want to save you time wherever we can. The less time you spend saving the more time you save. Of course we’re gambling that everything you do is worth saving…
  2. There were a couple of places in the app where it wasn’t very clear whether or not your changes had been saved. Adding new tasks in Treejack or Chalkmark for example. It felt like you’d completely added a task to your survey, and it looked that way, but you had to press the ‘Save’ button. Why!? Why would we assume you don’t want that task until you explicitly say so?! Personally, I feel much better now that we assume you want to keep everything unless you explicitly press ‘Delete’.

In short, we are always working to improve the user experience for our users, just like you are doing for yours by using our usability tools.

But how will I know if my survey has been saved?

Once you’ve made a change to your survey (and click outside the changed text field), we will automatically perform the save for you. While a save is occurring a small spinner will appear on the ‘Save’ button and it will be temporarily disabled. Once the survey has been saved a message will appear underneath the button saying “Save successful”. How appropriate :-)

Online or offline card sorting?

Posted on by

From time to time I hear people say that they prefer online card sorting to offline card sorting or vice versa. I think they complement each other! (and you should do both)

Let’s start with a quick rundown on the major differences in outcomes from moderated and unmoderated card sorting.

Remote & Unmoderated Card Sorting (Online):

  • Unlimited scale. You can have as many participants as required to get the answer you need.
  • Much closer to “fire & forget”. Set up a study, fire it out to potential participants, enjoy the afternoon in the sun.
  • Relatively cheap. Compared to the cost of having a facilitator, note taker, clients on site, reception, coffees, compensation… remote testing is clearly cheaper to conduct.
  • It can be difficult to know why things happen. Qualitative results are not nearly as apparent because participants are not facilitated, moderated, steered and often not recorded. You don’t get to hear them thinking out loud or discussing decisions.
  • Great for gathering quantitative results. If you have a hunch of your own or as a result of qualitative tests then remote and unmoderated user testing is a great way to back it up with some numbers.

In-Person & Moderated Card Sorting (Offline):

  • Limited scale. You can only bring in as many participants as you can afford in terms of time and budget.
  • Relatively heavy investment per participant. Each participant will have associated costs and will create work for you. (I’m not saying it isn’t worthwhile, it generally is, I’m just pointing out the differences)
  • Great for gathering qualitative results. This is where you get insight into how people feel about what they’re doing or saying in the study.
  • It is usually too expensive to get quantitative results from moderated testing. Yes, you will undoubtedly uncover most of the problems and convince yourself that something must be done, but many situations call for more.

So what should you do?

I recommend that people conduct from 1 to 5 offline, in-person and moderated card sorts to get a good understanding for themselves of how other people would organise their content and the rationale for it. Then I suggest that people conduct an online study using OptimalSort to put some numbers behind the hunches. By the way, I don’t mean to belittle any professional observations by calling them hunches, I’m just making the point that however duly convinced you might be it is usually not unreasonable for a stakeholder to want more data if a change will impact thousands or potentially millions of other people (or dollars for that matter).

If you are fortunate enough to have crystal clear direction from your qualitative research to propose an immediate way forward then I suggest you could skip the online card sort and move directly to validating your proposed new information architecture using tree testing. Either way you should be validating your chosen labels and content hierarchy using Treejack after a card sort.

We believe there is so much value in both qualitative and quantitative research techniques that we want you to do both. To assist you with this we have recently implemented an important change to OptimalSort: You can now print your OptimalSort cards (from a generated PDF) for moderated and in-person paper based card sorting and easily get the results back into OptimalSort for analysis alongside your quantitative research data. Hooray!

Step 1: Print the cards

OptimalSort's printed cards

OptimalSort's printed cards are printed with crop marks for easy cutting

Step 2: Sort the cards

OptimalSort paper card sort

Please don't analyse this card sort. I just laid them out to look pretty.

Step 3: Scan the groups back into OptimalSort

OptimalSort works with common barcode scanners

OptimalSort works with common barcode scanners so that you can quickly get your results into the tool for analysis

I’d love to know what you think of this new feature and whether it will be useful to you in your own card sorts. It certainly beats trying to moderate card sorts around a screen or retrospectively entering participant sorts by doing multiple sorts yourself (you know who you are!).

Participant Centric Card Sort Analysis

Posted on by

Two days ago I presented this poster to the crowd at the IA Summit 2011 in Denver, Colorado for the Poster Session. We’re trying to address two issues with Card Sort Analysis and this poster is a discussion piece for a proposed new algorithm for analysis. The two issues:

  1. Current methods for Card Sort Analysis are essentially qualitative. Although this is very useful, there are times when it is desirable to use a larger data set. Quantitative Card Sort Analysis using current methods is difficult, or damn near impossible with hundreds or thousands of results.
  2. Current visualizations for presenting Card Sort Analysis (dendrograms and similarity matrices) are not very helpful at showing alternate popular mental models that might come through in the raw data. Understanding alternate models can help you decide what to put in a sidebar or footer (for example) or provide valuable insights for second tier navigation or even copy writing. Traditionally you would need to wade through a spreadsheet to uncover these insights.
Participant Centric Information Architecture Analysis

Participant Centric Information Architecture Analysis

In short, we test each card sort result against all the others and come up with an “acceptability score” which represents the degree to which each participant agrees with the other results. In this way we can establish which particular results is most acceptable to the population, and from there, we can answer the question: “Of those who do not agree with this particular IA, how would they prefer to group the cards?”.

We have already developed a working prototype of our Participant Centric Analysis Method and hope to integrate the new visualization into OptimalSort in the near future. We’d love to hear any feedback you might have on this new method.

Download the poster (PDF)

Why card sorting loves tree testing

Posted on by

This article was first published on the Global User Research blog.

Card sorting is an effective technique for teasing out the important distinctions in our content inventory. Conducting card sorts is also a great way to gather insights about the nature of the content and your users’ mental models. I like to think of it as an opportunity to ‘load up your brain’ with the information you’ll need to design a well-informed IA. Sam Ng has called it ‘eye-balling’ the data Card sorting produces much more than just a ballpark in which to throw around ideas. However, as you move toward a final candidate for your site structure, you’re entering territory that card sorting simply wasn’t designed for.

When designing an Information Architecture, we start with a collection of loosely related content and work tirelessly to create an information structure that ‘works’ for as many of our users as possible. What we need is a simple way to validate our ideas so we can use our concepts developed through card sorting and refine them based on research and testing. We need a way to find out if our IA is actually going to work.

What card sorting achieves

Structuring information in a way that makes sense to anybody is not easy, let alone designing for everybody – often thousands of users from different perspectives. Even in simple examples, differences in perception and the effects of personal experience will manifest as disagreements about the nature of content and the interpretation of labels.

Card sorting guides the process of determining ‘what should go together.’ Or as I like to say: ‘what should probably go together… maybe.’ Results from a card sort usually require substantial massaging to form an Information Architecture (IA) and that IA still needs to be proven to work.

Picking up where card sorting leaves off

Users process information differently when performing a seek task as opposed to a sort task. Users process information differently when performing a sort task as opposed to a seek task. When in sort mode we are deeply evaluative, applying considerable effort to organize ideas in a coherent manner. In seek mode, we skim through content, readily discarding information we don’t need and selecting quickly when we think we’ve found something – a pretty close approximation of our web browsing habits!

So we take our card sorting insights from our sort mode respondents, and test the resulting draft IA against some ferocious seek mode users.

Tree testing

We’ve established a simple incompatibility between generative IA techniques like card sorting and the end goal of findable content on your website. With this in mind, tree testing aims to get as close as possible to the actual experience of navigating a website while remaining ‘pure’ about testing the IA in isolation.

From Wikipedia:
“Tree testing is a usability technique for evaluating the findability of topics in a website. It is also known as reverse card sorting or card-based classification. Tree testing is done on a simplified text version of your site structure. This ensures that the structure is evaluated in isolation, nullifying the effects of navigational aids, visual design, and other factors.”

Participants are given a task and set about traversing the IA to look for it. Every step they take is recorded for your analytical pleasure. Did they find the right page? Did they take any wrong turns? How long did it take them? I want every detail!

This provides a wealth of information that we can use to pinpoint problem areas in the IA and identify what the problems are likely to be. Tree test analysis is still a human-intensive process, but the data is decidedly more conclusive and easier to interpret when compared to card sorting. The ability to deliver a conclusive test result is as valuable to the IA design process as it is to overcoming project politics. For example:

“When asked to download a purchase order form, forty percent of participants incorrectly set out within the products and services section. Although some of those participants found the correct destination eventually, fifteen percent of the total participants never found the form.”

Unlike full usability testing, tree testing only deals with the IA. This streamlines IA development, as iterative refinement can be done rapidly and with minimal cost. By testing and refining findability early in the project, it is possible to avoid costly late changes that are likely to affect design, content management and copy writing teams. That is, if you are able to push late changes through at all.

Getting started with tree testing

This advice draws upon our experience with client projects and with helping Treejack users around the world to get the most from their tree studies.

One: Task authoring matters. A lot. Don’t ask your participants to “Find XYZ” twelve times in a row. You’ll see the boredom reflected in your results: a high skip rate and plenty of non-sequitur responses. Mix it up a little and create real-world scenarios. If necessary, ask your participants to “imagine” or “suppose” that they are coming at it from a certain perspective. Never use the same language in your task description as a label in your IA. As an example, if you ask participants to investigate a certain variety of your company’s provided services, any label with the word services in it will experience undue attention. Think of another way to phrase the task.

Two: Don’t bother testing your entire IA. Focus on the parts that matter and that you think are worth worrying about. If you write a task to test your “Contact Us” page, you’ve just wasted the precious attention of your participant, which could’ve been used to test something peculiar to your site. The world is very familiar with common navigation metaphors and its not worth your time to verify that hypothesis. This advice also goes for loading up your tree (the IA itself). Use discretion here, but in most cases you can probably leave out the really common ‘boilerplate’ navigation items.

Three: This isn’t a marathon. Ask your participants to complete ten to fifteen tasks. You might have thirty or more tasks in your overall survey, but for each survey participant you’ll want to keep the workload humane and display a subset to each participant. We recommend collecting 40 or more responses to each task. This means for 30 tasks displayed at 10 per user you will need 120 participants to complete your survey.

Four: Ask questions! We’re always here to help. Email support@optimalworkshop.com

New OptimalSort features released

Posted on by

One of the most commonly requested features for OptimalSort is the ability to change the way participants identify themselves.  We used to collect email addresses as the primary way of identifying participants to card sort administrators.  This can be a big problem for some organisations’ privacy policies.

So, OptimalSort now gives you the ability to:

    • Provide any label to identify participants for your card sorts.  Eg, you could use Full name, employee number, favourite food etc.
    • Allow participants to remain anonymous.  They will not be asked for any form of identification and still be able to complete the card sort.
    • Continue to use email as a way to identify participants.

      Other new features in this release include:

      • Automatically end a card sort after a certain number of participant responses has been reached.
      • Option to receive daily summaries by email on the total number of card sort responses.

        A large number of fixes were also deployed, including:

        • Improved cookie detection scripts to ensure that the “No Cookies” page is presented appropriately.
        • Fixes to ensure projects end automatically after a certain date.
        • Correct rendering of apostrophes and commas in a card sort.
        New project setting options in OptimalSort

        New project setting options in OptimalSort