How are Treejack scores calculated?

Treejack scores are provided for three independent dimensions of a participant's performance: success, speed and directness.

Success is the simplest score to understand. It refers to the percentage of participants that have selected a correct answer for a given task (in the case of the overall survey score, this is an average of the scores for each task). Because multiple leaf nodes (endpoints) can be configured as correct by the survey author, success refers to the percentage of participants that selected any of those correct scores for that task.

Directness is a measure of how certain participants were in a task. If the participant moves up the tree at any stage, i.e. they realise that they have gone the wrong way and move up the tree in order to navigate down again on a different branch, this will negatively affect the participant's directness score. Currently this is calculated as the percentage of people who did not backtrack at all when selecting an answer.

Speed is a box plot of the recorded task completion times. We exclude participants who took longer than 100 seconds to complete the task because we've found from our data that people who take longer than two minutes to complete a single task have almost certainly left it idle. We remove these outliers to give a better indication of the actual median completion time.

The overall score in the coloured box is a weighted average of success and directness, favouring success over directness at a ratio of 3:1, scaled to be a value out of 10.