Improving category taxonomy through card sorting and tree testing

In April 2020, to help enhance the User Experience of our platform, improving the category taxonomy of our website was one of our main objectives in Basalam. Basalam is the first and biggest social marketplace in Iran with the mission to connect buyers to producers and sellers of local and homemade products.

The problem

The category taxonomy not only serves as the backbone of our company’s website for our buyers but as a structure to which our vendors have to assign their products. Nevertheless, the content was structured based on what made sense to the company, not the users, and thus inefficient to both our vendors and buyers.

In most cases, the top-level categories showed an endless list of products that did not make sense as the list was too long or generic, leaving the users frustrated.

Hence, the scopes needed to be further defined before a meaningful and manageable list of products was shown.

Moreover, the taxonomy segmentation was not consistent. Several same-level categories were classified based on product usage (e.g., home decor). In contrast, others were divided based on profession (e.g., blacksmiths) or material (e.g., weavings), making the structure incomprehensible and hard for users to find what they were looking for.

How we solved it

One of the primary ways to figure out an organization scheme that best matches users’ mental model is through card sorting. Therefore, to ensure that the redesigned version was as user-friendly as possible, we made card sorting an integral part of the process.

Card sorting is a UX research method in which study participants group individual labels written on note cards according to criteria that make sense to them. This method uncovers how the target audience’s domain knowledge is structured, and it serves to create an information architecture that matches users’ expectations.

We chose an open card sort rather than a closed sort because we wanted to determine how users grouped Basalam’s content rather than imposing groups upon them.

In our experience, closed card sorting is best suited to situations where you either need to validate an information architecture with end-users or add content to an information architecture that has already been validated with end-users. Since Basalam’s pre-existing category taxonomy met neither of these criteria, we decided to go for an open card sorting, which involves presenting users with a list of note cards consisting of pieces of information that could be found on a site. Users are asked to group these in a way that makes sense to them and to name these groups afterward.

The basic categories

Each of the 300,000+ products on our website had an attached tag connecting them to their relevant product collection. Our engineering team assembled a category tree based on these tags and collections.

We refined the category tree through comparative analysis, making a basic categorization scheme to start things off with

Eventually, we plugged it into OptimalSort, a web-based card sorting application that digitizes the typical pen-and-paper card sorting process.

The major challenge was that there were 180 items to be sorted. The greater the number of items to be sorted, the more effort is required to identify patterns. And if it is time-consuming for users to do the sort, the likelihood of them not completing the exercise would be higher.

So we came up with a new solution. We divided the 180 items into 3 subsets, putting the items that were more similar in nature in the same subsets. We now had 3 separate exercises to be sorted.

To better track and speed up the process, we managed things in a Google Sheet table listing all the steps to be done, owners of each task and its status.

The process

And for the purpose of making sure our users had a good understanding of what needed to be done, we provided them with an instruction guideline.

Recruiting users

It is often said that user testing your website with one person is better than testing it with no one at all. The same is true of card sorting. However, in our experience, your results are only as good as your recruitment. If you don’t take the appropriate level of care in vetting and

selecting your users, you can end up redesigning your site based on the input of people who will never need to use it. This is obviously less than ideal.

Thus, we always make sure we recruit participants that are as representative as possible of the real-life end-users of the websites we test.

We had 3 sorts for each of which we recruited a set of about 50 of our different user segments and offered them a discount coupon for their next purchases from Basalam if they would participate in the exercise.

Although Tullis and Wood recommend testing 20–30 users for card sorting, we recruited 50 users for each test because first of all for card sorts with more than 30 cards it is better to recruit more users, and second there are some cases in which you would later have to exclude from the test results as their answers significantly vary from others.

Dendrogram analysis

Once all users had completed our card sorting exercise, we could begin our analysis process. If we had run the study in person we would probably have gone crazy at this point with 150 different results! Thankfully, as we’d used OptimalSort to run the study, this laborious data preparation step had already been taken care of.

Thus when our study was complete, all of the data was right there — in the right format — waiting for us to begin our analysis.

OptimalSort also has a nifty function that generates dendrograms, a form of tree diagram used to illustrate the clustering of items.

With 150 respondents, a big dendrogram was generated from our card sorting exercise. You can see a part of it below:

Basalam Card Sorting Dendrogram

Matrix analysis

We merged sub-categories with a high agreement rate into one top-level category. For those sub-categories with a low agreement rate, however, we looked at the similarity matrix results to find their right parents.

Basalam Card Sorting Similarity Matrix

Analysis

Using the different sorts of data tables in OptimalSort, we were able to assemble our category tree based on users’ expectations quickly.

Tree Testing

While card sorting was a helpful technique in identifying groupings, as there were 13 cards placed in different groups by different users and as a result had a really low agreement rate, we needed to employ a different user research methodology to test and validate the hypothetical hierarchy generated by our card sorting exercise. Tree testing is such a methodology and presents the user with a sample navigation menu which they are to navigate to achieve assigned tasks. Understanding their navigation pathways would help shed light on the hierarchy and placement of information under the groupings.

As the tree test results showed, some items were placed into correct categories that needed to remain the same, as if in the case of “Mushroom” that most of our users expected to be found under “Fresh Fruit and Vegetable” category.

Additionally, there were some items we had put in the wrong categories. As an example 90% of our users expected to find “Curtain” in the “Home Appliances” category, and not “Decorative”.

Yet there were still some disagreements. There were two groups of categories for a single item in a couple of cases. For instance, 43% of users had put “Dried Vegetables” — used in Persian cuisine — in the “Fresh Fruit and Vegetable” category, while 53% expected it to be found in the “Dried Goods” category. In such cases, we put the items in both categories so that both groups could easily find their desired products.

Conclusion

We were able to determine that frequently the language of our website taxonomy did not align with the language of our users. Consequently, we conducted worthwhile and compelling comparative analysis, card sorting, and tree testing to identify our users’ needs and how they relate to our product. We revealed the users’ mental models of Basalam category taxonomy using these three methodologies.

Through card sorting and tree testing as two critical parts of our research, we could help differentiate between what users said they thought and what they actually thought. The study, in turn, enabled a practical, realistic, and user-centered category taxonomy to emerge.