At UX Australia Design Research 2020, Dovetail’s CEO, Benjamin, presented a talk called ‘Research Repositories in 10 minutes’. He spoke about what a research repository is, what problems they are trying to solve and covered some principles to guide decision-making if people were planning to set up one for their organization. Below is a lightly edited transcript of the talk.
It started in 2017 with the article Democratizing UX published by Tomer Sharon, the then Head of User Experience at WeWork. In this article, which went on to become extremely popular, he coined the terms atomic research and nuggets. A couple of years later, Matt from Microsoft published an article called ’How Microsoft’s human insights library creates a living body of knowledge’. This was released in June 2019 after Microsoft built its internal system, HITS. In the article, he talks about collaboration with researchers and how insights should be created throughout the project rather than a big cliff to climb up at the end of the project.
The period of June, July, and August of 2019, was a very hot time for research repository stories. Sarah from GitLab wrote ‘Why we built a UX Research Insights repository’ and then Etienne from Uber wrote ‘The Power of Insights: A behind-the-scenes look at the new insights platform at Uber’, Kaleidoscope. Rosenfeld Media also held a video conference on the subject and Etienne shared that Kaleidoscope was built in people’s spare time over months. So these are the internal tools that organizations have invested in. After hearing about these platforms and also talking to our customers, we landed on the following definition of what a research repository really means. We think it’s a centralized, searchable database of research and insights that the entire company leverages to make better decisions. The whole point is to allow everybody to uncover user research insights and explore the context that led to them.
The question is why does everyone seem to want to set up a research repository and what exactly are the problems they are trying to solve? After talking to our customers and reading through the articles, I’ve summarized some of the problems to share with you.
Research is conducted differently across teams and departments, especially in large organizations. Different methodologies, tools, and formats for insights can cause silos. Here’s a quote from a customer interview we conducted last week:
We want to bring research through the entire business to all of the fragmented areas so that we can cross-pollinate our research. So, really about trying to break down those silos between teams.
The problem is how to help team members build a key insight that existing members have realized. The cost of relearning is expensive, especially in growing organizations.
Undocumented knowledge means you have to know who to talk to. When that person leaves the organization, they take their knowledge with them. We refer to this as tribal knowledge, and relying on it in any organization is dangerous. Another quote from a customer:
We want to stop relying on one person who has been here for seven years. People in your company are known as know it alls and often pestered for coffee.
Because everyone works differently and we’re in a gestation period for the discipline of user research, research reports aren’t standardized. We’ve seen Powerpoints, blog posts, Confluence pages, and more. Without a consistent format, it can be difficult to leverage the insights from the past. This is what Tomer Sharon was talking about – having consistency for the formats makes it easier to store, search, and leverage that past research.
Lastly, research data is spread across a variety of tools. Most people are familiar with Google Drive, Dropbox, Airtable, Word documents, and of course, sticky notes. While these tools are all great in their own right, they are limited in that you can’t really search them. When they pull that wall of sticky notes down, they are gone. Sticky notes also require people to actually be in the room, which is not so good for remote work. Another customer indicated:
We had developed a treasure-trove of research over the years and felt limited by traditional file-sharing platforms which only allowed us to put the research into the one place but not explore and digest the information in an intuitive and user-friendly way.
These appear to be the most common problems that motivate people to implement a research repository. We also found an article from Jakob Nielsen of Nielsen Norman Group from 2005, indicating that when people have to locate existing usability reports, they’ll fail, or they won’t know what to look for because there is no single place that lists usability results. Worse, if past project owners are the only source, you risk losing this when they leave the company or their roles are reassigned. While in this example, Jakob is specifically talking about usability reports, he predicted the whole trend 15 years ago, talking about the lack of a single place and the institutional memory when people leave the company or move projects.
Let’s say you’re sold and you want to set up a repository, and you feel your organization will benefit from it. Here are a few principles to keep in mind to guide the process.
We all know that researchers are busy and often outnumbered. The whole purpose of a repository is to help scale a research team to service the entire organization. This means that the repository has to support the ability to retrieve answers to questions in a self-service way. As a tangible example of that, something like search should be effortless, free of complicated queries, filters, or views.
We also know some stakeholders don’t feel comfortable interacting with research, especially if there is clunky software that requires training. We should also encourage everyone to make data-backed decisions. The head of research for one of our customers shared their Dovetail project with the board of their company, like a big public company, and had the board kind of playing around with it. She recounted that it was a huge win because the members of the board felt connected to their end-users.
Evidence helps to build trust in the research team. I’m sure you are familiar with bringing in a stakeholder—for example, a developer—into a usability testing session or interview and seeing their first-hand reaction to observing a user. Traceability between the insight and where it came from is very important and helps to keep biases in check by providing that extra context for people to explore the data on their own.
We know that researchers want to build participation to create empathy with end-users. It’s a very common thing that researchers like doing, but it’s counterproductive if you have the research data locked away somewhere. The idea of the repository is to create a level playing field regarding access to data. Tangible examples are features like open by default permissions and one-click sign-on for easy authentication.
Lastly, we live in the age of regulation like GDPR and the California Consumer Privacy Act. Your repository will store sensitive data, so it’s important to consider storage policies, deletion policies, features like encryption and anonymization, and further requirements this space.
There are a few other things to think about which might prompt you to ask further questions. Things like, how will you get buy-in from the org? The repository won’t be successful only one or two people using it. Who is going to have access? If you are going to allow everybody to have access, will they create insights or just be viewers? How will you migrate past research data? Will you pick a time and say that from here on out we’ll be using the repository? What kind of data will you store? So, for instance - interview transcripts, video recordings, usability testing notes, maybe survey responses and NPS? And who will manage the taxonomy? And guard it and make sure it doesn’t turn into the wild west? Who will standardize and be the guardian of the data?
Have you seen the adoption of repositories grow in the last 12 months? There is certainly a lot of talk about them, but are you seeing their adoption?
In response, Benjamin indicated that the folks at Dovetail have been inundated with people talking about finding ways to set up their repository, looking for tools to help them out, etc. Most organizations can’t afford to do what the big guys have done and build their own internal platform. They appeared to be either big teams working on the project or longer-term part-time projects that the researchers initiated and received developer assistance to help them actually build-out. Organizations trying to solve their research problems with tools makes a lot of sense. It seems to be a reaction to the general rise of research inside companies. As a designer, Benjamin highlighted the parallel to what happened to design when designers managed to get a seat at the table.
Another question that was asked was:
When beginning to set up a repository and using a tool like Dovetail, do you have a suggestion about whether you should bring in the past information? Like research that you already have? Or does it make more sense to start from scratch? Doing the former and bringing everything, your historical data in can feel overwhelming what are your thoughts on that?
There are so many different methodologies and formats of data including video, text, transcripts, notes, survey responses, both structured and unstructured. Importing it and transforming it into a format that makes sense for whatever tool you’re deciding to use is difficult. We see a lot of customers just draw a line in the sand and say we’re going to start from here. Or maybe they do a gradual rollout and use it for interviews but not for survey responses. When importing data, often unstructured qualitative data, you end up realizing just how much you actually have. The question is whether it’s really worth it, which is up to each organization to decide how valuable that stuff is.
Analyze data, collaborate on insights, and build your research repository.Try free for 7 days
Use casesUsability testing analysisCustomer interview analysisNet Promoter Score analysisSurvey response analysisSupport ticket analysisRemote user research
© Dovetail Research Pty. Ltd.
Made in Australia