What if your next ‘aha’ moment was sitting idly in the Google Play Store?
Well, it just might be. Experts in human-computer interaction agree that product reviews contain usability insights. For this reason, researchers continue to tinker with a diverse set of methods for gathering, analyzing, and visualizing user feedback.
Here’s the core idea.
We can collect user reviews from platforms like Amazon. We then group together similar reviews so we can extract themes about a product’s learnability, efficiency, and novelty. The result? Thorough design concepts that systematically check off user needs.
In this piece, we’ll see how user reviews are like miracle grow for design ideas — nurturing the creation of rich and comprehensive prototypes. But first, let’s discuss the role of feedback during the early stages of product development.
Let’s jump in.
Why Analyze User Reviews?
Reviews are a gold mine or more specifically a feature mine.
They contain statements typically centered on what it’s like to use different aspects of a product. In fact, one influential study revealed that between 13-49% of reviews feature usability concepts like ‘efficiency.’
Let me explain.
Broadly speaking, usability is a pragmatic notion concerned with how easy a product or service is to use. We can deconstruct usability into its constituent parts: memorability, learnability, efficiency, errors, and satisfaction. For instance, a review would express the concept of learnability if it said, “I was able to quickly get the hang of this tool.”
If we can systematically identify concepts like this in reviews then we can streamline product development. More specifically, we can add depth and reliability to the fuzzy front-end.
The fuzzy front-end is the messy process of figuring out what to create in the earliest stage of building a new feature, product, or version of an existing product. It’s tempting to hastily complete this stage to start making costly prototypes. Indeed, most teams spend less than 2 weeks on this crucial stage.
But that’s a mistake.
Instead, it’s imperative to diligently apply discovery methods to determine what problem your design will solve. The Nielsen Norman Group has found that projects which use discovery methods have a 58% greater chance of success than ones that skip this stage. For this reason, these methods are arguably the most powerful tools for de-risking a project.
A standard discovery involves conducting interviews, field studies, and context mapping to grok users’ needs. It’s an involved yet worthwhile process. We can add user review analysis to it — as a compliment — for a low-cost and accessible method.
In the diagram above, you can see how user reviews serve as fodder for the fuzzy front-end. Once a team has developed their product, put it on the market, and received feedback, they can leverage user reviews to inspire the second version’s concepts.
The accessibility of this approach, however, is limited because the vast majority of teams have little if any reviews at their disposal. They don’t have a product out yet. In this situation, teams can analyze reviews for competitors’ products to identify, and hopefully fill, gaps in the market with their designs.
In sum, whether a team is yet to launch or has hundreds of ratings, user reviews can facilitate ideation during the fuzzy front-end of product development. If you’re skeptical — I get it. User reviews have many imperfections as a data source and must be interpreted with caution.
Reasons To Not Analyze User Reviews
The greatest strength of reviews is their greatest weakness. They’re a data source that stems from natural feedback rather than a contrived laboratory setting, so their insights may be more applicable to the real world. That said, researchers’ lack of control in producing the data presents a host of problems.
- Sometimes Faked: Companies are highly dependent on user reviews for their success. This, unsurprisingly, creates perverse incentives. Some unscrupulous organizations will pay people to leave fake reviews, and there’s no easy way of telling what feedback is honest.
- Sampling Bias: Research indicates that users who have the most extreme feelings, either positive or negative, tend to leave reviews. A similar dynamic plays out on the self-descriptive service RateMyProfessor; teacher reviews select for students who either did exceedingly well or poorly in the class.
- Lack of Demographics: Even if the reviews were representative of how users felt, we’d still lack crucial information about the authors — namely, their demographics.
In other words, user reviews can be fake, biased, and lack crucial information about the author. However, suppose we can obtain a set of high-quality reviews. The challenges are far from over: we still have to wade through an ocean of words to answer our research questions.
Imagine we have 1000 reviews, and each contains 50 words. In total, that gives us 50,000 words to analyze, roughly the length of a short novel. Just to read through the reviews would take hours, let alone having to annotate them and identify major themes
Indeed, analyzing reviews is labor-intensive. But it can provide unique value above and beyond standard methods, as we’ll see in a study drawing design inspiration from feedback on Amazon.
How To Apply Affinity Diagramming to User Reviews
They call her the context queen…
Her name is Froukje Sleeswijk Visser, a design researcher at the Delft University of Technology. She specializes in context mapping: a qualitative method in which users work together with researchers to describe their everyday life in a visual format (e.g., collage). This technique is a potent empathy enhancer. It enables researchers to grasp the context in which people use products; that is, where, when, and under what circumstances.
As an added tool for understanding users’ needs she has proposed analyzing reviews.
To test this technique, Visser and colleagues collected 2,485 Amazon reviews for 7 office bags. Their next step must have been tedious. A group of design students broke the reviews into sentences and wrote them down on cards for use in affinity diagramming. The idea behind the technique is simple: take snippets of text, group related ones, and identify major themes (see below). Bonus points if you color code, and collaborate with teammates.
On top of affinity diagramming, the designers also ran contextual interviews. The aim was to represent the standard way of approaching the fuzzy front-end. The researchers converted the interviews’ results and the review analysis into insight cards that state a theme at the top and an example quote below (see below).
The authors then ran two different workshops: one in which designers developed the concept for a purse using only insight cards from the interview, and another that used results from interviews AND the review analysis.
Drum roll please….
Designers who used both the review and interview data sketched more detailed design concepts. They even went so far as to consider the color of zippers. The team using both data sources was more detailed, in part, because they used the review insights as a checklist to ensure their design satisfied all user’s needs (e.g., material functionality).
The authors argue that the data from reviews and interviews, respectively, each provided unique value. The reviews revealed nuances about specific product features, and the interviews shed light on the holistic context (e.g., office environment) in which users would bring their purse — much like context mapping.
Based on this analysis, the researchers note the promise of user review analysis. The technique complements user interviews as a concrete source of feature requirements for a new product or version of an existing one.
That said, the task of analyzing both data sources may seem herculean. Indeed, the authors lamented the tedious nature of manually clustering thousands of reviews. That brings me to an important problem. What if your product has no reviews because it is yet to release and the number of reviews your competitors have is massive? You could never analyze all by hand.
One strategy would be to diagram a random sample of at least a few hundred reviews. That way, you can ensure there’s no systematic bias in your selection of feedback like there probably would be if you only handpicked recent or interesting reviews.
Alternatively, you could take a computational approach. Using data science techniques, it’s possible to extract insight from virtually any amount of feedback — the topic of our next piece.
References
Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. (2018). Nielsen Norman Group. https://www.nngroup.com/articles/affinity-diagram/
Hedegaard, S., & Simonsen, J. G. (2013, April). Extracting usability and user experience information from online user reviews. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2089-2098). ACM.
Lee, B. Y., Saakes, D. P., & Sleeswijk Visser, F. (2015, November). Online user reviews as a design resource. In 2015 IASDR Interplay. Queensland University of Technology, Australia.
Visser, F. S., Stappers, P. J., Van der Lugt, R., & Sanders, E. B. (2005). Contextmapping: experiences from practice. CoDesign, 1(2), 119-149.