How We Lost 47% of Our Users After a Redesign


by Andrew Burmistrov.

The Internet is full of success stories. To balance things out I’ll share the story of one of our colossal fails. Here’s how we redesigned one of our services and lost 47% of its users in the process.

How Things Were Before



We draw icons. At the very start of Icons8 it became obvious that we weren’t able to hack google AI to invent hundreds of icons every month, so we needed some help. Who better to turn to than the people that actually needed these icons- our customers! That’s how the Request Icon feature was born 3 years ago. For the last 3 years it looked like this:



The idea is simple: people request the icons they need and other people vote for them. We draw the most popular requests. Every day. And it’s working.



This demands some explanation.

Strange bumps





We went back and forth with adding/removing an Ideas&Votes link from our main menu. It never got great results except for short-term popularity bumps. Basically it meant that old clients who knew about this feature before were reminded about it and gave it a second go.

However, new people who never expected requested free icons ignored the menu. Note. We tried to rename – Ideas&Votes to “Request icons” with the same result. Second bump. Based on our usability research, people tended to ignore the feature based on a set of incorrect expectations. (“It is expensive”, “It’ll take a long time”, or “It’s some kind of scam”)

Steady growth



Well, that’s easy. Icons8 overall traffic showed steady growth during the same period of time which led to some proportional growth of the feature’s popularity.



The red dashed line is the actual traffic. We had some analytics tracking problems so half the traffic just vanished from the chart.

Rapid growth



Apparently it took us a few years to realize that the best time to fulfill a need is right as it arises. Just like people forgetting to buy milk while shopping, they forgot about the menu link when they needed an icon.

The decision was obvious. This one:

Lost your dignity? Ask us, we'll draw it.

Compared to:

Lost your dignity? Ask us, we'll draw it. Can also help you with pride.

To be honest, the graphic below is pretty much useless. I just think it’s pretty, so it’s staying in. The missing an icon? hint was launched on January 18, 2016; and provided us with 75 people on a daily basis using the “request icon” feature.

January 12 to 18 shows how devoted our programmers were in testing things.

Everything was great. Until we decided to make things even better.

Redesign



After the redesign the feature looks like this:



    The intended goals of the redesign were:
  • Added 2 new features – fast track orders and custom orders
  • Replaced an old design, making it simpler and more modern
  • Saved our designers time they wasted on administrative tasks


    What did we actually achieve?
  • 50% fewer people requested icons
  • 50% fewer people voted for requests
  • I was forced to conduct usability studies


Important: our overall traffic didn’t change. The same amount of people came to this feature’s page. We definitely improved the overall “look” of the feature but lost a few good bits on the way.

Note that descriptions are visible in the old design, while hidden in the new one. I’ll write more on this later.

50% fewer people requested icons



    People requesting icons need to know 2 things:

  1. Does this “request” thing really work?
  2. When will they get their icon?


The thing is, the old design answered the first question and the new design answered the second one. Seems like if you don’t want a ~50% drop, answering the first one should be the top priority.

This is one of not many things that really add something to the feature after redesign.

To answer Does this “request” thing really work? we use Votes and Comments.

Votes



Voting means that it has a life of its own. People vote, there is a queue, and it’s moving. So the feature IS working.



That’s the critical difference between the two designs. In the old design I immediately understood that the number meant votes.

    Old design:
  1. Number of votes
  2. Text hint “votes” below the number
  3. Both number and text are enclosed in a square, forming a symbolic representation of an empty icon. I have the feeling that someday there will be an icon instead of digits and “votes.” It’s a very smooth, tiny detail, but has such an impact on the overall impression.


    New design:
  1. Some number
  2. Some text


Voting arrows appear only on mouseover. A big NO-NO.

What is this digit supposed to mean? Position? ID? Days left? Without knowing what the digits mean we can’t be sure if the request service is working.

Comments



Another sign of life is the comments. We’ve used them to comment about the status of an icon, but people were actually commenting on requests, flattering and bribing each other. The place used to be full of energy and life… until all of a sudden everything was gone. The new design had no comments- our users were speechless, just like me. 50% fewer people voted for requests

There is only one big question here, What am I voting for?

In the old design, the average number of votes an icon needed to get in order to be drawn by us was ~30. With the new design that number was ~15-16. Actually, last time I checked, it was 9. But 70% would be way too dramatic a number for the headline of this article. Nine votes may seem good for companies with 9 employees, but it could lead to a large number of custom icons being drawn for small, niche projects, not community approved ones. Suddenly all 9-person companies hate me now. The decrease in average votes is, partially, due to the “voting” problem mentioned above. People won’t vote if they can’t understand the system. Another reason is descriptions.

Descriptions



Within the old interface you were able to see the description of every request on the page you landed on by scrolling down, choosing what you liked, then voting for it. The new interface forced users to click every request in order to see its description. Every time. The initial motivation to scroll down through many requests, having some fun along the way reading other people’s descriptions and requests, and voting for the one’s we liked, was killed. Thus our voting numbers decreased by ~50%.

Solution



Quite easy. We need to bring back the “queue” effect.

To be honest, the new design has a Recently Created Icons section, an indication that the feature is live. However, you need to scroll down in order to find it.



People have no motivation to scroll. If they don’t understand what’s happening on top they have no idea what to look for below.

    For starters, here’s a short to-do list:
  • Properly indicate votes
  • Unhide the descriptions of requests
  • Making it all visible as soon as people landed on the main page.


By trying to make the service simple and minimalistic we complicated user’s experiences. Getting rid of a few tiny details dissolved the whole sense of a queue and availability was lost, leading to a bunch of unanswered questions and reduced usage. But it’s not the end! We gained a great experience and are determined to make things better now, and more important, we know how.

I hope you enjoyed our own private fail and will have better luck with your own good intentions on your projects.

About the Author



Andrew Burmistrov is a usability specialist of Icons8. He started his career as a phone support specialist, telling jokes while customers were rebooting their computers, then moved to usability testing and occasional writing.