A/B Testing: Usability From A to B

A/B Testing: Usability From A to B

Usability is an often discussed part of design, but it seems to be very poorly understood. While most people understand the concept of usability, many people don’t know how to use it in their workflow, how to get started with their testing, or fail to see the benefits it has for them.

A/B Testing: Usability From A to B

In the past usability testing meant huge labs, huge budgets and long timeframes. With the rise of the internet has come the rise of a number of remote usability testing tools that meet a wide variety of needs and a wide range of budgets that have made usability testing far more accessible to the point today where even the smallest design studio and freelancers can afford to usability test.

The point is now that you almost can’t afford not to usability test. Clients design better results, designers and developers have more exacting standards, and users will keep looking until they find a design that just works – you just need to look at the rise of Apple, the iPod, iPhone and iPad to see examples of this.

“Coding is long. Design is short. Paper is cheap.” – Anonymous

So, what exactly is usability testing, how does it work, and how can you get it to work for you? Read on to find out:

A/B Testing

What is Usability Testing?

According to Wikipedia, usability testing is a technique used to evaluate a product by testing it on users. Usability testing focuses on measuring a product’s (or in our case website’s) ability to meet its intended purpose.

This concept seems simple enough, and most of us probably do try to get some input from users on our site designs. The issue is how to achieve this in a quick, simple fashion that is easy to distribute, makes it easy to capture results, is low cost and time efficient.

As previously stated, with the rise of the internet has come a range of remote usability testing tools. Each kind has its own pros and cons, and can work better in some situations than others.

Some of the more popular types are:

Moderated Testing Tools

With moderated testing tools you are able to see a live view of the user’s screen (and possibly the user), and to converse with users during the testing process. Some services will arrange testers for you, and some you need to arrange yourself.

Pros: A lot of information can be gathered, real time interaction with users
Cons: High time commitment during the testing process from both testers and users; higher costs; more time required for data analysis, long set up time.

Self-Moderated Tools

Uses the user’s microphone and camera to capture data for you, but you can’t watch live. Generally these kinds of services will arrange testers for you.

Pros: A lot of information can be gathered; can be run at any time.
Cons: High time commitment from users during the test, and for data analysis afterwards; long set up process; higher costs.

Automated Tools

There are a wide range of these kinds of tools which record a user’s interactions with a webpage, graphic, or design mockup. They achieve this in different ways, but heatmaps of where users have clicked are the most popular method.

Pros: Lower cost (depending on service); lower time commitment; easier set up; easier to distribute; can be run at any time; less intrusive.
Cons: Data gathered is limited to pre-defined tasks; can’t observe the users as they interact with the site.

Within these groups are a range of different tools, again with their own strengths and weaknesses. You need to look at your project, budget, timeframe and requirements in order to decide what kind of tool will work best for you.

Today I want to concentrate on talking about automated tools, and a rising feature among them: A/B testing.

What is A/B Testing?

Imagine a tennis tournament. You start off with a number of players, they play through rounds, and each round the best ones should bubble to the top. In the end the two best players will face off, and which ever one wins should be the best. This is A/B testing in action, this is how it works. Eventually the best designs for your goal should bubble to the top, and whether you are comparing two designs or ten designs, you should get useful information that shows you what works best.

Up until now most A/B testing has been directed at live sites, and mainly for testing conversions, but A/B testing for usability is also an emerging trend that can do wonders for you and your designs.

From the beginning of the design process, and the original thumbnail sketches that we draw, to wireframes as the process develops, and on to the final designs as we get further down the track, A/B testing for usability on these designs can really help show us what’s what.

What is A/B Testing?

How Does It Work?

Regular heatmap testing helps you track where people clicked, and how long it took them to fulfill a task. This works great when you have a baseline and know what you are working towards. You can simply tweak things to improve on the current design.

With A/B testing for usability the same principles as regular A/B testing applies. You create two versions of a design you’d like to test and traffic is randomly directed to each version that you are testing. Roughly 50% of traffic should end up going to each version, and you end up with some pretty convincing data for comparing the two.

Of course, I could talk about this till the cows come home, but the easiest way is to show you the concept in action. We’ve recently been doing a redesign of our homepage, and I’ve set up a quick A/B test of the old design versus the new wireframe we are working on.

“It is far better to adapt the technology to the user than to force the user to adapt to the technology.” – Larry Marine

The Testing Process

Here is the test I’ve set up, it’s just 8 questions, and should take you 1-2 minutes to complete.

Alright, taken the the test? That was actually an A/B test – each person who takes it will be sent randomly to either version A or version B. That way we can see the results side by side when deciding on what changes to make, and we should have a fair spread of feedback from users.

For interests sake, you can see the two designs we are comparing below:

Current Site

Current Site

Wireframe of New Design

Wireframe of New Design

The images above are the current design of our site, and the new wireframe we are working on, trying to give the site a new feel and improved navigation.

So far we’ve just had a few results come through from this test, but already we can see that by having a navigation bar at the top of the page it takes users far less time to find the information they are looking for.

If you are interested, you can go ahead and check out the results yourself – you can find them here. The results are fairly self-explanatory, but there are two important things you should notice:

  • Where the users are clicking
  • How long it takes them to click

A/B Results

See the heatmap over the image, and average click time at the top.

If everyone is clicking the wrong place, you know you have issues. If everyone is taking a long time to click, you know you have issues (well, depending on the task you are trying to achieve). If both of these things happen, you know you’ve really got to work on your site, and if both are successful then you know you on the right track. Fairly simple really – hopefully you can see the value of that.

Even with some of the simple tasks on our current designs, users tend to get a bit confused because there is a lot of information. This can lead to a less satisfying experience when using the site. The average time for completing the tasks on the current site is also significantly longer than the new wireframe, again showing that there is more information overload, and less optimized navigation on the current design.

The more optimized the design, the more people enjoy it, the more conversions you will get – be that readers on your site, sales of a product, whatever goal you are aiming at, you are more likely to reach.

How Can I Use Usability?

This is just one example of what you can do with usability testing, but there are many other situations where it could come in handy. Some examples are:

  • Redesigning a site: run the old site against the new site
  • Testing alternative designs
  • Testing alternative navigation layouts and wording
  • Testing design sketches (you can just upload scans of them and see what works)

Everyday I see people trying different things and testing different concepts. You are really only limited by your imagination with what you can test.

The beauty of remote usability testing is that it is easy to set up, easy to distribute, takes very little time to run, and works at whatever stage you are in of your design development. Set up takes just a few minutes, and then you can send it out far and wide – on your Facebook page, Twitter feed, via email or any other fashion you like.

I suggest that today you could run a test on your own site: upload a screenshot, set some simple tasks and see what is working for you. This way you will soon see the benefit of usability testing, and will be better able to explain it to your clients in the future.

Further Resources:

People to Follow:

UXBooth@UXBooth is the official feed of UXBooth.com, a popular blog with great resources and articles on UX and usability. They share all sorts of useful related links.
Bolt Peters@BoltPeters is a remote user research firm. They produce lots of great content and share all sorts of interesting information on their twitter feed. You can find their homepage at BoltPeters.com
UXMatters@UXMatters from UXMatters.com, another very useful site, although with more of a focus on user experience than usability. Still a good array of articles to choose from.
Paulolyslager@PaulOlyslager is a web designer and usability consultant. He posts all sorts of useful usability related information. You can find his site at PaulOlyslager.com
UXMag@UXMag is a great UX and usability blog, and they share lots of useful links (and a few giveaways) in their twitter feed. Their main site is at UXMag.com
IntuitionHQ@IntuitionHQ is my feed. I spend a lot of time stumbling across useful articles and information on usability, and share it all here. Hopefully something useful for everyone


“Build it and they will come. Build it right, and they will come back.” via @DavidBardwell on Twitter

The main point you should take away from this article is the value that usability testing can give you in helping you optimize your sites, helping you deal with difficult clients, helping you create more usable designs, and ultimately through all of these points giving you much more satisfied clients at the end of your design process.

It doesn’t have to take you a long time, and it doesn’t have to cost you a lot of money – in fact in many situations you will earn more by having an added service to sell to clients, and by providing a higher quality service.

The better, more usable your designs, the more people will like you and your sites, and the better your business will be. Simple really.
So go out, get started, and run some tests today. You never know just what you might find. Good luck!

Any questions about usability testing or how to get started? Be sure to let us know in the comments.

Jacob Creech is a usability geek at IntuitionHQ.com - a new, quick, easy to use usability service. When he’s not indulging his passion for design, he is probably studying or blogging about Chinese (having spent three years in China) at sinomatter.com.


    • Ryan,
    • October 18, 2010
    / Reply

    Great post! A/B Testing can be incredibly effective, problem is it can also be a ginormous waste of time. Fantastic explanation of the pros and cons. Thanks!

    • Clipping,
    • October 19, 2010
    / Reply

    Awesome! excellent work & thanks a lot for sharing..

  1. / Reply

    Yes nice post and a good share. Anything to assist in the creation and development of a website is a bonus, as you say issues that some of the most awkward clients will have – can be identify early by doing A/B testing.
    I have worked in the past at companies whom wrote there own software and then internally tested it – believe you me – the amount of ver 1.02, ver 1.03 and bug fixes that were discovered before releasing products (and after to – lol) was unbelievable! So what ever the web site or software – A/B testing is your lifeline and final line of defense!

  2. / Reply

    Great summary and a must-read for all usability fanatics! Thanks for mentioning me!

    • Ahmad,
    • October 20, 2010
    / Reply

    Very informative post thanks alot for the tips

    • badmash,
    • October 23, 2010
    / Reply

    I just signed up to your blogs rss feed. Will you post more on this subject?

    • Vitaly,
    • September 27, 2011
    / Reply

    http://headmetrics.com/ – cheap and easy a/b testing tool

Leave a Reply

Your email address will not be published. Required fields are marked *