Outcomes vs. outputs

This blog post is currently under construction.

Follow, like or subscribe to find out when it’s available!

twitter-2012-negative-logo-5C6C1F1521-seeklogo.com facebook-flat-vector-logo-400x400 email
Follow me on Twitter Like my Facebook page Get email updates

Don’t ask your users what they want – a guide to User Testing

The whole user centred design philosophy is based around involving users in the design process. So why do some of the ‘brightest minds’ say we shouldn’t ask our users what they want? Steve Jobs once said;

Why do we want to ask what our audience thinks? We don’t care what they think. How can people tell you what they want if they haven’t seen it before? If we ask them what they want, we’ll end up doing Swan Lake every year!”

Note: there is a lot of debate about this quote being true or not. There is a lot of evidence that supports both sides of this argument.

And Henry Ford also said;

If I had asked people what they wanted, they would have said faster horses.

Note: there is a lot of debate about this quote being true or not. There is a lot of evidence that supports both sides of this argument.

Steve and Henry are right, we shouldn’t ask people what they want because they simply can’t tell you.

Asking this can actually do harm. People’s responses are only guesses of what they think they want. Also, their thoughts are shaped around what they currently know, which is why people in the 1800’s would ask for a faster horse.

Instead, we need to show people what we think they want. Doing this allows them to react to real experiences, which gives us information that improves our designs.

Here’s an example;

Imagine that it’s 2004. You’ve never heard of the iPhone  before, let alone a smartphone. You’re called into a meeting at Apple and Steve Jobs puts the first iteration of the iPhone in your hands. Can you imagine what your reaction would be? You’re used to using a Nokia 2280 and now you’re seeing a product that will change the world.

Steve gives you a task – “Imagine you want to find the score for the latest Lakers game. How would you do that?”

You see an icon called internet. You tap on it. A web browser opens. You type in nba.com and see that the Lakers beat the Celtics 110-98.

This is exactly the type of process that could, or should, have been used to create the iPhone. Steve and his team of designers would have watched your every action and reaction as you completed your task. This information would have been used to refine their designs and create the iPhone we know today.

Steve and his team of designers could have used this information to make improvements to their design. For example, it may have taken you minutes to find the Laker’s score through the web browser. Steve could have thought, ‘there’s got to be an easier way!’ So they decide to test an easier idea with the next person. This could have been the way they created apps or push notifications.

In order to get this type of information you need to facilitate user testing sessions with the right way.

User Testing

User Testing is the moment of truth for your designs. Here you will discover how they perform in a realistic scenario. You’ll learn valuable lessons about how your users interact with your product or service and how you can improve it.

I’ve never had less than 20 key findings the first time I’ve tested a design, so prepare yourself for feedback.

User Testing allows you to evaluate the two things that create a great experience; how much value your product or service creates and how easy your product or service is to use.

This is the moment of truth for your designs. Here you will discover how they perform in a realistic scenario. You’ll recruit participants who are, or could be, the people who actually use your product or service. Then you’ll give your participants tasks that represent how your product or service will actually be used. You’ll be particularly interested in what people think about each part of your designs and why. This information is vital to make improvements.

For example, imagine your user testing one of the first versions of the iPhone. You notice that the participants arrange specific sets of apps on each screen. You ask, “Why have you arranged those apps in this way?” The participant says, “Because they’re all similar types of apps; I have my games on this screen, my productivity apps on the next screen, and all my social media ones on the last screen.”

You can now use this information to create an ‘app grouping’ feature for arranging apps into folders. If you didn’t ask why you may not have discovered that this feature was needed.

Who to test with

The most important part of testing is finding who to test with. You can easily get misleading data by testing with the wrong people. To avoid this, only talk to people who match your personas.

Personas are used to represent different groups of people who use your product or service. They aren’t separated by things like age, gender, or geographical location. Useful personas are always separated by behaviours.

For example, a group of people who exercise three times a week and eat low-carb diets could be considered a persona, even if they were different ages and genders.

This contrasts with the way marketing teams define customer segments. These are based on factors like age, gender, location, and income. Customer segments often find their way into design. However, it doesn’t matter what age you are, you can still be passionate about exercising three times a week and eating a low-carb diet.

Find where your personas hang out and ask them to talk to you about your product. This is called  ‘guerrilla’ testing. Guerrilla testing is when you don’t formally arrange any participants. Instead, you go to where people are and ask them for 5-10 minutes of their time. This can be a bit awkward, but it helps to be clear that you’re not trying to sell them anything. Instead, you’re offering them an incentive, like a chocolate bar or bottled drink, for their time.

Aim to talk to five participants from each persona group. The Nielsen Norman Group conducted a comprehensive study and found that “elaborate usability tests are a waste of resources” and “the best results come from testing no more than five users.”

nn-5 users

As the number of users tested with increases, the percentage of usability findings discovered decreases.


There are two roles for your team in a user testing session;


The facilitator is the person who gives the participant tasks and asks them questions. It’s the facilitator’s job is to uncover exactly what the participant is thinking while making the test feel realistic.

Note taker

The note taker is someone who records precise notes of what’s happening during the testing session. This role is very important – if you don’t document your findings well, you won’t remember what happened and why. This information is vital when iterating your designs.

It’s important for other members of the team to observe the testing sessions. This helps them understand what works and what doesn’t, which is more effective than being told what happened.

Team members can observe the sessions by watching video recordings of them. If what you’re testing is digital, then there are multiple apps that record all the interactions on an Android, iPhone, or laptop screen. An alternative is to use a smartphone, GoPro, or other type of camera to film the session.

If you want to live stream your session then use Google Hangouts, X-Split, Skype or something similar. However, this may be expensive when there’s no WiFi.

Get the team to view the sessions together. While watching they should write down their own findings on post-it notes:

  • Green for positive
  • Pink for negative,
  • Blue for observations.

Remember to number each finding so it’s easy to identify personas and trace findings back to the participant they came from.

Write findings with a thick-tipped sharpie so it’s easy to see. It also helps keep findings concise.

As you write these findings stick them on a wall – you’ll begin to see that there are similar findings from different participants. It’s a good idea to cluster these findings together. You’ll end up with something like this:


How to run a test

Begin with an introduction. Ask the participant a few questions about themselves in order to make sure they’re the right persona.


Next, remind them about the principles of user testing.


Start the session by setting the scenario. Give the participant a realistic story so they can imagine they’re doing this for real. For example, you could say:


They’re now ready for their first task. This could be:


Each task should cover an area of your product. Keep going until you’ve covered everything you want to test.

Finish up by asking closing questions that aim to discover what the participant thought of the product as a whole, for example:

  • “What was one thing you liked the most?”
  • “What was one thing that you like the least?”
  • “How would you describe this product to a friend?”

Get the most out of your participants

Throughout the session, prompt your participant to give you feedback by doing the following.

Continuously probe

Regularly ask:

  • “What are you thinking now?”
  • “What would you do now?”
  • “What would you do if you were doing this [in the situation you’d normally do this. For example, at home]?”

The echo

If the participant says something like:

“Oh, that’s weird.”

The facilitator should say:

“What’s weird?”

This helps you clearly figure out what the participant is talking about without making assumptions.

The boomerang

Participants will ask the facilitator questions or state something to themselves. It’s good to understand what the participant is thinking or would do if the facilitator wasn’t present. This is where you throw the question or statement back to them like a boomerang.

“Do I need to register?”

“What do you think?”

Ask why

If participants say or do something, ask why. It will help you understand exactly what’s going on which you can use to make design decisions. It’s also good to repeat the participant’s language to remove the possibility of biasing a participant’s response.

“That’s cool”

” Why is that cool?”

After talking to all your participants, analysing the sessions with the group and clustering post-it note findings, you should be ready to iterate your designs.

After testing

Once you’ve tested with at least five participants from each of your persona groups you will be ready to make design changes to your product or service.

After making changes be sure to User Test again. This is to validate that your changes work and to discover more things you can improve.

Keep iterating and testing your designs until your participants begin to ask questions like:

  • “When is this going to be released?”
  • “How much will it cost?”
  • “Can I have it now?”

When you reach this stage, you’ll know your designs are working well.

Show people, don’t ask them

In conclusion, Steve Jobs and Henry Ford were right when they said we shouldn’t ask people what they want.

We should instead show them through User Testing while asking vital questions, which will help us make improvements. We need to continue to do this until people are begging us to release our product or service.

< Previous: How to create value


Creating value

To begin with, products and service create value in two ways:

  1. What problems it solves (I call these things pains)
  2. What existing activities it improves (I call these things gains)

Value occurs when you relieve a ‘pain’ or create a ‘gain’.

Value defined

Value is at the core of every experience. It’s the reason why we use one product or service over another.

It’s a simple equation. If something provides more value to you than something else, you’ll use it.

However, if this is true, wouldn’t everyone have an iPhone? No, because value is in the eye of the beholder. The way we perceive value is based on our view of the world, which differs between each person.

Value is all encompassing: price, durability, availability, and support  all equate to a product or service’s value.

For example, if the price of iPhones rose to $2,000 then this would be too expensive for a lot of people. The price would become a pain of the product.

Apple could relieve this pain by dropping the price down to $1,000.

If they did this, value would increase and people would begin to purchase iPhones again.


Value example-01

How to create value

The Value Creation Canvas is an adaptation of the Value Proposition Canvas. It makes explicit how you create value for your customers.

Value creation canvas-01

The Value Creation canvas has three steps;

  1. Identify the user’s pains and gains
  2. Creating the businesses pain relievers and gain creators
  3. Creating value by matching pains with pain relievers and gains with gain creators

1. Identifying user’s pains and gains

It’s easy to start filling out the canvas based on what pains and gains you assume your users have. However, this is not the way to create value.

The best products in the world don’t base decisions on assumptions – they base them on facts. If you do decide to base decisions on assumptions, you will struggle to get users and in a few months time one of two things will happen;

  1. You’ll realise you’ve gone down the wrong path and decide to start again by basing your user’s pains and gains on facts
  2. You’ll become disheartened and give up altogether

So make it easy for yourself – start your Value Creation Canvas by basing your user’s pains and gains on facts and not assumptions.

To do this, you need to actually interview your users. Ask them about your product or service, your competitors products or services, and any workarounds they may use. Taking a board approach like this allows you to identify all the opportunities where you can create value.

You can follow this blog post to find out how to interview users and how many you should talk to.


Once you’ve interviewed enough users, pull out common pains and gains and write them onto post-it notes. One pain or gain per post-it.

Stick these post-it’s under the appropriate area of the Value Creation Canvas.

For example, image you run a gym and want to create more value for your customers. The image below is what your Value Creation Canvas may look like after interviewing users about their pains and gains.

Value creation canvas - users done-01.png

2. Pain relievers and gain creators

Get your product team together.

Allow everyone to look over the user’s pains and gains then individually post-it note pain relievers and gain creators.

Pain relievers

  • Are how you will solve a customer’s pain

Gain creators

  • Are how you will make an existing activity better for users

This is what the businesses Pain relievers and Gain creators may look like for our previous gym example.

Value creation canvas - business-01

It works best if your team individually post-it notes their own answers. When sticking them on the canvas, you can then group similar responses. This is a design thinking technique that allows everyone to have a fair say – you can learn more about it in this blog about co-design.

3. Creating value

Now you’re ready to identify the value. Begin by trying to match up pains from the users side with pain relievers on the business side. Do this by drawing a line from one side of the canvas to the other. It’s ok to have one pain reliever solve multiple pains.

Value creation canvas - linked 1-01.png

Continue to do this for as many post-its as possible.

Once you’ve finished with the pains, match the gains with the gain creators.

Value creation canvas - linked 2-01.png

The matched up pains and gains represent the value you provide to your users.

To make the value more digestible, put the matched up items into a sentence. For example, a user’s pain could be that they are too shy to workout at the gym in front of the opposite sex. A pain reliever could be ‘Create gender specific workout rooms at the gym’.

This communicated into a sentence could look like this;

We provide gender specific areas of our gym to make everyone feel comfortable while working out.

This makes it easier for you to communicate your value.

Value creation canvas - value 2-01.png

Testing your value proposition

At this stage, if you’ve actually talked to users, you will have validated what your users pains and gains are.

However, you haven’t validated if your pain relievers and gain creators will create value, which is why you now need to test your value proposition.

Do this is by creating a Value Proposition Page.

valueprop page example 1.png

Google’s Nexus 6p value proposition page

A value proposition page isn’t like a landing page: your main objective is not to get sign ups, it is to test if your value proposition appeals to your users.

You can create your value proposition page by identifying your top 5 -10 value statements and placing them in order from most to least valuable on your page.

Don’t worry too much about the design of your value proposition page. The easiest thing to do is find one you like and copy it. Apple and Google have great examples.

Below is how our gym example’s value proposition page could look.


Consider using tools like Axure and InVision to create this page. Actually coding it will take too long because you constantly want to iterate and make changes. If you want to learn more about easily creating prototypes to test, then read my blog on prototyping.

Creating and testing a value proposition page allows you validate your value proposition without having to create anything. It will help you identify what’s important from your users point of view, which helps you prioritize your effort when creating your product or service.

Once you have tested it with users, you can iterate and make changes to it. If you continue to test and iterate this page it will only be a matter of time before you have fully validated your product or service’s value.

This is only the first step…

Value is only the first step of creating a great experience.

The second step is turning your value into something real and making sure people can use it.

For example, imagine if someone recently created a teleportation device. This product instantly has an appealing value proposition – you can travel anywhere in the world in a matter of seconds.

Now imagine the first time you use this teleportation device, you end up at the wrong destination! You put this error down to a simple mistake and decide to try it again. Once again, you end up in the wrong place! No matter how hard you try, you never end up where you intended.

No matter how appealing a value proposition is, it can mean nothing if it isn’t usable.

Next: How to make your experience usable >