Archive for the ‘user experience’ Category

Undercover User Experience

Monday, March 1st, 2010

At last, the big announcement. I’m delighted to confirm that Undercover User Experience, written by myself and fellow Clearleftie James Box, will be published by New Riders this autumn.

Once you catch the user experience bug, the world changes. Doors open the wrong way, websites don’t work, and companies don’t seem to care. Fortunately, anyone can learn the UX remedies – usability testing, personas, prototyping and so on – but, unless your organization ‘gets it’, putting them into practice is trickier.

Undercover User Experience will show you how to do great UX work with tiny budgets, no time, and even without official clearance.

The idea came about in a Utrecht hotel, where James and I got talking about the early stages of our careers, when we didn’t have the luxury of doings things ‘by the book’. Through the IA Institute mentoring scheme I’ve met several people in the same situation. For them, what makes UX work difficult isn’t lack of skill, but not knowing how to make headway in companies that don’t appreciate the need. Pioneering UX and inspiring colleagues who’ve never cared about design takes improvisation, persistence and diplomacy. So we’ll cover guerrilla approaches to the UX techniques we know and love, along with frank advice on how to make them most of them in your business.

On a personal note, I’m thrilled to be partnering with New Riders. They were our first choice publisher due to their outstanding UX portfolio, including the classics Don’t Make Me Think!, Designing for Interaction and Elements of User Experience.

The writing experience is already demanding and rewarding. There’s been much to-ing and fro-ing over titles and much confusion over the US tax system and self-assessment, but we’re well under way and hoping to wrap the writing up by June.

But enough – I’ve no wish to turn this blog into a marketing vehicle. If you want to keep up to date with our progress and be the first to hear when the book’s due out, follow UndercoverUX on Twitter or visit the Undercover User Experience website and sign up for updates.

Posted in book, user experience | No Comments »

Oxymoron

Tuesday, November 24th, 2009

This is the ELSE Mobile. It’s a touch screen phone. They’re all the rage, I hear.

I’ve not used the ELSE Mobile, but I know from their website that I needn’t bother. I know because they claim this handset demonstrates a:

“user-experience-centric philosophy designed to enhance man-machine capabilities through pre-integration services.”

With this lone sentence, ELSE instantly destroy any pretence of user-centred design. No user-centred company would let their copywriters produce such unmitigated nonsense. I barely need to mention the splash screen, the breaking of the Back button, the grammatical errors (“Most device are…”) and the autoplaying music on the Flash monstrosity they call a website.

This, dear reader, is the opposite of user experience design.

[Thanks to Lewis for the link.]

Posted in mobile, user experience | 8 Comments »

Latest Clearleftie happenings

Tuesday, November 24th, 2009

Lots to report, much of which I neglected to mention thanks to my brush with our porcine friends.

UX London 2010

First, we’ve announced our programme for UX London 2010, which features (amongst others) Jesse James Garrett, Scott McCloud, Whitney Hess and Bill Moggridge.

Once again we were delighted with how much of our Christmas wishlist came true. The north wall of the office has been awash with post-its of names and topics for several weeks now, and there’s a certain Machiavellian joy in seeing it come together into a coherent programme. I’m particularly happy to see some names I pressed especially strongly for.

Bands always say their difficult second album will surpass their first, but I think it’s true this time. Not a prog-rock bass solo in sight. It’s happening 19–21 May 2010, and tickets are on sale on 1 December.

Spring internship

We’re taking on a User Experience intern early next year. It’s the first time this is a dedicated UX position – our previous interns have come from across the whole web spectrum. It’s a paid position lasting ten weeks, and would suit anyone with a talent and love for good user experience design. More details are on the Clearleft site – drop us a line or talk to me if you’re interested.

The book

Finally, my big news is that I’m writing a book with James. Several daunting but hopefully inspiring months lie ahead. More details will follow when we confirm them.

Posted in conferences, personal, user experience | No Comments »

Statistical significance & other A/B test pitfalls

Monday, November 16th, 2009

Last week I tossed a coin a hundred times. 49 heads. Then I changed into a red t-shirt and tossed the same coin another hundred times. 51 heads. From this, I conclude that wearing a red shirt gives a 4.1% increase in conversion in throwing heads.

A ridiculous experiment (yes, I really did it) with a ridiculous conclusion, yet I sometimes see similarly unreliable analysis in A/B testing.

It’s logical and laudable that designers should seek data in our quest for verifiability and return on investment. But data must be handled with care, and mathematical rigour isn’t a common part of a designer’s repertoire.

Here’s an example from ABTests.com, a worthwhile project that I feel slightly bad to pick on.

The two versions are subtly different:

• Version A: Upload button bold, Convert button bold, Convert button has a right arrow
• Version B: All buttons regular weight, no right arrow on Convert button

Although minor changes can cause major surprises, I wouldn’t expect these small differences to improve the form’s usability. With the caveat that I don’t know the users or product, I’d even speculate that Version B could perform worse since it reduces the priority of the calls to action and removes the signifier of progression.

The designer claims that version B showed a 30.4% conversion improvement in an A/B test. Here’s why this isn’t quite accurate.

The role of chance

Any A/B test is a trial, so called because we’re observing evidence gained by trying something out. I can never truly know that there’s a 50% chance of a coin landing as a head or a tail – I can only run trials and observe the evidence. Similarly, we can never truly know that a design leads to higher conversion – we can only run trials and observe the evidence. If that empirical evidence is strong enough, we conclude that the design is an improvement. If not, we don’t.

To be valid, trials need to be sufficiently large. By tossing my coin 100 or 1000 times I reduce the influence of chance, but even then I’ll still get slightly different results with each trial. Similarly, a design may have 27.5% conversion on Monday, 31.3% on Tuesday and 26.0% on Wednesday. This random variation should always be the first cause considered of any change in observed results.

The null hypothesis

Statisticians use something called a null hypothesis to account for this possibility. The null hypothesis for the A/B test above might be something like this:

The difference in conversion between Version A and Version B is caused by random variation.

It’s then the job of the trial to disprove the null hypothesis. If it does, we can adopt the alternative explanation:

The difference in conversion between Version A and Version B is caused by the design differences between the two.

To determine whether we can reject the null hypothesis, we use certain mathematical equations to calculate the likelihood that the observed variation could be caused by chance. These equations are beyond the scope of this post but include Student’s t test, χ-squared and ANOVA (Wikipedia links given for the eager). Here’s a site that does the calculations for you, assuming a standard A/B conversion test with a clear Yes or No outcome.

Statistical significance

If the arithmetic shows that the likelihood of the result being random is very small (usually below 5%), we reject the null hypothesis. In effect we’re saying “it’s very unlikely that this result is down to chance. Instead, it’s probably caused by the change we introduced” – in which case we say the results are statistically significant. Note that we still can’t guarantee that this is the right interpretation – significance is about proof only beyond reasonable doubt.

Running the calculations on the above data shows that the results aren’t statistically significant: the evidence isn’t strong enough to reject the null hypothesis that the difference in conversion is simply down to luck. The main problem is the small sample size (128 and 108 users respectively), so I would advise the designer, Johann, to repeat the test with more users. Assuming the observed conversions seen didn’t change (a big assumption) a sample size of approximately 200 users per variant should be sufficient for significance. He could then either reject the null hypothesis or the results would remain inconclusive, in which case there’s no evidence the design has made a difference. In Johann’s defence, he recently posted that he takes the point about significance, and I’m looking forward to seeing more conclusive data for this intriguing test.

Percentage confusion

Significance isn’t the only slippery problem A/B tests face. For starters, quoting conversion improvements is always fraught with difficulty. Since conversion is usually measured in percentages (in this example, 31.3% and 40.7%) there are two ways to quote improvements. We can say that conversions increased by:

• 9.4% – the difference between the two
• 30.4% – the amount that 40.7% is bigger than 31.3%*

Any percentage improvement quoted in isolation should be challenged: which of these two calculations has been used? It’s dangerously easy to assume the wrong figure without sufficient context.

The A/B death spiral

A/B tests also suffer from a common quantitative problem, in that they tell us what but not why. I’ve written about this previously in What if the design gods forsake us. It’s wise to back up numerical tests with qualitative evaluation (eg. a guerrilla usability test) so we can make informed decisions if data suggests we need to rethink a design.

Even with backup, sometimes A/B tests are simply the wrong tool for the job. They can provide powerful insight in some cases, but in the wrong place they can be a blind alley or, worse, a weapon of disempowerment. Logical positivism and design don’t mix – not everything we do can be empirically verified – yet some businesses fall back on A/B testing in lieu of genuine design thinking. I call this the “A/B death spiral”, and it plays out something like this:

Designer: Here’s a new design for this screen. You’ll see it has a new navigation style, tweaked colour palette and I’ve moved the main interactions to a tabbed area.

Product owner: Wow, those are pretty big changes for such a high-risk screen. I tell you what: let’s test them individually to see which of these changes works and which doesn’t…

As the proverb suggests, sometimes you can’t jump a twenty foot chasm in two ten foot leaps. Cherry-picking only those design elements that are “proven” by an A/B test can be a route to fragmented, incoherent design. It may earn marginally more money in the short term, but it becomes hard to avoid a descent into poor UX and the long-term harm this causes.

Being faithful to data

Given the potential hazards, I’m concerned about the naïveté with which some designers approach quantitative testing. The world of statistics rewards an honest search for the truth, not dilettantism, and I’d advise any designer moving in statistical circles to pick up some basic stats theory, or at least partner with someone knowledgeable.

A flawed A/B test, be it statistically insignificant, misapplied or misquoted, is nothing more than anecdotal evidence. It’s the same crime as making a website red on the feedback of one user. Yet an impatient designer, seeing the example I quoted above, could quickly jump to a false conclusion: “I should remove arrows from continue buttons: it’s 30.4% better.” Perhaps this designer deserves what he gets. It’s likely he’s only really interested in shortcuts to good UX, and linkbait lists of “Twelve ways to make your site more usable.” Since he understands neither the mathematics nor the context of this trial (timescales, userbase, surrounding task) he will inevitably grab the wrong end of the stick. Nonetheless, he is out there.

Don’t let yourself be that designer.

Photo: snellgrove
* subject to rounding.

Posted in design, statistics, user experience | 25 Comments »

Q&A: getting into user experience

Tuesday, October 13th, 2009

For the past few years I’ve given an annual talk at UCL to students of the HCI with Ergonomics M.Sc. It’s always a pleasure to share my questionable world view with impressionable minds, and I look forward to the sessions in much the same way as one secretly enjoys a visit from a drunken uncle.

In an effort to make this year’s session a little more interactive, I pulled out an old Knowledge Management set piece:

1. Distribute post-its
2. Ask everyone to write one question they wish they knew the answer to (preferably about the topic at hand).
3. Stick the post-its on the walls. (It’s surprising how much people group them, despite your invitation to use any of the three free walls)
4. Ask everyone to read each post-it.
5. If they too want to find out the answer to a question, tell them to mark the post-it with a question mark. If they think they have an answer, mark it with a tick.

It’s not that surprising to find that a room of similarly qualified students share similar concerns. What’s more interesting is that many of them can also help to answer each other’s questions.

The purpose of this exercise is of course to show that networking and collaborating is valuable, and not just a case of awkward conversation and limp handshakes. However, having made this slightly facile point, I realised that most of the posted questions were damn smart and deserved to be shared more broadly. So here are a few that were particularly interesting, and some proposed answers from myself. I’ll throw a few more up later this week.

Please contribute in the comments if you have any opinions, particularly if they differ from my own.

Is the graphic design of a site more important than usability when initially attracting users to the site?
I say yes. Research shows users form an opinion on the credibility of a site within milliseconds of visiting it. To form a valid opinion on usability takes use, which may not happen if those impressions are negative. However, the line between the two is of course blurred, and a site can successfully convey usability through layout, visual design and information hierarchy. There are plenty of other factors that have an impact too: load times, content and proposition spring to mind.

How many hours do you work a week?
Define “work”. I’m paid for 37 hours, and most of that is spent on billable client work. But add in commuting, writing articles and conference talks, mentoring, and reading about my field and it would exceed 60. Yes, I’m aware that’s a little unhealthy. Good thing I enjoy it.

What’s the most useless skill you think we’ll learn from this course?
Probably rifling through academic papers to find an authoritative source that proves or disproves a detailed HCI argument. Truth is, not many people in industry will care. It’s more important to judge the the problem at hand and make the right design decisions based on context. HCI theory can give a strong advantage here, but you’ll need to state your case with something more real: usually how your client will make more money by following your advice.

How much do you get paid?
Not telling. But here are some approximate London figures: £25,000 is fair for a graduate-level position, rising to £35–40,000 with a couple of years of experience. Senior people should be looking at £60,000 and up (seven years and above, probably managerial responsibility). Freelance rates typically range between £275-£400/day.

What are the best design tools in HCI?
Thinking, conversation, sketching, software. In that order.

Can you be a good UX designer and a good programmer at the same time?
You can be good at both, yes. But who wants to be just good? Deep specialists tend to better than jacks-of-all-trades, and only extremely rare superheroes can be world class at both. I do, however, strongly recommend that all designers learn to code to a reasonable standard, and that all developers learn the fundamentals of design. Speaking each other’s language is the easiest way to ensure good designer-developer relationships, and one of the easiest ways to become substantially better at your job in a short time.

Do you need to draw well / be arty to be a user experience designer?
Some drawing talent helps, but sketching well is a skill that can be learned and that comes with practice. Its main value is when communicating with clients – a well-crafted sketch can simply convey more information than a poor one. However, it’s more important to develop a designer’s mindset. As Jason Santa Maria says, “sketchbooks are not about being a good artist, they’re about being a good thinker.”

To finish, two questions I don’t feel fully equipped to answer. How would you answer them?

• What’s the most interesting HCI related job out there?
• How do I get into the user’s head?

Posted in design, user experience | 5 Comments »

The behaviour you design for

Thursday, October 8th, 2009

I’m working on a site that’s grown from no deal to big deal. Earlier design oversights have created user coping strategies so ingrained that I mustn’t disrupt them with my new design work.

Another reminder that you get the behaviour you design for.

Photo: Tim Bradshaw

Posted in design, user experience | 2 Comments »

EuroIA 09 in review

Wednesday, September 30th, 2009

It’s important to accrue tactics to cope with the disruption of travelling. Quick currency conversions, self-conscious squints at unfamiliar coins, departure lounge distractions (ask Alain de Botton). In Scandinavia, I’ve learned to open clearly with “Hello” to announce myself as a foreigner, since the local salutation “Hej” is a homophone with informal English equivalents.

Copenhagen, site of EuroIA 2009, and Malmö, where my evening sofa awaited, share more than greetings, efficiency and cost of living. They are joined by the 7.8km Öresund Bridge, a zoetrope giving glimpses of distant wind turbines in the water.

This sense of mutual destiny – two nations connected by a single structure – feels entirely European. EuroIA was similarly interwoven with shared experiences of linguistically awkward networking and untold cultural unity. The sessions ranged from poor to intriguing (I’m still no fan of the blind review process) but there was something of a BarCamp atmosphere of willing each other to succeed. EuroIA is a gathering of the underdogs, feisty and proud, and it doesn’t have to be the way they write it in the States.

I particularly enjoyed Joe Lamantia’s peek into the architecture of fun, Sylvie Daumal’s struggle for acceptance in a hostile environment, and Andrea Resmini’s intricate analysis of how IA can bridge the real and digital worlds. Perhaps it was a shame that these sessions were book-ended by an American keynote and closer. Their sessions were undoubtedly interesting, but I hope to see a European presence in these elevated slots next year.

My talk The Future Of Wayfinding seemed to be well received. Since I’m repeating it shortly I’m not yet posting the slides, but here is a list of credits and source material for anyone interested. The topic fitted well with the conference theme of Beyond Structure. Topics such as the Semantic Web, ubiquitous computing and what I can only clumsily label ‘unhierarchy’ were prevalent, and I fully expect them to be reflected in next spring’s US circuit.

Next year we visit Paris, capital of a country almost entirely oblivious to user experience work. It seems we Europeans really do pull together in the face of a challenge.

Posted in conferences, user experience | 2 Comments »

Wayfinding through technology

Tuesday, September 15th, 2009

I’ve written a new article for Johnny Holland magazine, and it’s out now. It’s a short essay on the topic of my upcoming EuroIA talk, looking at how technology can improve our geographical models and help us get around.

The timing’s pertinent, given the recent public and professional surge in interest in augmented reality. However, it’s a double-edged sword. Hopefully the article will be of interest to a few people, but I do fear that we’re already slipping into a trough of disillusionment with AR. As with any early-stage technology, the infatuation isn’t really being backed up by practice – the applications are patchy or unreleased right now. I don’t wish to fan the flames of hype, so in a way, I would have preferred to write an article when the practical applications have matured slightly.

Nevertheless, I hope you enjoy it. It’s the first of a few writing projects I have up my sleeve for the next few months.

Posted in articles, user experience | No Comments »

• « Older Entries