Eyetracking Web Usability – review
6 January 10
Time to pick sides: Jakob Nielsen has written an eyetracking book. I can scarcely think of a more divisive pairing: mention either within earshot of a UX aficionado and you’re in for impassioned advocacy or scornful ridicule. Me? I’ll confess both subject and author have left me unconvinced in the past, but I approached Nielsen and co-author Kara Pernice‘s new book with curiosity and as objective an outlook as I could muster.
Eyetracking Web Usability is the outcome of the largest eyetracking study ever undertaken: 1.5 million fixations from 300 participants. Nielsen and Pernice are clearly keen to stress the magnitude and legitimacy of their research. Their test script, posted in full, is well considered and comprehensive, covering a range of tasks representative of real web use.
After a brief recap of eye physiology and saccades, the book begins in earnest with a detailed breakdown of research methods. Findings then stretch across chapters discussing specific web elements in turn: navigation, forms, images and so on. At their best, these chapters reveal flashes of usefulness. A chart of eye fixations versus layout density shows minimal correlation, demonstrating that busy pages simply dilute attention from the most important information. The book also touches on the important role of information scent and microcopy, declaring insightfully that “a link is a promise”.
In typical Nielsen style the text is heavily punctuated by summary boxes. Sadly, it quickly becomes apparent that these make the point just as effectively as the full text. Eyetracking Web Usability is all fat, no meat. Wasted space includes a page on why a 7-point Likert scale is better than a 5-point one, and five pages on male users’ propensity to fixate on dog genitals. The writing, meanwhile, veers from redundant to simply cringeworthy: “Give that Wii a rest, and go prioritise your Web page layout design. You can do it!”
A chapter on adverts (whose raison d’être is of course to attract the eye) starts promisingly. An ad has a 36% chance of being seen by a user, a figure surprisingly unaffected by user task. However, it soon descends into known generalities: banner blindness and users’ dislike of irrelevant advertising. The chapter encapsulates Eyetracking Web Usability’s main shortcoming. Eyetracking demands specificity: carefully planned tasks on an individual site. Nielsen and Pernice’s 300-person test can only dilute potentially salient points into generalisations that even a novice designer will already know. The conclusions cover ground so well trodden as to be barren.
Despite the authors’ focus on rigour and transparency, serious concerns surround the research methods themselves. Heatmaps from the tests are dated from late 2005. With lab time accounting for five months, the study was therefore complete by summer 2006. Why then was this book not published until the brink of 2010? It is hard to avoid the impression that the results sat untouched for years and were subsequently rushed out in a lull of client work. Eyetracking Web Usability also misses a huge opportunity by focusing solely on informational websites. Web apps are discounted since eyetracking can’t handle dynamic elements, including Ajax and even dropdowns. The results are thus only valid for an increasingly small part of the UX designer’s 2010 workload.
Most worryingly of all, it seems that the tests were conducted in Internet Explorer 6. Browser choice does not appear to have been offered to users, and where browser chrome is shown (it is stripped in the vast majority of the heatmaps), it is unmistakeably IE6. If this is indeed the case, it nullifies many findings since the primary browser innovation of the 2000s – the tab – is unavailable. In IE6 a link is an entirely binary choice: go there, or stay here. Modern browsers allow an important new behaviour: Open In New Tab, creating tentative and plural navigation steps. It’s likely Nielsen’s participants relied far more on the Back button and their short-term memory than today’s users. Their search engine use is also likely to be different, since IE6 lacks an inbuilt search box in the UI.
Eyetracking Web Usability thus lacks the rigour required to be taken seriously as an empirical work; however, its adherence to factual reportage make it a chore to read. Even the most ardent enthusiast will skip over paragraphs that merely disclose participant actions in minute detail. It’s sixth form science at best; utterly literal, over-eager for the praise of the adjudicators. The effect is exacerbated by the disappointingly scant acknowledgment of others’ work. Few external insights or breakthroughs are admitted, although NN/g reports are of course suggested as ways for the reader to supplement his knowledge.
The book’s conclusion will come as no surprise to the reader. “Eyetracking fills in the details… Most companies should not bother conducting their own eyetracking studies.” It is hard to disagree. The book does nothing for the eyetracking industry except cement its status as an expensive diversion; the excessive cover price of £44 only reinforces this. If this is the accumulated wisdom of the largest eyetracking survey in history, we can safely consider the technology inconsequential.
Remember those design principles you learned ten years ago? Eyetracking shows they’re right. Carry on.
Subscribe via RSS
Follow me on Twitter
19 comments on Eyetracking Web Usability – review
So you didn’t like it then? :0)
The conclusion sounds like the best bit to me. Of course nobody would take that conclusion seriously if they hadn’t actually done a lot of eye tracking.
You can do eye tracking on dynamic elements. it’s just a bit more tricky and you couldn’t do it in 2006.
Last time I did eye tracking, it was the software that restricted browser choice as it is this that launches the browser.
I don’t think they wanted to do anything for the eye tracking industry. I for one am happy they’ve come out with a book that says “You probably don’t need to do eye tracking”.
Concluding “You probably don’t need to do eye tracking”, is analogous to a person with little experience of carpentry saying to another would be carpenter “You probably don’t need a hammer to make a staircase”. Whist it’s true, you could make a staircase without a hammer, it is not good advice.
To be fair, at least he has the opportunity to give an unbiased opinion. Unlike someone who say, for example, based their entire business around eye tracking.
Bit like listening to Imperial Tobacco telling you that fags are good for you.
Robert, very few staircases are made without hammers. Almost all of the best user experiences and interfaces are, I guarantee, produced without the use of any eyetracking whatsoever.
Moreover, the conclusion is not just that of Cennydd, it is that of the authors of the book, who, like you, have a vested interest in the widespread usage of eyetracking. But unlike you, they have the intelligence (or is it honesty?) to see that eyetracking is in no way an essential tool for most interace designers.
Chris – let me get this straight. If I think eyetracking has value in the design process then I’m stupid and dishonest? Or is it only if I think it’s an “essential tool” for “most” interface designers…? Just want to be clear on the insult, here.
Nick, you know I love ya, but If you think eyetracking is exactly as essential to interface design as a hammer is to carpentry (which is to say that it is de facto 100% essential), then yeah, that’s pretty stupid.
I’m glad you love me, Chris, but I think you are operating below your typical intellectual level yourself, here. Robert’s metaphor was clumsy, but your (and others’) willingness to personalize this debate is really distressing. It does nothing to advance anyone’s understanding of the issues and it creates a negative impression of the UX community in general.
You’re right, Nick, I was being an internet jerk. But that metaphor, however clumsy it was, implied that those of us who don’t use eyetrackers are like moronic carpenters who try to build stairs without hammers, which kind of felt a little like an insult, too.
A modern hammer is essential to carpentry, but during the evolution of using wood to make things there was a time when a heavy stone that fitted well in the hand was the peak of carpentry technology.
A hammer is not a paradigm shift from a stone, it’s a technology advancement that allows an already skilled carpenter to do even better work. Eye tracking is not a paradigm shift, it’s a technology advancement that allows an already skilled moderator to do even better work by moving away from a concurrent talk aloud protocol towards informed use of a retrospective protocol.
So let me get this straight, Robert…
hammer : stone :: eye-tracking : all previous design methods, processes, and tools?
Jay, are you trying to employ a variation of the Chewbacca Defence: http://en.wikipedia.org/wiki/Chewbacca_defense ?
To be honest – I made staircases in the States for two years without a hammer – we used nail guns. Wouldn’t want to walk up them 10 years later on though… how does that fit in?
Lets not get too off topic here shall we….
Cennyd – excellent article – I think though you were mightly restrained in your critique of the book and their methods.
I read the book over the holiday and it made me cringe as someone who does use eyetracking as a tool to assist with some aspects of UCD – but hey its just a tool and if you use it badly then you produce the hakneyed analyses (the scatter plot of density v num fixations of 35 ‘random’ pages leading to some 8% conclusion based on a correlation analysis – oh pleassse – and check out the line graph for the baseball thing – smirk) and weak visualisations and interpretations that populate this book.
You don’t need eyetracking to produce great design or interaction design – but actually you don’t NEED usability testing either (or any of the other methods us UCD professionals use – but you tend to produce much better designs if you do use them). Eyetracking can help if you use it correctly and can help a lot for certain aspects of design – if you have access to one then you should probably use it (it helps) – if you don’t no worries actually doing the testing is the most important thing – we all know and appreciate that (although see Rolf Molichs common usability evaluation (CUE) reports for some interesting perspectives on that).
In the past I have read a number of rubbish books and atended poor seminars about many topics – it doesn’t make the topic necessarily bad – just reflects on the author or speaker…
@Robert, I’m simply trying to diagram your analogy:
http://en.wikipedia.org/wiki/Analogy#Identity_of_relation
Cennydd, as a reseller of eye tracking equipment (and getting back on topic after the chippy debate) – and as a reader of the book I agree with your summary and review of te book itself. I will of course disagree with the conclusion in the book and those that agree with it’s summary. The telling tale for why NN/g wrote this now can be found in the pdf they released as a ‘favour’ to their following where they openly stated they chose eye tracking in usability as they needed a topic for a new book. They are using 2nd generation hardware and software, an old browser and as you can now test ajax, web 2.0, flash, drop down menu’s and the like with eye tracking software AND whichever browser you want to use – the book is indeed outdated before it hit the shelf. Also as Jon has said previously there are many tools in a toolbox – each person has a preference to which ones they use, or that their skills or methods are suited – and the creaftsman should not be knocked for doing it their way. That is after all how things evolve and innovate.
And I will leave you with a quote as to why they wrote this book, taken from their Eye Tracking Methodology paper (which incidently there is a supplementary 8 page document explaining how technology has advanced in the 4 years between writing it, and releasing it…)
“Third, we have to admit that one reason we embraced eyetracking was in order to write several reports as well as our newest book. Eyetracking provides a new way to illustrate Web usability that’s very suited for publishing a full-color book that’s going to look good in bookstores.”
As a supporter and champion of eye tracking I cringe at this statement… if they don’t understand it, practice it becuase they believe in it or are not interested in eye tracking – please stay away from it. Or let someone talk to you about it and inform you how to utilise it effectively.
Interesting review and debate. However, I think the broader point, beyond carpentry, is that there are various methods that can be employed for similar outcomes. It is being wise to what is in your tool chest and what output you desire and the level of utility. Just like a monthly quant tracking study is not good for most companies (they simply can’t adjust quickly enough to do anything with the data) eye tracking is not a quick fix for all design woes. Nor, does it replace good basics found in standard IA, UX and programming.
It also does not hit the 80/20 rule. You can find out if you forgot a “next” button or something is below the fold with standard usability. However, what eye tracking gives you is unique and valuable IF done on the right projects and set up the correct way. We had a client that had low click rates on tabs at the top of a page and heavy clicks on the left hand nav. The initial thought was these tabs were not seen since they were below a banner and thus the client was going to delete them and redesign. What we saw immediately in the eye tracking was that user did see the tabs and then moved to the left nav losing valuable time. The issue was with the naming of the tabs not being clear, not the design. Given that eye tracking separated the “why” from the “what” in a way not possible with other methodologies.
And, eye tracking creates cool looking output and some quant methods that teams from C-level to designers can get around. On many projects that is more important than anything else.
My two cents as I am putting off reviewing a report late!
Interesting conversation. It definitely shows fair scenarios about eye tracking and when and how to make the best use out of it. We know our profession needs all the help it can get.
Still, there is no toll that by itself is indispensable, there’s always another way to do it great. Always. Like there’s great wood furniture (and even architecture) made without one single hammer or nail for that matter.
The release of Eye Tracking Web Usability reminded me to blog The Great Eye Tracking Debate Kara Pernice and I had at the UK UPA, so here it is:
http://thinkeyetracking.com/Blog/?p=360
I hope it clearly communicates the benefits of eye tracking and web usability testing.
Add a comment