A friend from a successful e-commerce site got in touch recently. He’s been steadily redesigning the site, with the help of an external design team. I know the company he’s working with. They’re good. But he hadn’t yet seen the bottom-line rewards he’d hoped for, so he asked for my thoughts.

Here’s my response, edited for confidentiality. Perhaps it’ll be useful to others, and I’d also love to hear any suggestions you have.

From: [xxx@yyy.com] to Cennydd Bowles
The site looks a million times better, but unfortunately our conversion rates have actually dropped. There is certainly noise in the data and an increasingly competitive environment but […] do you have any idea why our conversion rate would be worse?

From: Cennydd Bowles to [xxx@yyy.com]
The short answer is “I don’t know for sure”. The long answer is, well, a lot longer and needs me to talk a bit about the nature and philosophy of design. Please bear with me.

Design is inherently less predictable than most other product fields, since it closely involves emotion, comprehension, taste and all those complex, deeply human attributes. That means that design is a gamble. A good designer will improve your odds, but there’s always a chance that their hypotheses (which, after all, is the most any designer can provide) will prove to be false. A solution that works in one context may fail in another. Because there’s not this replicability of process, there can never be scientific ‘truth’ in design; experiments, observation, and iteration are the only way forward.

Much to the design community’s chagrin, sometimes “good design” doesn’t provide the commercial benefits we all expect. Sometimes “bad design” performs better. If I knew why, I’d be a millionaire by now :) I’ve been bitten by this myself – design changes that were “better” by all recognisable theory and good design practice performed worse than the original design. It’s frustrating for all concerned, and embarrassing for the designer.

Figuring out the cause can be difficult too. Introspection of design doesn’t tend to work well – barring major usability problems, it can be tricky to isolate specific points of a design that cause certain actions. The design as a whole has a certain irreducible complexity. So sometimes these surprises just happen, and it’s hard to diagnose the cause. Does that mean design is a poor investment? No. But I would say that it can be riskier than, say, marketing or SEO, which are more linear: generally, put more in the funnel and more trickles out of it.

However, I do suggest seeing user-centred design as something wider than just a means of optimising a conversion rate. While there may not be a noticeable uplift in any specific metric, the raw material of design is frequently intangible: trust, loyalty, engagement, etc. These things are much harder to measure, but they still make themselves felt indirectly in other metrics: support costs, referral rates, customer retention, and so on. Separating the effect of design from these long-term figures is, of course, pretty much impossible, but the long-term aggregated data makes it clear that the effect is genuine (see Apple, etc). Strong design also gives you a better platform to innovate from, and all that good biz school stuff.

But all this philosophising doesn’t answer your question, and I appreciate that the pressures of the bottom line mean you’d hope for a more realisable output for your investment. So let me take a stab at some more direct suggestions:

Natural dip

There’s always a performance dip after releasing a new design, no matter how good or bad it is. This is probably because existing customers’ mental models of how things work have been broken, and it always takes a little time to reestablish those patterns. What can be surprising is the length of this pattern – I know of a company that allows six months to pass before they evaluate the success of a redesign, so the smoke has truly cleared. This particular organisation has a very high number of users, so the effect is naturally prolonged, but do make sure you’re confident there’s still not a temporary effect lingering.

Details

The things that could make the difference in a design might be the little details. I don’t know exactly what your designers gave you, but check to see whether you’ve overlooked small points that might reduce friction. The easiest way to do this would be to ask your designers to run a quick review on what you’ve put live, to make sure it’s working the way they expected it to.

Usability testing

The major problem with metrics is obviously that they tell you what, not why – hence the existence of this email, I suppose. A well-designed round or two of usability testing would give you qualitative data that should help you understand the sticking points. If your designers have already done this, it might be worth asking for the videos so you can go over them yourselves. (I’m not suggesting they’d underreport anything – just that the time pressures of a project mean details can slip under the radar.)

If you haven’t done any face-to-face testing or don’t want to, it might be worth throwing the site into a remote usability testing programme like usertesting.com or usabilla.com. You’ll get some cheap feedback on what’s working and what isn’t. The feedback can sometimes be variable, but as an extra source of data to investigate an issue they can be useful.

Other data

Are there other data points that might guide you to the answer? e.g. have complaints gone up or down? About what? Have you seen a conversion drop among just a particular group of customers, or particular groups of products? (As above, it’s often the existing customers who have to adjust the most, while a new design is often targeted mostly at attracting new customers, who convert well.)

Analytics config

I’ve heard of a surprising number of companies that have reprimanded their designers, saying “Hang on, what’s happened…?”, only to finally admit that their analytics software was looking at the old URLs and conversion funnels. Once or twice that’s even happened only after they’ve spent thousands of pounds to fix the non-existent problem. So it’s worth triple-checking everything is in the right order there.

I wish I could be more specific but for the reasons given that’s inherently quite difficult. What I can assure you of is that that the effects of great design will make themselves felt throughout your business, even if those effects are indirect.

Cheers,
Cennydd