StatCave 8: Roy descends into madness doing a UX review

Ever wonder what on your site causes your customers to make this kind of face?

From actual footage of me shopping HomeDepot.

From actual footage of me shopping HomeDepot.


One of our long-time clients (the site and name withheld to protect the innocent) was getting ready for a site redesign, and they wanted to be sure they had everything lined up that they needed to update.  They asked me to do a quick site review for them, as they knew I'd been through redesigns plenty of times, too, and I might catch some things they'd overlooked.

While the review covered four of their sites, even I didn't expect it to take more than 40 pages, but it did!  Nor did I expect the review process to be as much fun as it was.  I felt like Louis Black doing a roast of a website, and it seemed to be kind of awesome!

There were a lot of screenshots, notes, tips, and commentary.  For example:


And, of course, plenty of animated commentary, as well.


When A/B Testing isn't the right tool...

While I was just being my usual irreverent style, I felt I'd stumbled upon something really valuable.  Most folks I know in eCommerce are, by necessity, amateur conversion rate optimization experts.  However, this often takes a very inside-out approach--testing individual elements on the page, trying to A/B test your way to greatness.

Now, I'm one of the biggest fans of testing you'll find out there.  However, it's important to realize that A/B testing comes in one of two major flavors.  First, there are major tests, where you're replacing something major, and testing across a large audience (say, replacing your product page template, site-wide).  These are usually critical to ensuring that your iterative improvements aren't actually sliding you back down the hill.

The more common type of test, however, is a tiny, incremental, and even more frequently inconsequential change.  These are button sizes, moving elements around on the page, or only testing things on a single, niche landing page.  While sometimes interesting, these are, at best ways to sharpen the blade--something that only makes sense when the blade is already forged.  You can't take a chunk of raw iron, and a whet stone, and expect to make a razor-sharp blade out of it.  That's what A/B testing is like, when applied to an uncertain User Experience.

So, how do you avoid the navel-gazing trap of A/B testing on stuff that's too small to really matter (or worse, end up playing micro-conversion whack-a-mole)?  

Remember who you are serving.

The reason the site reviews I provided this client were so interesting was that I'd done my best to emulate the shopper's perspective throughout, only dropping character to make a point about a potential solution.  It was all written in the first-person, present tense, entirely stream of consciousness.  While this was anecdotal, and therefore arguably of less value than a large scale heat map collection project (or similar), it was far more effective at getting the reader into the shopper's head.

Most of us in eCommerce will try to think like a shopper when we're looking at specific decisions on a specific page or template.  Of course we do!  However, we are always looking at our own site, day in and day out.  We become numb to our site's own ridiculosity.  The other thing that made this work so well was that I went through the entire shopping process, not just mentally hopping to the product page, or a specific step in checkout.  This allows you to realize the stacking effect of multiple little frustrations over the course of a shopping path, which eventually build up to an abandonment

So, the punchline is this: you should have someone else, someone who doesn't work on your site, give you feedback on your shopping experience.  Have them write down their thoughts as they go.  You'll be surprised the obvious stuff you've been missing all these months.  

And, when in doubt, recall the old CRO saying: "Assume your shoppers are drunk," and give your test shopper a couple of glasses of their favorite adult beverages.  Then, sit back, shut up, and listen to their shopping process narration.   

I wanted to try it in video format...

The challenge was to try to honestly shop for something I wanted, but to document the process along the way, this time in video form.  

I tried to trim this more-than-an-hour-long painful, frustrating chapter of my life down to under ten minutes, but it started to resemble a liquid (ie. refuses to compress, just changes shape) at around 16 minutes.  Still, it's a pretty fun result, as you can't help but want to see your site through fresh eyes after watching my descent into madness, as I fought with several sites before giving up, and buying from Amazon.

I generally try not to be a part of such a train wreck, but I was the conductor, so...

Admittedly, there are two things I'll do differently next time.  First, video may allow for some entertaining moments, it's pretty hard to compress down to a reasonable length.  Second, I gave up too easily, and jumped to another site (several times).  While this is what a real shopper would do (which is why I did it--I didn't want to spend my real earth monies on the wrong thing), I think following through on a single site at a time (in video or text) may be better.

How else might we help one another see the forest for the trees, when it comes to our shoppers' experiences?  Comment below, or send your thoughts to me at  If you'd like me to review your site's shopping experience, and aren't afraid of letting me share the result on this blog (probably the written version), let me know at, and I'll see what we can do.  ;)

Up and to the right,