Bronto Summit 2017: Let Your Data Be Your Guide

2017 marked my fifth presentation at Bronto Summit, and this year, I had all kinds of interesting new data to share.  Hopefully, I'm able to challenge some assumptions, and help you get a clearer vision of your own shoppers!

Full Transcript:

Roy Steves:


Thanks for coming, everybody. This is actually my fifth year presenting at Bronto Summit, and this year is going to be a little bit different. I need to sort of give you some background context so that you can understand what I mean by that. If you run across anything in this presentation that you can prove me wrong on, I'll buy you a drink, so that's the addition to the normal Oracle legalese. The presentation was difficult to title. I appreciate you taking a risk on it, but I'm a little bit all over the map with what I was putting together, so I think you'll see why that was a challenge. Back story, in 2010 I started working with Pool Supply World. I was an engineer. They were doing quite a bit of business but it was mostly organic and comparison shopping engines, and so I told them, I'm like, "I'm pretty sure AdWords is just applied algebra. Why don't you give me a shot at that?"



We added 30% to the business in the first year, and then I'd teach myself another marketing channel, find somebody better at it than me, which usually wasn't very hard, and then hire them and add them to the team, so over the course of the first three years I went from an engineer to CMO. That was quite the roller coaster. Then in 2013 was my first Bronto presentation, and I did a talk on multi-touch attribution because I was collecting all of this browsing behavior directly off of my site into a database and I was showing off what I could do with that. 2014, I did data visualization and storytelling, so taking all this data that was in my warehouse and trying to make it actionable and useful, something that the rest of my team could understand and use. Then the next year was after we had been acquired by Leslie's, so in 2015 I had access to more data than God.



I mean, it was just ridiculous. I had billions of rows of data in a bunch of different slices, and so I presented on some segmentation strategies that I'd been using with that larger brand. For those of you that don't live in the 35 states they're in, they have over 900 stores, and do just a colossal amount of business, the only national brick and mortar chain in pool supplies, so I had huge amount of data to play with, which was great. Then, I left that in September of 2015 to start my own thing. Then I was thinking last year, presentation-wise, I'm no longer a retailer. What can I tell retailers that they're going to find interesting? What I did was I took basically the greatest hits of the first two chapters of my career, and did a presentation on A/B testing and site optimization, so some of you I know were there, and I appreciate that.


The big difference was that I had an incredible depth of data, so obviously I had platform data like Google Analytics and Pronto and AdWords and everything like that, but then I also had a ton of data layers ... Oh, yes, SkyNet was our order management system, so it had a ton of data. Grinder was the session tracker and things like that, so I had just an absolutely colossal amount of data to work with. The depth of data was unsurpassed. Now I'm out here on my own doing something else and I don't have that depth of data, but now what I have is access to an incredible breadth of data. The things I'll be sharing from here are an aggregate from a set of sites that total almost a billion dollars in annual revenue. There are people with more data than that, but it's enough that I have quite a bit of fun, and we can find some interesting things out from sifting through it, so what I traded for depth, I now make up in breadth.



What we're going to do is we're going to take a look at some of these types of things. When I talk about best practices and industry standard kind of common knowledge type things, I'm talking about stuff like this. StatCounter's not a bad source of information, but there's a lot of information that you don't have in terms of how you need to be able to interpret this. From a statistics standpoint, you don't know what the sample size is, you don't know what the error bars should be, you don't want the standard deviation is. There's just a lot of missing pieces, and that ends up being sort of like just sort of feeling around the outside of the data. I mean, it's analytical phrenology basically. You're not really getting down into the guts of it. In order to understand the data, we're going to have to peel it back, so instead of phrenology we're going to go with X-rays and MRI style.



We're going to look at aggregate data kind of like that StatCounter one, but then we're going to peel it back and I'm going to show you how different anonymized sites make up those totals, so you can get an idea of whether or not you should take anything you see on the Internet seriously when it comes to e-commerce. We're going to be looking at a bunch of dimensions. These are just some examples. Sometimes we'll use more than one, just so you can get a flavor for what we're getting into. The first thing I'm going to be doing, and most of this is from Google Analytics with a little bit of flavor text pulled in from AdWords data as well, but we're going to start with the channels and we're going to focus on three channels that I selected both for their size, the audience, and then how much influence you can have over those. We'll start with organic PPC and email, obviously, and this is in Google Analytics under conversions, multi-channel funnels, and overviews.


This Venn Diagram does a pretty good job of giving you a rough idea of the overlap of your channels. Overlap is a different conversation, but we're just looking at the relevant sizes. The reason that I wanted to do that was when you are a retailer you're in your own echo chamber, and that's all you hear for most of the year is how you're doing, and you don't really know what is normal. You don't know if you're weak or strong in any given channel. That's part of the reason we all love events like this, is because it's one of the few opportunities where we get to compare notes with people. Well, imagine this as being able to compare notes to a couple of dozen people all at once. We're going to start with typical. This is fairly normal. This is a relatively large site, it's fairly mature, and it has organic is the largest, followed by paid, followed by email. I'm not saying that this is necessarily virtuous. I'm just saying it's normal.


There are other variations on this. This one is also very typical. This one is actually a little bit different. You'll notice email and paid are the same size. Paid is a little bit smaller relative to organic, so they might be under-investing in paid relative to the size of the organic audience, but that might be a function of operating on a narrow margin, for example where the organic ends up being more critical to them because they can't afford to participate in PPC in the same way, and then email relative to paid is doing pretty well, and relative to organic is fairly typical. You'll see this pattern again and again, so these are three examples of fairly normal looking channel contributions. What does it look like when it's not normal?



This is the first example. It looks similar to the ones in the previous slide with one notable difference. Paid and organic have switched spots. This is actually a bit more expensive. This happens sometimes when a site does a re-platforming or something like that, or has been penalized by Google and is otherwise trying to fill in a gap using paid. It's a perfectly reasonable strategy but it does take some monitoring of your overall profitability to make it work. That's a little bit concerning even for somebody who's in the paid search space now. That makes me a little bit uncomfortable. Then there's this one. Not only is paid much smaller than organic, which is a problem, email is practically non-existent, and anyone here can recognize how that doesn't quite make sense. In this case, this is very likely a one-person shop. They're responsible for all of the channels and they've resorted to just batch and blast emails out of necessity, right? Some of us have been there before.



That's something that would be at least an opportunity in email, if not an opportunity in PPC as well, but that's not even the worst we've seen. This is pretty ugly. Now, there's two possibilities here, and these are ... I made them smaller. They're smaller sites, so not to scale, but you get an idea. This is one of two possibly catastrophic situations. Either they're not doing email at all, which is disastrously bad, or they're not tracking email, which is also disastrously bad. Nothing about that makes me happy with how they're doing on email. It's not the only one. I've seen that again and again, and then this is actually the worst one yet. This one makes me cringe down to my bones because not only are they not doing or tracking email, then paid is also larger than organic. That is a very expensive way to scale a business.



If we line up all of the different channel types with direct on the far left for those at the back, all the way down to social on the right, this is the kind of data that I could have generated if I were trying to do the same kind of sort of white paper nonsense that you see out on the Internet everywhere. It's very easy to produce these kinds of charts, but they're not that insightful. The reason they're not that insightful is if we then kick on the X-ray machine, we see that there's a huge amount of variety of distribution between the sites. Each of these sub-bars is then a different site that's adding up to that total, and as you can see the blue, the largest site at the bottom, makes up almost all of that display. If you had been on this slide and you saw that your display channel is smaller relatively to this chart, then you'd overestimate the value of that channel if you weren't that blue site. I'm going to refactor this data a little bit. We're going to show you another way of visualizing the same thing.



Now, each of the channels are broken into three types of sites: brands, which sell something with their own name on the label, so they are manufacturing and direct to consumer sales. There are retailers who sell things manufactured by other people, and then there's a few sites in my sample that do enough of both that they defied that categorization, but each of these where you add up the three columns responsible for those site types would be 100%, so that gives you a relative revenue share for each of those channels by that site type. The reason I was doing that is I wanted to validate some assumptions about how brands versus retailers work in this space, and so just so you know how you're reading this channel, remember when I pointed out display is almost all blue? Well, that blue site over there is a brand site over here, and you can see that it then dominates the relative charts, so that's how you interpret this.


The first thing I wanted to look at was the brands. I thought that brands would have a more dominant direct than organic because if you know Levis for example, then you seem to me to be more likely to type in Levis in Google or go to, but if you look at it, the retailers are actually holding their own on direct and actually have a little bit of a lead on organic, so I was completely wrong. Then that actually ends up flowing into the paid section as well, because I assumed that retailers were going to be trying to make up for that brand identification gap by investing more in PPC. I'm wrong. The retailers have a lower average share than the direct consumer brands in my sample. That was interesting to me. The next thing I looked at was email, and this did follow that a pattern a little bit.



If you know the brand that's selling you the product directly and it's a brand that that name is on the item that you have, that physical object, then I figured you're going to have more loyalty because you're going to be more familiar with the brand of the manufacturer than the brand of whoever sold it to you. If it happens to be the same brand, then you're naturally more likely to repeat purchase from them. Retailers on the other hand, don't have that advantage, so when I was with Pool Supply World, we'd send you a pump. It said Hayward on it. It didn't say Pool Supply World on it, and we would have to make up that gap in customer loyalty through the email program. You can see that we weren't the only ones using that pattern by how the retail section was dominant in email. Then the other three are much, much smaller samples, and pretty noisy, although it's interesting with the amount of bluster in the social space, that nobody in this sample was really generating a ton of revenue off of that.



The summary is basically if your Venn Diagram when you go into Google Analytics, doesn't look vaguely like this, either you're handling a special use case and that might be, or it might mean you have an opportunity. I'll be sharing all the slides to you so you can use these benchmarks however you like, and then hit me up with any questions thereafter, but if it doesn't look like this maybe it should. Then Google Analytics obviously gives us a ton of data on device types and there's this fetish for mobile first, mobile optimization, mobile site speed, and all those kind of factors, and it has been around since 2008, and it's not wrong, but some of the ways that we've been brainwashed into thinking about mobile didn't quite line up, so I'll use my own assumptions as examples in this. We're going back to the brands versus retailers first, and my assumption was that brands would have a stronger performance on mobile because that trust is there.


The product is the brand, then there's not a barrier to get over to say, "I need a Hayward pump, but I don't know about this Pool Supply World company." If it's a Hayward pump and you're buying it from, then that trust is already sort of implicit if you're already going to buy the product. To sort of go through the rest of these slides, I do want to point out that I'll be using this metric quite a lot, value procession. The reason I'm using this is it takes into account both average order value and conversion rate, and both of those will vary depending on which segment we're looking at, so this ... You may not sit in front of this every day, but it's a very powerful metric and in the AdWords space, obviously value per click is sort of an analog somewhat. This is where we'll start and then I'll break out differences in AOV and conversion rate where appropriate.



I'll be doing a lot of scatter plots like this. What I'm plotting here is mobile value procession on the vertical axis and desktop value per session on the horizontal axis, and then I've got brand and retailers and both split out once again. The size of the dot is roughly descriptive of the size of the site, so you can get an idea of which ones are big, which ones are small, and for most of these ... The small ones are also younger sites. They're not one million dollar a year that had been one million dollar a year for 10 years, at least in my sample, so those might behave differently than this group, but you can see there's not really a correlation. It's just sort of all over the map, which I didn't really get. What I did was I broke this down into conversion rate. If I can't make sense of the conglomerate, then I'm going to go down to the atomic level and this is what I saw.



This obviously does tend to have some sort of pattern to it. You can see it goes along this diagonal, which is three to one, so a lot of these sites down here in the lower corner are converting on mobile at about a third to about a half of whatever they are on desktop, so this is sort of where I'm getting these metrics is saying that if your mobile conversion rate is a tenth that of desktop, then you're out of the band of normal behavior and that might be a problem, but there are some exceptions. This guy way up here, that mobile conversion rate, and this is over 12 months of data, is higher than the desktop conversion rate. It's not a huge site, but that is a very unusual behavior indeed. Then obviously these two down here have a very rudimentary mobile experience and a miserable mobile checkout, and you can see that reflected there.



This is another way to visualize this. I broke this one up because I wanted to get a look at tablets, but obviously three dimensions in a two-dimensional graph are kind of tricky, so I couldn't quite do it in that last graph, but what I've done here is I've just flat-out plotted revenue versus sessions. The reason I've done that is that depending on where you're getting your data, tablets will get lumped in with desktop sometimes because they're a large form factor screen. The browsing behavior is very similar to desktop in a lot of ways, but they also get lumped in with mobile a lot because it's a touch screen and the conversion rates are lower than desktop. Why is it that there's no consistency in our industry about what to do with this tablet segment? It's smaller than the other two, sure, but it seems to vary day by day whether it's treated as a mobile device or a desktop equivalent, and that's because those yellow dots literally are just sort of mashed in between. The reason that it's not consistent is because it ends up being muddled right in there.



It's a complicated topic, so we're going to dive into some more tablet stuff, but here's an example of what I'm talking about. This is a mobile retail commerce sales, percentage of retail, et cetera, and this is not telling us whether tablets are included, so with or without that segment that's relatively high converting compared to mobile phones, I can't really use this data. This is that phrenology problem all over again. Also, while it's difficult to read at this size, 2016 through 2020, all our asterisks, I think this is a single year's worth of data in 2015 that they just extrapolated with a suspiciously linear curve, so there's a few reasons I don't really buy this chart. What we can do is find other examples like that. I already showed you this once. This one at least did us the favor of telling us that it lumped mobile and tablet together. Now, if you came across this as I did, and were looking for this to inform how important your mobile site optimization projects might be, you'd be forgiven for thinking that it's about as important as your desktop optimization projects.


However, there's a couple of reasons that that would be an incorrect interpretation of this chart. One, this is global data, so outside of the US market there is a higher preponderance of using mobile as a primary device, so that exaggerates the effect if you're a domestic company. If you're global, that's fine. The other thing is I don't know where traffic is on your PNL, but it's not on mine, so you can't buy a beer with traffic. Therefore it doesn't actually count for anything, and this is guilty of that. It's looking at traffic only. The other thing that we're going to do is this kind of breakdown, and this is also from StatCounter. The odd thing is that it doesn't add up to the same thing as what the other chart was showing us. There's a difference in their methodology and they don't really tell you what the difference is.



This is once again global. It's still traffic share, but once again, mobile looks like it's about as important as desktop and then tablet is almost nothing. Now, global data, just like I said, it emphasizes mobile, it deemphasizes tablet, and that's why you see such a small slice here. If I take my data and try and reproduce this chart, what does it look like? It actually looks pretty similar. This is still on traffic, but you can see the tablets are obviously a much larger slice and then the relative performance of desktop and mobile is still there, but this is still the phrenology version. Let's kick on the X-ray. I've put each of the contributing sites that fit this type of research in here as a horizontal bar, but if you just sort of blur your eyes a little bit, you can see that that's about the same kind of average spread, but there's a huge amount of variety within there. In fact, I'd like to highlight a few of these.



The blue line and then the purple line above it as well, to a different degree, are fairly typical on desktop, but they have a relatively small mobile share of traffic, and a relatively high tablet share of traffic. That's an interesting thing. We'll dive into some possible explanations for that, but the first thing I want to do is once again point out that these ... The larger sites are on the top and the smaller sites are on the bottom. These bottom three are all kind of interesting. The orange one is almost exclusively a mobile site. I almost want to go back and look at that account, and see what are they doing inbound wise that's leading to such a preponderance of mobile users on that site, and then almost no tablet users. There's something almost suspicious about that one, and then the other two, despite being even smaller still, have something that's more typical in terms of performance, so even the smallest sites can have a fairly normal distribution and then sometimes you get these weird outliers.


Once again, traffic isn't revenue, so when we kick it over to revenue, all of a sudden mobile doesn't seem nearly as dominant in terms of where we should be putting our attention. The interesting thing is still that that orange one continues to have a huge amount of revenue, but that might not be because they're exceptionally good at mobile. It's just because they don't have that much other traffic to measure. There's different ways to read that same data there, and we're looking at the smallest sites definitely have the most mobile revenue and the most mobile traffic, so that's interesting that in the middle section over here, we have more share to the mobile side. My hypothesis is that because those are smaller sites, and they're newer sites, they're probably running templates that were responsive from the get-go, but there's an alternative interpretation.



Larger sites with more resources might have a more developed desktop experience, so even that conclusion might not be sound because the majority of the money is still coming from desktop, so even though you see all of these trends that say mobile first, mobile first, mobile first, yes, mobile's growing. Mobile's critical, especially multi-device shoppers and such, but right now don't forget that desktop is still paying the bills, so as you're working on your responsive templates, be sure to spend a good chunk of your time on the full-size view and not just myopically focus on the mobile version. I'm not saying don't focus on mobile, just remember where the money is coming from that keeps the lights on. Another kind of behavior that I've seen across a lot of site owners is an excessive level of attention paid to new to file customers.



A lot of sites are overly optimistic about their retention rates, and so they think that they need to spend more money where there are more new customers versus returning because in their minds that customer's going to come back anyway. They're more likely to come back, but by no means guaranteed to, so I was always sort of skeptical of this. The other thing that happens is using the new versus returning stats in Google Analytics, as if you were comparing new shoppers to people who have purchased before, and that is actually wildly incorrect. What it's actually telling you is whether or not that person bought in their first session, if you're looking at the new customer segment in Google Analytics, or whether they are a multi-session shopper. That could be multiple ... You know, just two days back to back. They might have placed two orders. They might have placed one order. They might have not placed any orders, so just be sure that when you're looking at this report, you're not thinking in terms of repeat customers. These are repeat visitors.



But there still is a difference in the behavior of those two segments, so this is fairly typical. It actually is a little bit more biased toward returning than the sites that I personally managed, but most of the time I see a lot of this 50/50 split. Once again, just like we saw with the mobile versus desktop, one of these has a much larger revenue contribution and that's either higher average order value or higher conversion rate coming from those multi-session shoppers. What this tells me is that we as a group are probably not spending enough time thinking about multi-touch attribution, but the bottom line is don't think that the new customers are ultimately more important than the returning customers, just because they're new to file. Realistically, most of those returning customers could just as easily have gone to Amazon this time around even if they've been to your site before. Yes, you need new customers, but the money that's keeping the lights on is mostly coming from multi-session shoppers and returning customers.



Even that 50/50 split is not really a great rule of thumb. Here are all the sites that I had split up, and you can see, and this is obviously I didn't sort it from largest to smallest because it was way too hard to read, but you can see that there's a lot of variety in there. If you were one of the top several sites or one of the bottom several sites, you'd be forgiven for looking at that 50/50 split and thinking, "Well, that's what I should have too." Well, not necessarily. The other thing is that while I didn't sort this by site size, there was a correlation between sites' age and this, and that's because older sites simply have more returning customers to possibly draw in. Not only that, smaller sites are adding fewer new members to their returning customer segment per day, so there's a couple of factors where larger, older sites end up biased more toward returning than the smaller sites, which tend to bias far toward new.



The way you can think about it is if you spin up a Shopify Plus site tomorrow, all of your customers will be new visitors, right? This is the same kind of pattern, just taken out over years instead of minutes, but once again, what about revenue? Now, this is far noisier, so even though the bottom ... This is, you know, if it's bottom here, it's bottom here, so these are the same order of sites. If you're the smaller sites, even though you have a ton of new shoppers, most of the revenue's still coming from multi-session and returning shoppers, so that ends up being sort of thrown back in the face of that assumption that new to file is somehow more valuable out of the gates. Now, if you want to do some LTV calculations, that's a different story, but just using the Google Analytics data like this, you can see that maybe that's not the best approach. Maybe there should be some sort of hybrid approach that takes into account the fact that those multi-session and returning shoppers are more likely to convert.



Like I said, this is tied to the value of multi-touch attribution because they came from somewhere if they're a multi-session shopper, so if they came to my site yesterday and then they came to my site today, I should know where they came from yesterday if I want to get a full picture of the story. There are other ways to approach this data. Here I've plotted the percentage of new visitors of revenue, so if all of your revenue came from new visitors, it would be very high on the chart, versus the percentage of sessions. If you have a lot of new sessions coming to your site, single session shoppers, then it would be far to the right. The brand versus retail split is a little bit interesting here. The big brands tend to be clustered down into one corner, but the smaller dots up and out to the side show that that preponderance of more new traffic, but they also have a higher share of revenue coming from those new, relative to their numbers.



What I found interesting about this actually is those big dots, especially the gold dot and the big blue dots down in the bottom, you'll notice that the larger, more established sites do have a smaller percentage of new sessions in traffic. That's because they have a lot of returning sessions. They were the ones at the top of that session stack slide, but you'll notice they're not much farther down on the chart, so they're not getting a larger share of their revenue necessarily coming from that larger percentage of returning customers. Oh, I had two of the same slide. No problem. All right, so here's another way to look at this. I'm now plotting returning value procession versus new value procession, and we can see a pattern here. Everything is pretty much above this one to two line, so returning shoppers tend to be worth about twice as much compared to new shoppers.


If you look at the big dot in the middle, that's four dollars per session of a returning visitor, versus two dollars per session of that new visitor, so remember when I was showing you the pie chart with new versus returning on revenue? This is why, and this is the distribution of that. Here's another approach. Now we're just looking at the conversion rate itself, and this also tends to have an interesting trend here. We're looking at a two to one here as well, so we know that that two to one pattern is producing that pattern on the value procession, but I want to highlight one of these sites specifically, and that's this one. While the value procession is pretty much on that two to one trend, it's under the average trend on the conversion rate. Let me say that again. The value procession is on trend, but it has a lower than average conversion rate on returning customers.



If that's the case, if the conversion rate is lower but the value procession is the same, then the AOV must be higher to compensate for that, and if we swap out this other chart, we can see that that's the case. Here is returning average order value versus new average order value, and the new average order value is $150 for that dot, but the returning average order value is only $100. Now this was very interesting to me, and maybe this was just me and my company, we thought that if somebody was going to buy a $1500 pool pump, that maybe they'll buy a $50 bucket of chlorine first to get a sense of whether or not they could trust the site, and then they'll come back and place the large order. We don't see any behavior that suggests that that's the case, because if it would be then you'd see a much higher average order value on the chart on the left relative to the first purchase. The first purchase generally ends up being equal to or higher than the second purchases.



Bottom line here is yes, new to file customers are important, but multi-session and returning customers are producing two to three times as much revenue per shopper and per session. Once again, think about multi-touch attribution, do a little bit of research if you're not familiar with it. Google Analytics makes it easy to sort of get a sense of it without having to be as technical as you needed to be five years or so ago. That's the new versus returning. The next rant is demographics. This is the most abused and misused thing, type of segmentation I've seen in our entire industry. It's effectively, in my opinion, stereotypes gone rampant, and it is ... I couldn't believe that the data would support it. Now let's see if I'm right. There's a couple of different stereotypes I'm going to go into and attempt to blow up. We'll see what my hit rate is.



This is the first one. We were definitely guilty of this. Millennials are the ones on their phones more than any, Gen X tend to be on their laptops and desktops usually shopping from work, and then the Boomer segment tends to be using tablets for a variety of reasons that were mostly driven by stereotypes. This is something we can measure. We have enough data. We can use the demographics data and analytics compared to the device types, and see if there's anything to this. Here's traffic, and there's definitely a difference between the three of them. The mobile and desktop have the same scale and then obviously tablet is a smaller sample, so I used a different Y scale there, but relatively speaking, tablet does skew to 55-plus. Desktop, it goes down to the 25-plus, but it's definitely in that sort of 25 to 45-plus range, and then mobile has the largest percentage of traffic coming from 25 to 45.



The mobile segment ends up being a little broader than we expected, but this is an interesting pattern. What happens when we switch from traffic once again to revenue? It exaggerates, and that's because the oldest segment and the youngest segment potentially are producing the least revenue per share per device type, so the possibility here is this might not be a behavioral pattern so much as an economic one. It might not be because they're old or young that they're producing a difference in the revenue per device type. It might just be an economic one. If we then go back to the sites, so we start to see the variety here. This is ... Each of the colors represents one of my age segments, and then I have younger to the far left and older to the far right, and then now I have switched back to having the sites stacked from largest site to smallest site.



This is noisy. There's not really much of a pattern here. There are a few sites in here that are particularly interesting. The top one of these two has 75-plus percent of its revenue coming from users 55 and older. That's unusual, and it's not a small site. At the bottom, which is smaller by traffic but not necessarily by revenue, has almost 75% of their revenue coming from ... What's that? 35 and under. If either of these two sites saw general, broad spectrum, here's national averages, their site wouldn't fit that model at all. They wouldn't be able to use that data and if they did, they'd be led astray by it, but if we switch from revenue to value procession, then the pattern starts to normalize pretty nicely. Even if you have a random set of people in terms of ages coming to your site, most of them are producing about the same value procession.



There's a difference between quality and quantity in your shoppers, and the standard assumption is that if 75% of my shoppers are from this demographic, then I should focus on that, but it might be the case that all of the shoppers are roughly the same value. I wanted to point out one of these. This one does have a higher value targeting the 25 to 35 range, so they're the exception to my rule, but as you can see, this is way more predictable than it was with the by revenue, obviously. The VPS is more consistent. The next one is ... That was true by volume but not by quality, so I'll give myself half credit for that one. This one's been a part of online shopping since Amazon first came on board, gender stereotypes, and I intentionally chose the most deplorably stereotypical female shopper image. I mean, she's got a couch. This poor woman's laying on the ground with a credit card with no numbers. She's obviously disturbed and fictional.


My assertion is that this person doesn't exist, and that we all need to stop thinking in terms of this shopper that's been crammed down our throats for the past decade. Am I right? Well, here's the revenue share by gender. I'm going to skip traffic entirely and just go to the brass tacks here, and as you can see, there is a wide variety of gender distribution on this, so if I'm selling product X and I'm assuming that my shoppers are all female, not only am I potentially wildly incorrect, what am I going to do with that? Am I going to be one of those schmucks that just makes it pink and calls it targeted for females? It's just ... It drives me crazy. The next thing we do is check out the value procession, and this is very interesting for a couple of reasons. One, the value procession is not only fairly evenly distributed with that one exception of the site at the bottom, second from the bottom, the correlation is actually inverse.



What you can learn from this is if for example I am on a skin care site, I might be the smaller demographic on it, but I might be really into skin care, and if a woman is on a after-market Harley parts site, she might convert more highly because she's really into after-market Harley parts. Once again, quality and quantity are not the same, and I hear too many instances where people try to use one to explain the other, and it's just not the case. Differences in volume are not the same as differences in value. Then I gave you some examples that I see all the time in AdWords where people are like, "Well, there's more business during the hours of 10:00 to 2:00, so I'm going to increase my bids." That's not reasonable. It's not what that tool is for. If this all sounds a little vague, here's an analogy. I'd rather have one good craft beer than an entire case of Natty Ice, and that's what we're talking about here. The volume and the quality are completely different variables.



The next thing I'm going to go after is page speed. Especially for the past three to five years, this has become dogmatic, like you have to work on making your site faster. It seems more reasonable. This one, I was going, "This is probably correct," right? Google's even using it as a ranking factor. That alone means that you should probably put some weight into it, but let's see how it correlates with other metrics. You'll see this summarized in a bunch of blog articles and infographics and things, and I love Kissmetrics blog. They're excellent. They're one of the best out there in my opinion, but they're guilty of this kind of nonsense, where what falls in the graph of what users thought. What people say and what people do are not at all connected. If you look at any of the research done on the correlation between survey responses and actual behavior, this is not going to tell you anything useful. Love Kissmetrics though I do, that's not really great.



Here's the average page load versus conversion rate for the sample of sites that I was working from. There is a correlation, in fact the R-squared is 0.41, which in very rough terms means that I can explain about 40% of the differences in conversion rate across the sites using page speed alone. That's a massively powerful correlation, so you've got the slow and low converting sites on the left, and a fast and very high converting site on the right. Page speed confirmed, does matter, but there's a sub-assumption here that people have been harping, especially in the past 12 months to two years maybe, that this is even more true for mobile. It's just not. It's just not. There's no correlation here. It only explains 7% of the variation in conversion rate when I narrow it just to the mobile segment. What's going on here might be that the expectation is lower. People know that the phone experience is slower, so they're more patient. I'm not saying they'll be patient tomorrow, but today they still are.



You should definitely work on your page speed. You should definitely work on mobile, because it is a growing segment, but just because you see it on the Internet doesn't make it true. That concludes my sort of survey of all of this data. It's sort of stream of consciousness because that's how I do analysis, but I find these little gems along the way. Check your relative channel contributions. If your Venn Diagram doesn't look vaguely like the standard ones I showed you, understand why. It might not be a bad thing, but you should think about it and understand why it's different. New visitors are important, but multi-session shoppers pay the bills. Regardless of age, device, or gender, quantity does not equate quality, so volume and value are not necessarily correlated, and mobile shoppers are more patient than you think, caveat, for now. Remember, when you're reading things, including on my beloved Kissmetrics blog, that there are three kinds of lies: lies, damned lies, and statistics. Thank you.

What Are Some AdWords Growth Strategies?

Hey Roy, we have a small business we are looking to grow and are looking for some AdWords growth strategies.  We'd love to get to the point where we are spending $100k or so in ad spend (and making money).  We can afford a ROAS of 5.6.  From what you can see of our account, what are your recommendations? Thanks for any advice you can offer up.


Hey Artie, that's actually an excellent exercise--one that I was putting in some time on recently, which you can read through here.

If we're talking about a 5.6 ROAS or 18% COS (hence the making money caveat), then that would require roughly $556,000 per month in topline revenue.  To start with, you only have 2,450 products currently active in Google Shopping. While the impression share (for the past 30 days) is only about 24%, that means that even with an infinite budget, we can only realistically expect to get 2-3x the impressions from the market for those products--nowhere near the 25x increase you're talking about (going from $4k to $100k). I'd say you need to expand your catalog by at least 10x, spreading that net wide enough to capture that many eyeballs (if you'll forgive the mixed metaphor).

If you had ten times as many products, let's assume you could get 15x as much traffic, as the traffic compounds as you build brand recognition, retarget that audience, and continue to tune the site's conversion. If we also get a 25% increase in conversion rate during that time, that gets us to almost 19x the conversion count--a good chunk of the way to the 25x target.

One client I have experience with had 40k products, of which about half are parts which are drop shipped, so the meat of the catalog was probably 25k products. 85% of the company's orders are fulfilled through their own warehouses and inventory. About 100 products made up 50% of the revenue and 800 products made up 85%. If even one of those top products runs into trouble (some jerk blowing them out on Amazon, or the like), it is a big problem for the company.  Over the last couple of years, they've worked to continuously explore new products, manufacturers, brands, and categories. Some products would over-mature, the price would drop out, and they'd need to replace it with something new--it was and still is a full time job for two people.

They've managed to spread out revenue to 500 skus making up 50%, and 1,800 skus making up 85%. That's still a lot less than the 25k products in their core catalog, but without those 25k, they would never have found or had the 1800 that really mattered. Also, the long-tail became healthier and healthier as they started to stock anything in that parts catalog that hit some traffic and sales threshold, which also boosted margins.

As I mention in the article, you then need to present the new products as effectively as possible. This includes product photography, but also supporting content (check out some of the videos on this site, for example), and conversion rate optimization. The great thing about CRO is that everything depends on conversion rates, so even a small change there gives you the freedom to do all kinds of things elsewhere in your funnel. For example, if you can amplify your conversion rates by a little, it actually frees you up to consider lowering prices (which has a dollar-for-dollar greater impact on sales than a larger marketing budget, for example), or other strategies. Sites live and die by conversion rates. But there are lots of tools out there that make that easier and easier to work on--but it does take a lot of time and attention. That's part of what I presented on at eTail last year, actually.

But the short answer is that the first step in that direction is to focus on sourcing and merchandising, as you need a lot more products to offer in order to have the draw for an audience that size.

Thanks for the question Artie.  Drop me a line if you have any other questions.  I love to talk shop.

Up and to the right,


Diagnosing Impression Share in Google Shopping Campaigns

Using Google's Auction Insight Report to Understand Impression Share Relative to Competitors

Roy, at first glance our impression share in our Shopping Campaign is very low (~15-18%) compared to our other campaigns that are much higher, and I’ve always assumed this was a huge opportunity for growth if we could get it right.  If I view the auction insight report it shows me that most of our direct competitors are lower. I have a feeling that there is some component of this that I’m missing and you could really only compare apples to apples if a competitor was bidding on the EXACT same brands and/or keywords.  I was wondering what you would make of this?  Basically is this typical or are we really missing the boat on impression share?

My understanding is that Google is showing you their impressions share across the same eligible impressions that you are being compared against. So, for example, a competitor might have an overall share of 30% for all of their eligible impressions, but only 17.24% of the impressions where you overlap with them.

But you're right to interpret impression share as opportunity. I also use it to detect match rate issues with product feeds. If you had a formatting issue with your MPNs, for example, then Google would report very high impression shares, as no one was competing with you for your typos. A low impression share means that you are matching against the products correctly, and that there are shoppers you're not meeting.

However, simply increasing your bids to try to reach them is a dangerous game. Rather, I focus on shifting money from areas of relatively low performance to those of higher performance. This is done through regularly updating the taxonomy to adapt to available performance data. For example, if a product is doing significantly better than its siblings in a Product Group, then it's subsidizing the rest of them, and once you isolate it, then it can bid much higher, and the rest of the group's bids shift down to compensate. Likewise, if an outlier in a group is underperforming, and soaking up costs, splitting it off frees up the rest of the group to fly a little higher. However, splitting them all out individually drives the sample sizes into the ground, and you end up guessing on almost all of them--that's the genius of Google's current structure.

The long game, big picture strategy is to also work on conversion rates overall. Any time you can boost either AOV or Conversion Rate, it is equivalent to increasing your COS, without affecting margins--as far as the bids are concerned. AOV * COS target * Conversion Rate = target CPC, after all, so Conversion Rate is often the most powerful lever, if the most challenging to actually affect. If you guys aren't using a tool like Visual Website Optimizer, Optimizely, or similar, then you might consider doing so--while most tests are washes, there are plenty of small wins to be had. And Scot Wingo was right in his Bronto keynote--a small change in your mobile conversion rate can have a huge impact, as that shopper segment is always growing, and a small change to a low conversion rate is a larger shift, as a percent.

Have questions?  Drop me a line via the Contact form--I love to talk shop.

Up and to the right!

What is a Customer Actually Worth?


You know what's scary to hear when you're discussing advertising costs?

"It's okay, we'll make it up on the LTV."

And yet, I've heard it dozens of times, from fellow retailers in a wide array of verticals.  On the surface, it makes sense, though, doesn't it?  You can afford to spend a bit more up front, if you take into account the fact that your customers will come back and place subsequent orders, right?

Possibly.  But you'd better have data to back that up, and only a handful do.  For every replenishment-driven site with their retention dialed perfectly, there are fist-fulls of other retailers that don't know where to start to answer questions like "how much is an email sign up worth to us?"

There are a ton of ways to approach the task of determining Lifetime Value (LTV), but hopefully sharing my favorite puts you on a path toward creating yours.

You don't have ONE conversion rate.  You have several.

First, you need to toss out the idea of a universal "Conversion Rate".  You actually have a 1st Conversion Rate, a 2nd Conversion Rate, a...

  1. How many visitors become first-time customers?
  2. How many first-time customers place a second order within a year?
  3. How many two-time customers place a third order within another year?
  4. How many three-time... etc.

I use a year as my window, simply because I come from a seasonal retail background--your business may have a different cadence.  The likelihood of additional conversion do tend to increase as you go, assuming you provide a desirable customer experience.  That part is true!  But that likelihood starts to roll off at some point.

Let's say this is that roll-off, for the sake of demonstration:



These are based loosely on data I've seen from a number of commodity, non-replenishment retailers, so they're sufficient for our purposes.  You can pull the 1st Conversion Rate from Google Analytics.  The latter Rates, you could pull from looking for counts of distinct email addresses over time from your order history.  How many of your customers have two or more orders, relative to the total population?  How many have three or more, relative to two?And so on.

Now, you have the tools to actually estimate the Lifetime Value of a new customer!

Next, I'll show you how to use these values to actually determine a dollar value for your customers!

Calculate Forecasted Revenue Shares

Let's assume an Average Order Value (AOV) of $100.  Consider a customer who's only placed one order so far.  All things being equal, in our example, the 2nd Conversion Rate is 22%--there's a 22% chance they'll place a second order.

You've already collected $100 from them, on that first order, so you know you have that money.  Let's start there--LTV so far is $100!

With a 22% chance of a repeat order, since you're actually using this customer as a metaphor for a very large group of customers, you can treat that as being worth $22 (that's the chances of it happening, times the AOV).

You now have an LTV of $122!

How about that third order?  Well, you have to place two orders before you can place a third, so the odds of our one-time customer placing a third order is 8.58% (that's 22% for the 2nd times 39% for the third).  You've just added another $8.58 to the LTV.

You now have an LTV of $130.58!

You have to place a third order before a fourth, and a second before that, so the odds that our one-time customers will place a fourth order is 3.95% (.22*.39*.46).  That's another $3.95.

LTV is now $134.53!

And so on!  The incremental additions get smaller and smaller, and your iterations end up rolling off, like so:



It doesn't take that many iterations to get to the point where the additions are less than a dollar--and not many more before they're less than a penny.  In this example, it's safe to say that the LTV is about $135.

If you know your margins, you could then convert this revenue-based LTV into a margin-dollars version, or even a profit-dollars version.

I'm biased--I like making money on every sale, but I know that's not the only way to do business.  I knew one cosmetics retailer that knew exactly how much they could lose on the first sale, because their retention rates were so carefully measured and nurtured, and they used that as a weapon.  If our example company were to follow in those footsteps, they could theoretically set their Cost Per Acquisition (CPA) targets based on the $135 number, rather than the $100 one.

This can then empower you to answer another very common question!

How much is an email address worth to us?

Let's assume that the address is well-qualified, and is entirely expected to behave like other addresses you've collected before.  Buying lists, for example, throws this entire approach out the window.

To go from our LTV value of $135 to our new email address value, all we have to do is go back and plug in a relevant 1st Conversion Rate.  Multiplication is commutative, so it's fine that we're doing it at the end.  This Conversion Rate ought to be a measurement of how many new email addresses turn into first time customers within your timeframe (mine's a year, recall).

If that's 10%, then you know that you have to collect ten email addresses to produce one customer (on average, once again).  That means that each address would be worth $13.50 in LTV revenue, all else being equal.

If our hypothetical margins are 35%, then that's $4.73 in gross margin per email, so if my cost to collect those addresses is $5 or more, I'm setting myself up to lose money!

I hate losing money.  This approach to LTV helps me avoid it.  I hope it does the same for you!

Up and to the Right!

Behavioral Analysis for Sites of Any Size

DIY Behavioral Analysis for your E-commerce Site

This presentation was recorded on February 24th, 2016, at eTail West.  The full text transcript of the presentation follows the post.

All too often, when retailers hear about "Behavioral Analysis", they think of big-data-esque stuff.  Typical use cases are  for personalization, or advanced segmentation--and those are really cool things. However, I'm of the opinion that they're a distraction from some major wins that retailers can gain even without expensive software, an army of PhDs, or their own private data centers. Rather, I show you how I've learned critical things about my own site's shortcomings using tools like Google Analytics, if you can just think about the problem a little differently.

I discuss A/B split testing, heat maps, and other user behavior-driven systems I used to make iterative improvements over five years in my previous role. I hope you find it interesting!

Have questions?  Drop me a line via the Contact form--I love to talk shop.

Up and to the right!

Full Transcript Follows

This is what I've been looking forward to the whole time because, as he mentioned, I do like talking shop and this a unique type of situation. I spent more than five years at a company called PoolSupplyWorld and during that time ... I started as an engineer, and we were eventually acquired by Leslie’s Pool Supplies, which is the national brick-and-mortar chain, where I was vice president digital marketing for a couple of years before branching off.

What I'm going to do is, I'm going to take those five years of experience and show you how we learned what we learned. It's less about insights I gleaned in 2010. Hopefully a lot of this looks obvious in hindsight, but the bumps along the road are where I want to tell the story. If you glean anything from that, you can start to learn and in the same way we learned that allowed us to grow in the way we did.

To start with, behavioral analysis is sort of a buzzword. It means a couple of things in a couple of different contexts. The most common is using customer information to try and identify things about the value of the customer themselves. That's being segmentation, personalization basically, or you're doing some sort of modeling, or machine learning, or something like that. I'm not going to be talking about that, because it doesn't work on a small enough scale. If you are a large company like Amazon, you have infinite traffic, you have infinite data, you have infinite amounts of resources, you have an army of PhDs to do all of this, and as I think the panel asked, “who's a small company?” I was about the range you should talk about, 10-20 million, and we could barely hit statistical significance in about most of the things we did.

I'm going to show you our approach, which grew more technical over time, but at no point am I going to just be showing you statistics. When it comes down to it, if I want to try to beat Amazon at predictive modeling, behavior analysis kind of stuff that they're doing, they're going to win. I can't beat them, from a commodity retailer perspective, at their own game. If they have an army of PhDs, I'm going to take a completely different tack at it, and see if I can produce a customer experience that is sustainable, and gives a customer a reasonable and viable alternative to just buying everything on Amazon.

Instead of trying to learn something about the shoppers, I'm going to let the shoppers teach me about my own website and about my company. Instead of taking the entire group and making it statistically less and less relevant, I'm going to aggregate it all back up and say, "Okay, I'm going to apply Pareto’s Law, or the 80/20 principle, and say everybody's using the site, how do I make it awesome for the 80% group? I'm not going to try and focus too hard on the fringe groups, and especially when you're small, you have to do that, because that's how you grow. Then you have the luxury of worrying about fringe cases.

Effectively what we're doing is, we're taking data and turning it into money, and I love that line, it's on my LinkedIn profile. But it's incomplete. There's a middle set between data and money, and it's analysis and information. You turn data into information, and information is much easier to turn into money. So, this is a five year journey, mostly through mistakes that we made, that gets us from point A to point B.

I wanted to contextualize how long ago 2010 was, because it doesn't feel that long. I was looking for pop culture references and news headlines and things like that. So, I was looking at this, and thought… this doesn't do the thing at all. This feels like yesterday. Now, if this doesn't do it, this does. It may not look stylishly much different than Amazon does today, but if you look on the left, their primary focus was selling books. If Amazon was still just selling books, it would be a completely different ecosystem. This was the reality in 2010, when I started with PoolSupplyWorld.

There were a couple of things here that were starting to become industry standard practice. The navigation on the left was very, very common. They hadn't really gone all-in on the search yet, because their catalog hadn’t gotten so broad. You were already starting to see shoppers trained by Amazon to expect certain types of experiences.

The current team I'm working with, their site looked pretty modern for 2010. You see a couple of the things that were really, really popular and trendy at the time. They still have the side nav, which worked well. They have this slider, with things that the marketers and merchandisers decided the customer needed to see, and everybody had the same thing. Even Google shopping had that as a home page for a while. It looked like that, and when I started at PoolSupplyWorld, they were very proud of having just launched this. It's a website. I'll give them that. It was cutting edge for 2006. It was 2010, which is rough. I was, as an engineer, looking at this going, "Ugh. It's bad." If I read the text to you, which I promise I won't make any more slides at you, but Pool Supply wrote, "We carry all the well known brands in the Pool and Spa Supply Industry, as well as innovative new manufacturers. Our role is to," oh my gosh. Bruce Clay was responsible for that text. It was old ... It was terrible. The title of this thing was, "Pool Supply World. Your online pool and spa resource center and more, ellipsis" No.

I was looking at this. I'm new to this company. I'm brand new. I worked eCommerce a little bit before that, at what's now, and I was looking at this going, "Okay, what's the point of this huge banner section that everyone built?" Everybody is doing this slider. Is that a good thing? Does it do anything? The reality is that there is almost no chance that you, as a brand new customer who is coming to Pool Supply World, is looking for an Intelliflo pump. You're looking for a $1,400 dollar pump on a whim? Not likely. This is a waste of time. I was convinced of it. Because we had an eCommerce platform that was built from scratch, and because there were no real A/B testing solutions way back in the day, I built one myself. I baked it in. It allowed me to compare two different templates to each other. Simple. Dirt simple. We just dumped out the numbers I had to crunch them myself.

I took this, stripped out that entire bar, moved those four subcategories up, and went “ok, cool”. They're not good, but at least they're relevant. They're redundant to the navigation, but at least they're relevant. Let's go. However, being the arrogant engineer, I quickly hacked the skin. Awesome. I just built a new testing platform. Launch. Okay. Here are the templates. Good. Launch. Um ... Lunch! Went to lunch. Here's what I actually launched on a live website in the middle of our busy season and all of our goods. Oops.

I'm at lunch, halfway through a sandwich, about two miles from the office, and my phone is exploding. My boss, the CEO, is, "Roy, you've got to get back here. You broke the site. The home page is crap. It's a disaster. You've got to get back here right now!" I'm like, "Oh, okay." Vroom. Oh. Yeah, okay. Fix it. Am I fired? Okay. I'm not fired yet.

But I'm like, “look, we have the data!” Let's see. As it turns out, I was looking at the conversion rate, and everybody who saw the home page during that maybe 45 minutes to an hour, they were statistically significant sample sizes, because it was the middle of the busy season, everybody was hitting our site at that point, and I learned that none of the stuff on our home page mattered at all. The entire thing was useless. It wasn't just the stuff at the top, it was the stuff at the bottom, the middle, all bad. There was no difference between these two, and like I said, the math worked out: that was a waste of space. You'd think we'd stop doing that. We'll get there.

Now we have this A/B testing system online. All right. I'm not going to make that same mistake again. I'm going to test it a little bit before I launch it live. This was the product page. It's a little clunky. Simple but effective. It's like it was designed by a programmer, because I did. What we were testing was the in-stock status. Back in the day, we had our own warehouse, but we drop-shipped a lot of stuff, so the expectation of shipping times would be different based on whether it was coming from our warehouse, it would be coming lightning quick. Some very clever engineer, looks a lot like me, build the warehousing system. It was great. But, the drop shippers we a little Excel spreadsheet, PDF, via email and fax machine kind of world.

We wanted to emphasize our own inventory, which, it sounds very obvious. But, at the time, it was kind of a big deal that it's trying to distinguish these two things. The two messages were supposed to be, "In stock. Order today, to your door by," a date that involved the estimated shipping time for that particular product, etc. The alternative was, "Out of stock." We wanted to try out of stock, backorder, in stock but not bright green, things like that, because the secondary thing was, switch back, it wasn't really out of stock, it was just drop shipped. We needed to control that. Then we were just saying, "Ships in 2 to 12 business days," which is what the whole site used to say. Used to just say, "Oh, we do our own shipping," but nobody cares when you ship it, you care about when it arrives at your door, and we tested that already, so now we're playing with details. This is what actually launched and went to lunch. Subtle difference. You saw a big difference that didn't matter. This is the opposite. This is a tiny difference that no matter how reassuring that green text is, it says, "Don't convert." It was statistically significant within an hour, out of the season, because it was deployed across in every single product page. That was the other end of the spectrum.

Then this is like 2011-ish now. We're starting to refresh the brand. We have a new logo. We have a new designer on staff, finally. Everything is looking like Pinterest. We're going to look like Pinterest. As you can see, the navigation stuff is all very 2011. Tablets are going to take over the world. We should really be worried about tablets. We don't have the resources at this point to manage a mobile site by itself, so let's just make the desktop site not suck too badly on a tablet, then maybe it won't be so bad on the phone either. Phones were still a minority, at least for traffic.

We crammed all the navigation up until there's a nice big fat-finger-friendly buttons at the top. Put these, at least thematically and seasonally relevant things on the home page. This is when there's starting to be more tools available, so we're playing with Visual Website Optimizer, and there's other tools that do similar things, and clickmaps. We don't need these tools to get this insight. I'll show you examples in Google analytics. 80% of everything I'm going to tell you, you can do with free tools nowadays. Using VWO we learned that nobody was clicking anywhere. I didn't need to break the home page to know we went backwards. We went back to the same problem we had before. None of this mattered. They were hitting the navigation and the search, right?

What do we do? What do we do to make this more relevant to shoppers? Okay, well, we know our catalog. We know what shoppers need better than they do. Shoot. Let's just pick the things that we know they should buy, slap those into the side bar like Amazon (evo was doing that before we were), and I'm sure it will be wildly successful.

There's actually two wild assumptions here that are wrong. One, we know what's best for the customer, and two, I was convinced that pictures were clicked on. Nope. The content of the home page continues to be utterly useless. Our most trafficked page is doing nothing for us. There's a little bit of activity on the side bar there, because these categories do matter, and since we buried everything into that allegedly tablet friendly navigation at the top, we were surfacing something, right. Okay. There's a incremental progress. This is a step in the right direction. But, you can still do the same kind of thing. I promised a Google analytics reference. This is the same information, but several years later. This is last week at evo, and green means not that many clicks, and red means lots of clicks. A feature and major sections here that are things that I would love if they explored and bought things from, but realistically that was not where they clicked. They want the snowboarding stuff, and you can see the same sort of thing here. There are perfectly good merchandising reasons to have the other stuff, but the UX challenge is identical to what it was back in 2011.

Back to PoolSupplyWorld. If only there was some way to know what they're actually looking for when they landed. If maybe there was some sort of eye tracking technology I could use to tell me what's going on. Maybe, if only there was some way ... Maybe a service like a live chat, push it on them or something. If I know what they're looking for, then I can merchandise to them.

One day it was like, "Oh, wait. They've been telling me all along." They're like bounced user plus plus. A bounce goes away and tells you nothing. A person who hits your search basically is telling you the same thing. You failed to tell me what I needed to find out. I failed to learn the answer to my question. I failed to find the product I actually wanted. But then they go ahead and they tell you what it was they wanted. They're practically doing your QA for you.

You could do this with Google analytics. Didn't used to be able to back then, but you can now. It's just under behavior, under site search, Pages, then you select the Start Page. In our case, we’re talking about the home page, but it's even more relevant for category views and things like that. You can get the list of queries that are being submitted. We took that information, and this is August or September period, around 2012. The winter covers, safety covers, and leaf nets are perfect products for that time of year, because people are starting to think about closing their pool. As you can see, we then used that same insight to do some of the merchandising decisions, but the cover pumps in the middle is an interesting type, because that type of product wasn't on my radar.

At this point, I was spending mostly marketing. I was still doing programming, but I thought I knew the catalog pretty well, because I had all the data. I had all the data all the time, but it would get lost in the wash that are five or ten orders a week, which was nothing more in the season compared to other products on the site, but they converted really well. Not a lot of traffic, but they produced a small amount of healthy orders. The conversion rate was spectacular. I just wasn't drawing enough interest in it. That cover pump problem was surfaced by looking at the search queries that were being hit on the home page and then not converting.

I was looking through these e-words, I'm going, "Cover pumps? Sump pumps? What are these for?" When you put a safety cover out over a pool in the Northeast, snow and rain and stuff accumulate on top of it, and if they're a solid cover, it will start to sag, so you have to have a cover pump to keep the water off of the cover. They're also used for other winterizing things, but it was something that wasn't even on my radar, because I was looking at the top hundred skus, all the time. This was much better. We felt confident about it. We brought buttons back, because I learned my lesson. Finally, we're starting to see some action that looks like a competently designed web page. We have, navigation is actually getting hit in roughly the right order, buttons are getting clicked. There's still action on the search bar. I think it's universally going to be true on a home page, because it's the least relevant shopping page in the entire site, because it's the least specific. Then the nice thing is that the pools category is a little less, just destroyed with clicks.

This is from about a month ago. Long after I left. I left in September. You can see they've updated pool cleaners, pool cleaners, pool heaters, major equipment. This time of year, pools are thinking about opening up, they're on the other end of the season, and they're starting to replace equipment that failed or was damaged in the winter, things like that. I would bet you dollars to doughnuts, including if they're on the wall [eTail had a wall of doughnuts just before this--you’ll have to attend to see what that means], that in another month or so you'll see opening kits and chemicals, and things like that. The navigation is much better, and it's basically the same story, but it's been iterated upon again and again.

However, once you get to a certain point, you're looking at these keywords, and you're going to start seeing this happening. This is from evo’s home page. The queries that make this up are very diverse. The largest single keyword makes up less than 1% of searches. What do you do with this? There's two ways to look at this. First, this could mean that you've succeeded. There's no big glaring gaps in what your home page is trying to accomplish. The other thing you can do with it, is you can start aggregating synonyms. You do this in Excel. There's tricks you can do there. We were using database, but effectively, you can take the keywords out of Google analytics. In my case, I was taking things like pool cleaner, and pool cleaners, and pool vacuum, which are effectively synonymous depending on where in the country you are, and I treated those as a single term. If you start to aggregate those, it's not ... You don't have to build a semantic engine that can do crazy language interpretation AI stuff. You can just say, "Well, that one's really common. Add that to the list. That one's really common. Add that to the list," and in just an hour or two, you'll have probably have 90% of use cases. You'll have a much better list.

The other thing you can do is, if your home page looks like this, go to the category page. The category pages are going to have more specific searches, because the conversation is different. This is the accessories page on evo. Now, with context of the conversation in accessories. People aren't looking as often for every single product or every single category in the entire website. It becomes narrower and you can focus on this. When you do that synonym analysis on this kind of thing, it gets even better.

What's the next step? Product pages. This is the product page. It's a fairly heavily hit product page. Those first two keywords made up a sixth of all of the searches from there. Those two keywords likely represent things that I can probably add, either if it's product related, I could do a frequently bought together kind of plan. If they are information queries, then you can probably add content to that page, and make that page stickier. They're telling you what they're looking for.

There are other types of data. You can search behavior and click behavior, as in behaviors, but there's other ways to do it. Product reviews. Most of us, if we're modern day retailers, are collecting product reviews, either doing in house, like I was, or using a vendor, but you're collecting these things. You think, "Okay, cool. My customer is telling me about my products, and the qualities of those products." Almost. Not exactly what they're telling you, actually.

These two things are robotic pool cleaners. They do the same job. The one on the left is a Polaris. It's a huge brand within that tiny niche. It's big. This thing is a work of art. Looks like an Aston Martin for a reason. It is hyper intelligent. It can do crazy patterns on your pool, no matter what shape your pool is, to make sure it hits everywhere. It has a remote control. It'll actually drive up out of the water and you can pick it up, so it feels lighter than it is if you're dragging it out of the pool. It's magnificent.

On the right, is a Smartpool. This Smartpool is anything but. It weighs about four pounds. It's just this plastic toastery thing. When you get too much sand and silt into it, into the bag that holds that stuff, the bag just stops letting water through. Then the thing just seizes and dies. Their solution was to put grommets in the bag, to let the water through. All the sand and silt goes with it, and it basically just ends up finger painting in mud at the bottom of your pool.

But Polaris had a lower average rating. What? What am I looking at? It dawned on me. That's the difference. If I'm going to spend $1,400 dollars on a robot that cleans my pool, I want that thing to get out of the pool and make me latte afterwards. If I spend $350 dollars, I'm glad that it picked up that leaf. The expectation is different. The context is different. It's not just a measurement of the direct quality of it. Now all of the sudden I had insight about my own products. My customers have told me something I didn’t really think of. I knew it was more expensive, but to the point where it would affect how you're rating the product overall was mind-blowing. It's a really good product, and it's rated three stars, versus a three-and-a-half for the Smartpool. Ouch.

I'm saying, "Okay, how do we do our messaging?" Knowing that we know this thing about how these are perceived, we can push bargain messaging against the Smartpool. Not just because it's cheap, but because people expect things from that cheap product that it can deliver, and we're setting it up for success. Likewise, with the Polaris, we can play to its strengths in terms of it being a feature play.

Further, buy in behavior, and this also applies related to acting on behavior, wish lists are spectacular for this kind of thing. Anything where people are self-identifying as being interested in related to this specific product. But, I like buying. Short of calling me and shouting at me on the phone, it's the loudest thing we can hear, I get from customers, because as soon as they give me their money, there is no larger endorsement for whatever path they took to get there. Money is the loudest thing.

The buying behavior is useful in some ways, and this is the most common. Everybody's seen this. This is straight from Amazon. Frequently bought together. This phone case and this slightly more different phone case. What's fun about this that you don't really see until you sit back and think about it, is that, well, if this product is related to this product, and you click to the next product, that's related to other products, and it's related to other products. All of those purchase behaviors overlap, and you can actually build a map of your entire catalog, as if it's an ecosystem by recursively doing that same kind of analysis. This ... It's a little dark, I apologize for that, but it's a diagram of an entire month of sales across my entire catalog. There's a bunch of little dots here on the right, and they're all tied to that big orange dot. What they are are specialty chemicals that are not needed for every single pool, but every single pool requires chlorine. That's like the bread and eggs in a pool business. Then the big swath of larger dots down the middle are major equipment purchases. The large dot represent large volume, large AOV, therefore higher revenue.

Further out is another constellation, effectively, of spa products, that are totally unrelated to the specialty pool products. You see parts and things like that as well on this. If you render this several times, no matter what other filters I put in place, because of the recursion, it's kind of like the Kevin Bacon thing. You end up with the same constellation showing up every time, and they’re semantic groups. The importance of that is that this is like a magic robotic merchandising machine. If a merchandiser were to look at a bunch of products, and they could tell me that these are all products in this similar use case, these are products that apply to these use cases, and if I ask them to list off all the major groups, they do a good-ish job. However, there are clusters in here that reveal different affinities than I would have thought of. Sure, you see brand affinities in here, if I were to label the brands and stuff like that, and that's obvious, but what I didn't expect to see were there were affinities between products based on the pool size, and other variables I wasn't really thinking of.

Here's another way of rendering it. We have a bunch of big dots up above. This is actually narrowed to Florida. I use Florida as like a root node. I drew all of the products that were the top 500 products that were really popular in Florida, and then I drew all of the products that were associated with those products, as in bought by the same customer, I was using email address. I can't do frequently bought together, because my average line items per order is 1.1, but you have a pool last year, you have a pool this year. They come back. Our repeat customer rate is pretty good, so we do have all the products you bought over time, and can do the same kind of thing.

What was interesting here is this island down here at the bottom where they're all connected to the white dot, but they're not connected to anything else. What automatically formed this kind of visualization, because these are orphans. People buy that and that's the only thing they buy. Isn't that sad? Some of these are pretty big dots. You can't tell me that I can't figure out that K0400 is a suction-side pool cleaner. I know I can sell them another product that goes with that. I know the pool that goes to. I know everything about that. As a marketer, I can go, "Oh, man. I'm sorry Kreepy Krauly Kruiser. I'll find you a friend!" And I can go and get a better marketing angle for those products, so I'm under utilizing their traffic.

However, sometimes frequently bought together logic and go a bit astray. This is a harmless example of that. Ray Ban sunglasses and Furbies. I don't get it, but it happens. If you are a fashion retailer, and you sell ... If you recommend two products that don't go together, and I buy them anyway because I'm illiterate on the topic, I just look like a fool. If you buy this pool filter, and this valve that looks very much like the correct valve, they will literally explode.

A customer did this. We were very sorry. We tried to help them. However, their sales information was now in my product recommendation engine, that it built on top of all those lovely pretty graphs you saw, and now I was recommending those products together. The second time these products exploded, we finally started to go, "Oh, wait. Those are the same SKU's. Something's wrong here," and by the time the third one exploded and we replaced three pads of equipment in three different back yards, we finally said, "Wait. I think we need a incompatibility matrix," because it's not like car parts, where we say, "Okay, this goes to just that." It's basically, "This could go to anything, except that," and then manage to bake that back in. Customers were telling us things. Some of them were wrong. We needed to be able to tell the difference, or things would explode.

This one is a little less violent. Contact request source. I haven't figured out a better key phrase for this, but effectively, imagine something goes to your contact us page. If you think bouncing or searching on a page as a decree of your failure, contact us is among the worst. Many reasons to contact your customer service agents are perfectly valid, and our customer care team was amazing. Our sales team was amazing. If you did have to contact PoolSupplyWorld, you were taken care of. It was never about the quality of the experience, but it was a condemnation of the quality of the page. At first, we're just like, "What are the topics? Is there any data out of the call center?" Call center doesn't have time for that. We were, seasonal as crazy. If they're getting those questions, they're already working overtime out of their minds. They don't need to hear from some engineer that gets to go home as early as 7:30. It's not what they want to hear.

I'm like, "How do I reduce their pain, and increase the stickiness of the page and the site rather?" What I realized was that you can use all of the sources of traffic coming into the contact page to inform where you should focus your content efforts. Imagine these very subtle metaphoric arrows are sources of traffic from where else on your site, so it's almost everywhere someone's looked at the contact page. Wouldn't you want to know what the big blue arrow is? Because you could start there.

For PoolSupplyWorld, the culprit were very technical products, which was a very two sided story. If we simplify it so we can actually (and now, don’t actually try to read this, it’s wildly irrelevant, but it's a good demonstration)... If we simplify this too far, then we reduce the confidence in a shopper that this is the right product to do what they want. If we overwhelm them with it, we reduce the confidence to the shopper, and they don’t buy--both end up contacting us. The trick was trying to figure out a way to take a very complicated product and maximize the confidence and identify situations where they should contact a professional on our team, and walk them through it. Many people would be fine, but we scared them with this, and they wanted to contact us form.

The other end of the extreme is this. This is half of the catalog--we had a large parts catalog. There's not much to say about this gasket, except for, "Well, what size is it? Does it replace the gasket I already have?" You don't need to say much. There's not much there to buy, but we have really failed, and when we started looking at these products sorted theme sources in aggregate, we saw categories that were guilty. We started to put our content development efforts towards those categories, and that started to increase our conversion rate across those categories, and a number of distinct SKU's that were selling were increased. We fattened up our long tail within our catalog, because we knew where we were attracting traffic but not getting the job done.

This is something we can do with Google Analytics. It's under behavior, all pages, and then previous page path. Previous Page Path is the secondary dimension, and is the most like under-utilized, unsung hero of the Google Analytics, because anything that you're looking at in your account that is an indicator that something went wrong or they needed trouble, or they couldn't find what they were looking for, you can use this and look back, where did they come from? Everybody is focused on exit pages. Where did they go after this? Well, that's important too, but people forget we can tell where they came from to begin with, and this is an amazing thing.

This is evo’s. You can see most of this is not very interesting. In the first ten, you see there's pages that aren't a lot of traffic, so it's going to biased that way. When you dig a little deeper and further in there, we start to see pages that are kind of, it's apparent what's wrong here. Throughout the entire section of these links: “we have free stickers”--it’s a brilliant email list generation system, but we blew the wheels off of it, and haven’t gotten to fix it yet. I ran across this and went, "Oh." Even on this page I'm telling you I'm going to give you a free sticker. The arrow literally points at me telling you to pound sand. This is five years after the first slide I showed you. It's the same thing. This iterative of improvement, it's low tech, it's just consistently applying the same methodology again and again to make the site less and less terrible over time, and you can grow from very small to a lot less small pretty quickly.

If you remember anything from this talk, and I know it's the last talk of the day, so thank you for coming:

  • Click behavior guides merchandising. If you don't have a UX team, and I very rarely had one, just look at what people are doing and it will become apparent.
  • Search behavior data is like being psychic. You can read the minds of the customer. They are frustrated with you for not having given them what they wanted to begin with. You can get in there.
  • Previous page path is wildly under utilized. It's in there, and nobody really talks about it, but it's awesome.
  • Customers are play testing your site all the time. That's a sort of gaming metaphor, for video games. They are in there. They are trying out the experience that you designed for them to have, and what they're doing and the action to your attempts to shape that experience, will tell you whether or not you succeeded. Sometimes you're using conversion rate, but you don't always have to use just that metric and you may never hear me say that again, because dollars ... I can't buy a beer with engagement, but there are leading indicators that you've used in here where you make decisions about your content strategy, and where you're investing your resources, that will do more than just focusing myopically on conversion rate optimization and where that can get you.

Audience Question: What percentage of your tests succeeded?

Roy Steves: What percentage of my tests succeeded? 30%? We were of the opinion that if every test succeeded we weren't really testing. That took some ... That caused some heartburn with the ownership originally, but we would try wildly crazy things. Intentionally try to shake things up. After that homepage test, we basically discovered that most things don't matter. For a long time, the color of the button didn’t matter, the size of the button didn’t matter, the… all this stuff that you would think, "Oh, I have an A/B testing system. I know. I'll try that," it didn't matter. Very few things actually mattered. We started doing really drastic things. That gave some people a little bit of anxiety, but the more drastic the difference between your test and your control, the faster the test goes, because the divergence of behavior increases the statistical significance of the two groups. We were willing to do almost anything, as long as it wasn't profanity-ridden or something like that. As long as it was appropriate to the context, we'd do anything.

We tried a huge parallax pump-selling work of art, and it never worked. Then we went back and said, "Why don't we just double the size of the buy button? That worked!" We were just willing trying anything, because if we weren't failing often enough, we weren't learning anything, and as you saw, failing is the best way that I process work and my team worked as well at the time, so we weren't afraid of it. It was definitely way less than 50% success rate.

The other thing about that testing process, and this is a question that I liked that the panel got, was what happens if a test comes up insignificant? Our approach to that was, most of the tests were coming out of the engineering team or the merchandising team, so it tied us to the design. The designer says, "This is pretty, and this is a good experience," and we're like, "Yeah, but we've got to try this, because it makes them buy the thing." If we’re wrong, and it just was a wash, or the designer says, "We should update this horrifying thing that Roy designed, to this prettier thing," and it's still a wash, and you know it's tie goes to the designer. When in doubt, we iteratively get less hideous, and it made us much better over time.

Audience Question: Is your experience primarily B2C or B2B?

Roy Steves: 95% B2C. In B2B, it's a challenge of numbers, for sure. If you're playing a numbers game like that and you're doing lead-gen as a first step, as opposed to B2C, merchandising, then you end up doing the same math, but you have an additional conversion rate added in there. You have your visitor to a lead, your lead to a conversation, conversation to an end. Things like that. It ends up being the same math, it's just by multiple times through, and you're doing the same kind of thing. You're looking for, what are they trying to find that they're not finding?

A great example in the SaaS space for example is if you don't have a pricing description, or worse, and friends of mine are guilty of this, you have a pricing tab that takes you to a page that doesn't describe pricing at all. It's the same kind of hard dead end. You don't have the advantages necessarily of running the same search bar that lets people tell you what's wrong with what you’ve done, but you can intuit it based on bouncing. They came to the home page, they clicked the pricing, and they're gone. It's the same kind of approach to this, just a different context.

Audience Question: On the $1,500 dollar cleaner, did you see benefit from video?

Roy Steves: Yeah. That was something that, with that kind of price point, the manufacturer was also producing more content as well. We can do a great job at getting that content to change the position of product. We tried different media displayers, where we include those videos and things like that, and we embed the video above the product description, and those might speak to the quality of the video that's coming out of those sources, including our own internal videos, when we first started doing video, hurt conversion. It was P99, bad idea. I'm not saying video is bad. Video is excellent. You can make a lot of money with video, but if you're using videos produced by a pool cleaner manufacturer, even a very expensive one, it might not be as good as a car manufacturer or a shoe manufacturer or someone. It's a slow moving industry. When we started doing our own, we spent a solid year trying really terrible videos before we got better, and started getting some traction.

Audience Question: Other than Contact Us, what other pages yielded insights?

Roy Steves: The next thing after the Contact Us one was sort of the inverse problem, and that was, we were drawing a lot of organic traffic to our blog. Our blog was not doing a great job of turning those into shoppers. We were using the search activity on the blog as the corollary, because people would land on, how do I find the right size pool pump. That's an important question, we answered it, and then they'd just go away, or they'd search for what's the best pool pump. As soon as you saw, "How do I size it?" Followed by “which one is best?”, this is the difference between a research question and a buying question. Then we were starting to try and pump that stuff back in.

The other type of activity we were keeping an eye out for was live chat, so we had a live chat solution in place, and we had staff on it year-round, so I could look for the originating page for those live chat sessions as well, and it was much the same story.

I love talking shop. You have all probably picked up on that. If you have any more questions at all, feel free to grab me and I'd be happy to talk about any of this or eCommerce in general. I really appreciate your attention.