A/B Testing Retargeting with Google Tag Manager

Why A/B Test Retargeting?

Measuring the value of retargeting display ads can be tricky.  Not everyone who clicks on the ads and subsequently converts on your site is truly 100% added revenue.  Many of those customers may have continued on to a purchase through another channel in time.  And even if the retargeting ad was responsible for closing the sale, what about the contribution of the higher funnel channels that brought the shopper to your website in the first place?  How much of the credit does retargeting deserve for bringing a browsing shopper back and getting them to add the item to their cart?  How much credit if it brings a cart-abandoner back to complete checkout?

The relationships between the ad clicks and the likelihood of conversion are a little too complex to be well represented by a touch-based attribution model (not even to mention the influence from non-click impression interactions).  But lucky for us, when it comes to retargeting, there is a much better option than attribution models. As it turns out, the mechanics of retargeting lends itself perfectly to an experimental test design, or as it's commonly referred to in e-commerce, an A/B test.

The key for any A/B test is that we have some population of test subjects (typically users or sessions in e-commerce) that we can randomly assign to a number of different treatments.  In the case of retargeting, that population is all of the new users to your website.  And because retargeting is essentially a treatment applied to users once they are cookied by the retargeting tag on your site, the only requirement for a clean A/B test is that we can randomly assign who gets cookied and that we can track the outcomes for those users by which assignment they were given.  And with the help of Google Tag Manager and Google Analytics, we can do just that.

Random Cookie Splitting with Google Tag Manager

(Note: To execute this you will need your retargeting tag to be implemented through Google Tag Manager.  For this guide, I will use an example GTM set-up with Criteo's OneTag installed.  But the exact same functionality can be achieved with any retargeting tag as long as it's executed by GTM.)

There are a few separate pieces that are needed to get this functionality out of GTM which I will outline here, but the simple overview is that you want GTM to randomly assign a treatment group variable to every new user that enters your site, track that group assignment (control or treatment) over repeat visits with a cookie, and only allow your retargeting tag to fire when a user has the treatment group assignment.

So, let's get down to brass tacks.  Here are the pieces you will need to add to your active Google Tag Manager instance:


Variables

You will need to create two new user-defined variables:

    variant
        Variable type: Data Layer Variable
        Data Layer Variable Name: variant

    variationCookie
        Variable type: 1st Party Cookie
        Cookie Name: variantCookie


Triggers (VARIATION FUNCTIONALITY)

That variable and cookie will store the group assignment for users.  Now we need triggers to know when to assign those groups:

Dom Ready - Variant Cookie Not Set

dom_ready.PNG

variantSet

variantSet.PNG

Control Variation Set

control_variant_set.PNG

Triggers (variation specific retargeting triggers)

With the framework for our variant cookies set up, you now need to edit the triggers that your Criteo tags run off so that they only run for users in with the test variant cookie.

(Note: if any other tags are currently using those triggers, you'll want to create new copies of the triggers with these changes and assign the Criteo tags to these new triggers).

I'll demonstrate an example of how the Criteo triggers should look with the new condition added.  Here is what a modified Criteo transaction tag would look like:

Criteo Transaction

transaction.PNG

Obviously the exact names of your triggers may differ and the way you define the page types may differ for your site as well.  The idea here is you want to replicate your current retargeting triggers only with the added condition for "variantCookie equals 2".  For Criteo, you'll also want to add the same condition for the Criteo Basket, Criteo Homepage, Criteo Listing, and Criteo Product triggers (or whatever the equivalent triggers for those page types are named in your set up).


Tags (set variant tag)

This is where the magic happens (okay, I may be over-selling it a bit).  This is the custom HTML tag that will check for a variantCookie on any user who hits your site, and if they don't have one already, will randomly assign them one.

Variation Cookie Set (Tag type: Custom HTML Tag)

<script>
var variant;
var rn = Math.random();
if(rn >= 0 && rn < .5){
    //Control
    variant = "1"
}
else if(rn >= .5){
    //Variation
    variant = "2"
}

var d = new Date(); // Create cookie
d.setTime(d.getTime()+1000*60*60*24*730);
var expires = "expires="+d.toGMTString();
document.cookie = "variantCookie="+variant+"; "+expires+"; path=/";
if(variant!==undefined && variant!==null){
    dataLayer.push({
        'event':'variantSet',
        'variant':variant
    });
}
</script>

Triggering

Firing Triggers: Dom Ready - Variant Cookie Not Set


For the ambitious

(Note: The current split in the HTML tag above is set so that, using a random number between 0 and 1:

 [0 - 0.5) <- control

[0.5 - 1] <- variation

This achieves a 50/50 cookie split, but you can easily change those ranges if you want something else (like 90/10 split) or you could even add more 'if else' statements to create 3 or more variants (for a 33/33/33 split, etc.).  Obviously keep in mind that adding more variants means you would need to create additional triggers for those variants and so on.


Tag Modifications (use your new triggers for retargeting)

Now that you have variant specific triggers, you want your retargeting tags to be assigned to those triggers.

If you just edited your existing retargeting tags and didn't have to create new ones, you shouldn't need to do anything for this step.  But for those who had to create new versions of each trigger, make sure you switch your tags over to fire off of the newly created triggers.


Tags (GA EVENT)

Now we need a way to track these variation group assignments so we can compare the results.  To do this, we can just set up a custom dimension in Google Analytics:

Creating a Custom Dimension

In Google Analytics, go to the Admin menu.

Under 'Property' (making sure the right property is selected), near the bottom of that column of options you'll find 'Custom Definitions'.  Open that and select 'Custom Dimensions'.

From there you will see a list of your custom dimensions (if you have some already) and a red button with "+ NEW CUSTOM DIMENSION".  Click that and create your custom dimension like so:

cust_dim.PNG

You can name it whatever you want.  The name won't matter.  Just make sure Scope is set to 'User'.

After you save it, return to the list of custom dimensions and you should see it now in the list, along with an index number (make note of that number, it's what you'll use in GTM):

index_num.PNG

Now that you have a custom dimension you can use to help GA track these variation assignments, let's go back to GTM and create our final tag:

GA - Event - Variation Set

ga_event.PNG

the set-up is complete! hooray!

That's everything.  You can preview your new GTM configuration before publishing to make sure nothing is broken.  However, make note that you yourself will be sorted randomly into one of the two treatment groups once you preview the site, so if you are assigned to variant = 1, you will no longer see your retargeting tags firing in the GTM preview).

Because of this, the best way to verify that the cookie split is working correctly is to check the results in GA after a day or two and make sure you have a roughly even number of users in each variant.


Monitoring the results in google analytics

You can now set up a custom report in Google Analytics using the custom dimension you previously created.

In the menu on the left-hand side in GA, select 'Customization'.

Then select 'Custom Reports'.

Click the "+ NEW CUSTOM REPORT" button to create your new report.

Feel free to select the metrics of most interest for your site, but for a simple example, you could set up the report as such:

custom_report.PNG

And save.

Now, from the custom reports page, this report should be listed there as a quick way to compare the performance of the variation(s) and control.

But there's one last step we need to have the exact data we want...

create a user segment to filter out pre-test users

There is one last thing we need to do to ensure clean, uncompromised data.  We need to make sure we are only looking at users whose first visit  was after the implementation of our cookie splitting.  Because, for example, we don't want our data set to include users who may have been cookied before the test started and then were assigned to the control group after we launched the test.

To add this condition to our custom report, we need to create a segment only including users whose first visit was after the launch of the test.

From pretty much any page in GA (but you can use the custom reports page for example), you should be able to select the "+ Add Segment" button near the top-center of the page.

Then click the "+ NEW SEGMENT" button and create the following segment:

custom_segment.PNG

Save the new segment, and you're ready to go!

In your custom report, select the segment you've just created and you should now be able to start viewing the data you will use to evaluate the effect of retargeting on your users!

(Note: the one limitation here is that GA can only apply segments over a maximum of 90 days, so if your test is going to exceed that, you may need to create a second segment for new users starting 90 days after your test start date and just add the results from those two segments together).


That's all, folks.

Thanks for reading!  I'm Jaret, resident data scientist at StatBid, and I love to talk shop and help people tackle their e-commerce data challenges, so if you have any questions on this post (or testing in general), feel free to reach out to me at jaret@statbid.com

Happy testing!