The causation-correlation puzzle. How to piece together the jigsaw

It rained. The ground is wet. Therefore, the rain caused the ground to become wet.

It’s human nature to see two things that have changed at the same time and then assume that one of those things caused the other thing to occur. It is one of the core skills that we begin learning as babies and develop over time.

In the example above, because it rained, and the ground is wet, our brains make the link that it was the rain that caused the wet ground. It would be ridiculous to assume that it was the wet ground that caused it to rain. Why? Because our observations and experience in this world have taught us the causal effect that rain has on the ground.

Unfortunately, our brains aren’t always correct in creating a causal link. While the above example is straightforward, in the world of the wide web it’s not always this obvious. When we make changes to our websites or apps, we can sometimes fall into the trap of building a causal bridge that doesn’t exist.

I have seen these rickety bridges time and again, from me included, which tells me that it’s one of those ‘gotchas’ that we need to keep reminding ourselves about.

Consider this article a reminder to stay vigilant!

First, let’s break down what is meant by the terms causation and correlation.

Causation

If A occurs, then it will cause B to occur

Real world example:

A = press the light switch

B = the light turns on

If I press the light switch, then the light will turn on. A has caused B to happen.

Correlation

A and B are related, but if A occurs, then it does not necessarily cause B to occur

Real world example:

A = buy a 10 pack of train tickets

B = catch a train

If I buy a 10 pack of train tickets, this does not cause me to catch a train. I was always going to catch a train and buying the tickets is just a part of that process. If I were to buy more packs of tickets, it wouldn’t cause me to catch a train more. The two actions are related though, i.e. there is a correlation and potentially other causes of both events.

Finding correlations between variables is a straightforward statistical task. Finding causal links can be a lot more complicated and requires human judgement.

Now that we’ve got the basics down, let’s look at a typical real-world case study that we might come across in our work.

I’ve mastered wet ground. Let’s move on to the web round.

Let’s imagine we run an eCommerce store selling hipster shoes. We’ve analysed the data in Google Analytics and noticed a correlation between 2 variables.

Variable A = user toggles the colour of the shoe

Variable B = user purchases the shoe

We know that 5 of every 100 people purchase a shoe. That is, there is a purchase rate of 5%. We can also see that when someone toggles the shoe colour, this purchase rate goes up to 10%, i.e. 10 in every 100 people.

So, what do you think?

Is there a cause and effect relationship here between the toggle and the purchase?

Yes? No?

Maybe.

1. Does toggling the colour of the shoe (A) cause users to find the product more attractive and therefore purchase (B)?
2. Are users with a pre-existing inclination to purchase (B) more likely to toggle the colour of the shoe (A)?
3. Is there some other factor that is causing more users to both toggle (A) and purchase (B)?

We need to run some experiments to find out

A quick, low effort test could be to conduct customer interviews. Let’s find 10 people that are representative of the users of our website. We can then ask them what factors influence their decision to purchase shoes online. After that, we can put some prototypes in front of them with and without the toggle and see how they interact with them.

Even though our interviews told us that customers loved this feature, we still need to conduct further research. While customer interviews are fast and simple, they are also less definitive. We might call this a low effort, low confidence experiment. It might help guide us in the right direction, but it is unlikely to provide enough evidence on its own for us to make a definitive decision.

We could also run further AB tests to isolate the effect.

To test the first hypothesis, that toggling the colour effects purchase, we would want to encourage more people to toggle (the cause). If we succeed in encouraging more people to toggle the colour and the purchase rate of the people toggling remains stable, then it will support our hypothesis.

Great. So now we just need to find a way to encourage more people to interact with the toggle.

But let’s say our experiment showed the opposite result. That is, the number of toggles went up, but the purchase rate decreased. This result would disagree with our hypothesis and suggest that the toggle doesn’t itself cause more people to purchase.

If the toggle isn’t increasing the conversion rate, is it just a waste of space?

Not necessarily.

AB testing can help guide us to make more informed product decisions, but it shouldn’t be used as the sole data point. Our customer interviews told us that people loved this feature, so even though this experiment didn’t prove the toggle’s value, it doesn’t mean that the feature has no value at all.

Good product design is all about taking in as many customer data inputs as helps us to make better product choices.

If experimentation was easy, it wouldn’t be so much fun.

This shoe store case study highlights some of the complexities in AB testing and finding causal links between correlated variables.

Next time you’re analysing your Google Analytics user behaviour data, pause before creating an automatic cause and effect link. Try and analyse the two events as objectively as possible, consider the assumptions you’ve made, and plan what subsequent experiments you could run to test those assumptions.

• Have you used experimentation to find a causal link? How did it go?
• Do you have any tips for analysing user behaviour?