Marketing Analytics Part 1: Market Basket Analysis
Fabian Kok & Danielle Paes Barretto
This first tutorial on Marketing Analytics opens a series of smaller tutorials introducing techniques that will help you (or your client) make better data-driven marketing decisions.
We start by introducing Market Basket Analysis (MBA) , a powerful tool used for product promotion and recommendation.
Time estimation: 7-10min
- Marketing Analytics in Python
- Market Basket Analysis
- Apriori Algorithm
Hello and welcome!
Within this blog we go through what Market Basket Analysis (MBA) is, why it’s useful, and in the most basic of terms what some important concepts are if you want to get started with it. This is our first blog in this new format, so feel free to give us feedback to more accurately fit to your needs (:
Market Basket Analysis
Market Basket Analysis (MBA) is a powerful tool that transforms customer transactions to simple rules that can be used for product promotion, recommendation, etc.
By applying it you can, for example, identify a simple rule of “if someone buys a croissant, they will also generally buy jam”. Using this information then, you could improve your sales by recommending jam to someone when they buy a croissant to further sales, or place a section of jam pots close to the croissant section in the store. Even further it could help improve inventory management, dictate which product bundles would sell well, and so on.
Basically MBA is incredibly useful for any businesses that sell a variety of products and/or services to customers.. which are quite a few of them.
Association Rules
MBA is based on association rules. Association rules show us items (being products or services) that, as its name suggests, are associated with each other. For example, if we find that buying croissants is associated with buying jam, then we state it as the following association rule:
{croissant} -> {jam}
and we read it as “If croissant then jam” in formal language, and “if someone gets a croissant, they also tend to get jam” in informal language (and “people be liking jam with their croissant” in the most informal language).
In common terminology, the relation between these two is defined as:
{antecedent} -> {consequent}
Now the previous example was with one antecedent (croissant), and one consequent (jam), for ease sake, but you can have multiples on either side:
{croissant, coffee} -> {jam}
{wine} -> {baguette, cheese}
As we are dealing with sets, there are no duplicates in the rules. So if people buy generally 2 croissants with 1 jam pot, the quantity is lost but the relation of “if croissant then jam” is learned.
Alright, that was enough theory, let’s get practical!
Our Little Bakery 
Imagine we have a little bakery at a local train station and we would like to offer an interesting bundle to our rushed morning customers. To find out the most interesting options we’re going to use MBA on our collected transactions. This dataset consists of 7 products in total: sausage bread, croissant, pain au chocolat (
), orange juice, coffee, cookie, and brownies.
The basic steps we need to apply for our MBA are:
- Step 1: Prepare the data
- Step 2: Generate association rules
- Step 3: Use some metric to choose the most interesting rule(s) for the business case.
As we can see from the table above, our data is already presented in a pretty good way. We have 298 transactions, each one with unique products from our set of 7 products: sausage bread, croissant, pain au chocolat, orange juice, coffee, cookie, and brownie.
Generating Simple Association Rules
As said before, a good way to start an MBA is by using simple association rules, i.e., rules with only one antecedent and one consequent.
We can easily do this by using permutations from the itertools python module as seen here below.
Alright, that’s a lot of rules, but many of them won’t be interesting. How to decide which are useful ones to keep? Some metrics can help us decide exactly this. The following six metrics are generally used depending on the situation:
Support
frequency of item appearance
Value: frequency
Confidence
Lift / Leverage
how much better does the rule perform compared to random chance?
Lift is the ratio, leverage is the difference, they measure the same thing (lift is more commonly used, leverage generally more easy to interpret).
Value Lift: > 1 for good association rule
Value Leverage: > 0 for good association rule Leverage
Conviction
Zhang’s metric
Value: < 0 if there’s a negative association (dissociation), > 0 if there’s a positive association where -1 and 1 are the extreme values.
These metrics are then used to prune the association rules we’ve generated to come back to the best/most sensible ones. This can be done by setting thresholds for metric values until you get the best batch of association rules to further look into.
Check the notebook tutorial for a hands-on with the small bakery dataset, where we go through this process step-by-step with code included.
Dealing with Big(ger) Data
Now, for a small dataset this is rather straightforward, but the larger the dataset – and the more items are within it – the more difficult this becomes. In order to deal with larger transactional datasets, we need to use some smart tricks.
Why is this important? Well, if we take a dataset consisting of transactions of customers in the Netherlands from a online retail dataset, we find it has 814 items. If you take 814 items, calculate association rules, and you slooowly increase the number of items chosen together.. well.. this happens:
Hence we need something to help reduce the big numbers to smaller numbers. Gentlewomen and gentlemen (gentlepeople? gentles?), introducing: the apriori algorithm.
In the most basic of terms, the apriori algorithm helps reduce the complexity of the above numbers explosion – that’s inherent to MBA – by using a very simple rule of thumb called the Apriori principle:
Apriori principle: a subset of a frequent set must also be a frequent set.
The significance of this is that you can prune (eliminate/terminate/86) lots of rules without needing to calculate any measures for them. For example, if the calculated support for the item cookie is deemed infrequent, that means that – according to the apriori principle – any set that has a cookie can be discarded without computation! This can very quickly reduce the explosion of possible rules to something more manageable. Then, if you would like to still look at more infrequent rules/items in the future, you can just change the threshold of what makes an item “infrequent” and analyse it as such (to prevent melting of CPU’s you might want to then also prune items with too high support).
As useful as the apriori algorithm is, it’s not the only tool in our toolbox. Another method that is very useful for simplifying MBA problems is Aggregation. Aggregating items into categories allows for rules to be calculated between categories instead of items.
The big benefit here is that you no longer lose low support items! There is however some added complexity in the choosing of the categories, as this dictates in large part the results you will get.
There’s a multitude of ways to find these categories, from using domain knowledge, to text mining descriptions, to one of many clustering algorithms, and so on. The way to go about this is very context dependent, and may necessitate the experimentation with various methods. Luckily you’re a smart individual so you will figure something out (: (or just type out some search engine queries and go from there like most of us mortals).
- Bundle of products: Which products could we offer together?
- Cross-promotion: Which product should we use to promote another product?
- Cross-promotion to sell a target product: If we want to promote a specific product which product(s) should we use to promote it?
- Organize layout of Store: Which product should be close so they would promote each other? Which ones to avoid putting together?
Conclusions
- Market Basket Analysis (MBA) is a powerful marketing tool that helps getting insights for product promotion and recommendations.
- All you need to start your MBA is a simple transactional dataset consisting of transaction ids (e.g., invoice numbers) and transaction items
Thank you for reading!
If you would like to actually look at some code implementations, or just dive a bit more in depth, take a look at our GitHub repository! It explains more about the methods, prepping the data, how to use plotting to make informed threshold decisions, and more!
Of course, feel free to use the code available in our repository to your own data and needs.
This blog tutorial is the result of a collaboration between me and Danielle Paes Barretto that built the material available at the GitHub repository. As such, if there are any questions and/or comments feel free to send an email to f.kok@jadsmkbdatalab.nl or d.paes.barretto@tue.nl and we’ll make sure to get back to you quickly (:
Adios! See you in the next one~
– Fabian & Danielle
