I do quite a bit of A/B testing and find it to be a great tool for experimenting and ultimately improving things.
But what’s “Coffee A/B testing”?
The idea came to me when I was visiting my wife’s family in Japan. We went to a restaurant and my father and brother in-law ordered two types of Sake. They let me taste both and decide which one I liked the most. It was a simple task, but an interesting one. The tastes were subtly different, but enough that I could clearly pick my personal favourite.
It then occurred to me that as much as I love coffee, and tend to pick some beans over others, I don’t quite know what makes me like a certain type, or what it is that I’m looking for for my “ultimate” coffee.
What if I could A/B test coffees? Try two types of beans (or blends), and pick the one I like. Then repeating the process I could gradually find the one I like the most. And in doing that, I can also figure out what it is that I like, and pay more attention to the difference. I rarely compare coffees. Well, not any more!
Contenders
To get things started, I ordered 5 different types of beans online:
- India Monsooned Malabar AA (my usual choice for day to day coffee. Gives very nice crema)
- Guatemala Genuine Antigua shb (strictly hard beans)
- Ethiopian Sidamo Grade II Organic (which I took with me on the trip to Japan and enjoyed drinking with the Aeropress or filter)
- Ethiopian Yirgacheffe
- Kenya AA Plus Kianjiru Estate
All beans are 100% Arabica, and were packaged and roasted by the same roastery (Speicherstadt Kaffeerösterei, Hamburg). I noticed that the expiry date on all packages was the same, so I’d assume they should have roughly the same level of ‘freshness’.
Testing method
Trying to compare all 5 at the same time shouldn’t be impossible, but I wanted to try out the A/B testing approach. Always taste and compare two variants against each other. I wanted to make it as simple as possible. The only question I wanted to answer during each tasting session was “which one is better? A or B”. The answer could be A, B or “the same”.
I think this approach has several advantages:
- You only need to make two cups of coffee. This is pretty important if you want to reduce inconsistencies, and don’t have 2 identical grinders, coffee makers etc (which gets much worse with 5).
- For the same reason, it’s easier to compare two beans using different coffee making techniques (Aeropress, espresso, cappuccino, filter).
- Your smell and taste do not get overwhelmed with too much choice.
- I find it hard to decide on things in general. But picking between two is a relatively easy decision to make.
- You can actually drink both cups and enjoy them. No caffeine overload.
The number of combinations of (unordered) pairs from a set of 5 is if I’m not mistaken 10. That gives an opportunity to do 10 different tasting sessions and score each coffee (2 points for a winner, 0 for the loser, 1 each for a tie). Then calculate the winner. I’m not sure it’s the best way, but it’s a fun experiment.
Experiment setup – The “Gingerlime Tasting Technique” ™
To get things started, I wanted to make sure the tasting is “blind”, i.e. I didn’t want to be influenced by any opinions or impressions apart from the actual smell, texture and taste. So I assigned A, B, C, D and E blindly to each bag of beans (more on that later).
I am still considering an even more extreme “blindness”, where I won’t even know which variant I’m tasting against which. This is to avoid later tests being influenced by previous ones. Lets say I picked A over B, and then I taste A against C. I might already pick A as a likely favourite, before even starting.
To do this, I printed 5 QR codes and I cellotaped them onto the actual label. This way it’s easy to scan them and see whether a bag of beans is A, B, C, D or E. But it’s not immediately visible without the scanner. Printing the QR codes also helped me assigning them to different bags blindly without remembering which one was A and which one was B and so on…
First taste
On my next post I’ll cover the first tasting session. I first tested the beans using Hario V60 filter and then again in other separate sessions compared the beans when making an espresso, using the Aeropress and finally making a Cappuccino. I’ll update the link to the next post when it’s available, so stay tuned.
UPDATE: first A/B tasting session post is now published here.
