Testing Culture at trivago: A Marketing Perspective
“We make decisions based on data” – I had heard plenty of companies say that and passed it off as a lip service. That was until I came to trivago. This was not just something advertised, but a concept we truly believe in called, “Power of Proof”. If someone was casually passing through our campus corridors they would hear, “How can we test this hypothesis” at least 167 times (alright, this one hasn’t really been measured yet).
How does the testing culture work day-to-day?
The testing culture means that decisions in trivago are based on evidence from data. Additionally, we invest a lot of effort in creating tests and analysis to get to these learnings. We believe in ideas and creativity, but these are used to come up with a strong hypothesis. Then we attempt to create a clean test setup to evaluate if the assumptions are indeed true and if the result is positive, then the decision is scaled. We also use data from our strong reporting and analysis tools to make decisions to challenge best-practices and get the best “bang for the buck” on our marketing activities.
Consider this example:
Person A: I have an idea; I think if we ran our campaign in California instead of New York we could get a better ROI.
Person B: Why do you think we will get better results in California?
Person A: Because in the past campaigns this year we saw 10% lower CPM (Cost per Thousand Impressions) in California than New York.
Person B: That is interesting, however the spend in California was much lower than New York in the past. Can we compare the CPMs here?
Person A: I propose we frame a hypothesis that, “We should move the budget of New York to California as will see better ROI in California on our marketing campaign because the CPM is significantly lower in California than in New York”
Person B: What would be a good test set-up for this?
Person A: A short test campaign with similar budget, creative and seasonality in both states to have a clean test set up.
Person B: Sounds like a plan 😀
In the various teams in the marketing department, many such conversations are going on at any given time, be it in planning forecasts, creatives, channel mix, further details in each channel for instance, platforms, placements, going up to even time of the day sometimes!
Who actually does the measurement?
Our testing philosophy is supported by our strong measurement team. Marketing channel owners work closely with the measurement team to create a roadmap for the channel. Here we discuss goals for the channel, for instance it could be driving Brand Awareness, traffic to website or ROI. We further align on KPIs, targets and tests and test setups. We use data across 53 markets, multiple channels and testing products offered by our channel partners to create and measure tests. We use the results to determine next steps, whether it is to scale, retest or on some unfortunate days, kill a topic.
For instance, to understand the impact of new channels, we turned to using incrementality testing, where we tried to isolate the impact, a channel brings over and above our regular marketing activities. This was done using the Brand and Conversion Lift Studies offered by our partners and using internal data.
Over the years we have now developed very detailed yet friendly dashboards that can be used to do repetitive analysis. As we learn more and experiment more with data and channels, new dashboards are added to make decision making faster.
In this way, we bring technical knowledge of marketing channel owners together with data science to optimise marketing efforts be it in forecasting, marketing channel performance attribution models, elasticity of markets, Direct Response measurement of TV and more.
Crux of the matter
But how do we stay motivated to follow, “Power of Proof” consistently? Simple, it’s the people. I have found that people at trivago are sincere about getting results, there is always a challenge and a good brainstorm to arrive at important decisions. This is what motivates each one of us and in turn, each other. There is joy in finding a strong data point, executing a test and seeing the results to make the geek in me happy! This also makes the work decisions very democratic as decisions are led by numbers and not just by top-down.
P.S.: The example is fictitious, please use Power of Proof before moving your spends to California 😉
Thanks to Social Media Marketer Sheetal Ninne for writing this article!