P L E A S E  W A I T...

How Nabler Developed a Framework to Measure ROAS for Online Campaigns


The Client

The client, a leading media agency in North America, specializes in running multiple online ad campaigns for FMCG and electronics brands across the world. As a creative and technology partner for these brands, the media agency helps them to make strategic campaign management decisions in the modern marketing landscape.


The Objective

The client wanted to analyze the effectiveness of online campaigns — for brands they manage — on sales in physical stores. The client partnered with 10+ brands that have physical stores across major cities in the US. For instance, one partner, Hewlett-Packard, has more than 100 retail stores across the US.

The online campaigns predominantly focused on video and display content delivered through video and social platforms. Nabler’s task was to design and develop a methodology to measure their advertising success with ROAS.


The Challenge

  • The Scale of Ad Campaigns: The ad campaigns were large in number and spread across diverse media channels including Facebook and YouTube. Since the client was running 100+ campaigns for multiple brands in parallel, they needed a robust solution to accommodate this scale.

  • No Existing Framework: The client had no existing framework in place to measure the impact of online campaigns on sales. The manual intervention that the client was following to measure the campaign effectiveness was time consuming, labor intensive, and was deviating the client from the original goal.

Nabler’s Approach to Measure ROAS

Nabler conducted extensive data discovery sessions, exploratory data analysis, and Design of Experiments to collect the best data for modelling quickly and efficiently. Nabler’s Data Science Consulting team then developed and deployed a framework for ROAS measurement. The implementation was executed in two phases by setting up test and control Randomized Control Trials (RCTs):

1. Pre-Campaign – Screening Phase for Measuring ROAS

Nabler used historical time series of sales data and demographics data for the Designated Market Areas (DMAs). Upon selecting the test and control DMAs using time series correlation, Nabler launched the campaigns in test DMAs while restraining control DMAs.

Execution Challenges:
  1. Missing Relevant Data Fields: The Design of Experiments initially considered only the correlation of past data but not socio-economic factors that brands often care about which led to a problem of missing relevant data fields.
    Solution: Nabler added the demographics data from the US census in the pre-screening phase.

  2. Revenue Risks in Restraining Geos: By restraining certain geos in RCT, there was a potential loss in revenue. Furthermore, some geos may also hold a strategic advantage that was not measured by past revenue.
    Solution: To mitigate this risk, control geos were selected such that the total sales were less than 5% of overall brand sales across all geos.

2. Post-Campaign – Inference Phase for Measuring ROAS

Sales in control DMAs were used as regressors to generate a counterfactual estimation to predict sales in test DMAs without a campaign. Since highly correlated test and control DMAs were selected in the pre-campaign phase, Bayesian structural time series models were used to conduct the analysis. The difference between the predicted value and the actual sales value in the test DMAs provided an estimate of the lift in sales due to the campaign.

Execution Challenges:
  1. Control group contamination: In a few campaigns that ran for 2-3 months, the restrained DMAs still ended up being exposed due to process gaps. This contaminated the control groups.
    Solution: To counter this, the contaminant geos were excluded from the ensemble, and the control groups were redefined in the post-campaign analysis.

How It Helped the Client

  • Campaign Prioritization: By accurately measuring ROAS, the client was able to quantify the true value that it was delivering to its customers. This also enabled the client to identify and isolate campaigns that would drive more ROAS than others.

  • Streamlined Process: As the process of measuring ROAS is streamlined, scaling the framework across campaigns became effortless and allowed the client to perform 100+ analyses in two years.


Want to learn more? Let's Talk.