Today, companies have more access to information about their customers than ever before. But just because you have data doesn’t mean you know how to use data. Processing, and organizing data so it’s useful for driving business decisions is no easy task. Especially for a company collecting massive amounts of data – to the tune of 20 billion records per day.
Clear Channel Outdoor (CCO), a leading out-of-home (OOH) media company, turned to Aqfer in 2022 to help develop the data processing mechanisms to organize and activate data about its inventory of 70,000+ billboards and digital displays across the US.
Clear Channel Outdoor RADAR, Powered by Aqfer
AT A GLANCE
Records Processed on an Average Day
Static and Digital Billboards and Displays Measured
US Metros with CCO Operations
Activating Billions of Mobile and Location Data Records
Long considered a digital-first company revolutionizing out-of-home, data processing and activation wasn’t new to CCO when we began working together. Their revolutionary RADAR solution was actively assisting in campaign planning and measurement for their advertiser customers. But due to the complexities and cost associated with managing the massive amounts of data required to power RADAR, it was only available in a handful of markets.
Our challenge was to work together with CCO to build a data engine that could process billions of mobile and location data records to supercharge RADAR. This complex task consisted of making sense of all data to accurately process it with confidence, and to design the technology stack to ensure optimal coverage with minimal overlap. To right-size the tech stack, Aqfer worked with CCO to determine the right mix of vendors to contribute third-party data and technical support.
How Aqfer Built the Data Processing Engine to Supercharge RADAR
RADAR works by leveraging aggregated and/or anonymous mobile location data to analyze OOH campaign performance.
In the course of an average day, a single person (and their mobile device) may drive to and from work or school, run a variety of errands, etc – thus passing many individual billboard displays. Our goal is to stitch together the entire activity of that mobile device (in an aggregated and anonymous manner) to determine when that device exposed (viewed) individual displays and billboards. In this example, there may be 20B mobile data records available – our task is to sort through those records to identify and provide contextual insight like audience segment affinities or attribution of sales/conversion of the people that actually viewed the billboard.
Aqfer helped CCO scale and streamline this work to make RADAR more sustainable and efficient. We worked together to accomplish the following.
Define Ad Viewership
Creating a definition of who would count as a viewer, taking into account the direction of travel and the angle of the billboard.
Enhance Reporting
Creating audience reports and surfacing the data to a front-end UI.
Connect to Third Parties
Once “traffic” data is known (by panel & by DMA), layering in audience data from other trusted providers (Bombora, Nielsen, etc.).
Establish Proximity, Time of Exposure, and Directionality
Based on the geographic location of the device and the latitude and longitude of the billboard display.
Select Vendors
Helping CCO determine which third-party partners have the “best” data – meaning the most coverage that fits their needs, while also considering the cost of acquiring and processing the data.
Plan for Effective Data Management
Determining the types of technical jobs required to manage the data effectively. Particularly, we needed the ability to aggregate data at the individual panel level and individual DMA level.
Improve Collaboration
Making it possible for the CCO team to directly evaluate data, create customized reports and work with Aqfer to fine tune data models and processing methodologies. None of these direct hands-on capabilities were previously possible.
Partnering with Amazon Web Services to Drive Success
Although Aqfer technology can be deployed across multiple platforms, AWS was the logical choice for this initiative. Here’s how we set it all up:
Automated Resource Provisioning
AWS CloudFormation
For each of the individual stages of the complex processing pipeline CCO is performing on Aqfer’s platform there are tailor-made settings that define resources to be requested based on particular workloads such as sorting data, collating/joining data, and applying transformations to data. These definitions are specified in the abstraction of AWS CloudFromation APIs and ensure the most efficient cost/time value for processing data.
Scalable Data Lakehouse
AWS Simple Storage Service (S3)
With a focus on security and ease of use, Aqfer’s platform affords CCO to bring in trillions of records on a monthly basis from multiple 3rd party data vendors via the Secure Data Sharing (SDS) feature. SDS is built on top of the AWS S3 service and allows ingress/egress of data at fast speeds and very low costs. What SDS also does in conjunction with AWS IAM is prescriptive access control ensuring only explicitly enumerated parties have access to data they are entitled to. Lastly, Aqfer relies heavily on S3 data storage classes and retention policies to ensure favorable economies at scale.
Access Control for Inbound Data
AWS IAM + Secure Token Service
To facilitate the secure transfer of data into the AWS S3 storage, Aqfer utilizes AWS Identity and Access Management (IAM) roles. By creating and assigning specific IAM roles to 3rd party data vendors, Aqfer ensures that only authorized entities can upload data to designated S3 buckets. Each IAM role is configured with finely tuned policies that grant specific permissions, such as s3:PutObject and s3:ListBucket, which are necessary for uploading data while maintaining strict access control.
Additionally, vendors can use temporary security credentials, obtained via the AWS Security Token Service (STS), to assume these IAM roles securely. This approach eliminates the need for vendors to manage long-term access keys, reducing the risk of compromised credentials.
Access Control for Outbound Data
AWS IAM
For data egress, Aqfer employs IAM roles to control access to data being downloaded from S3. Only entities with the appropriate IAM role can read or copy data from the S3 buckets. These roles include permissions like s3:GetObject and s3:GetBucketLocation, ensuring that data can only be accessed by authorized users. Aqfer can also leverage AWS IAM policies to enforce additional security measures, such as requiring multi-factor authentication (MFA) for role assumption or specifying conditions under which data can be accessed (e.g., IP address restrictions).
Balancing Relevancy, Speed, and Cost
AWS S3 Standard, Intelligent-Tiering, and Glacier
Aqfer relies heavily on S3 data storage classes, such as S3 Standard, S3 Intelligent-Tiering, and S3 Glacier, to optimize cost and performance. These storage classes allow Aqfer to balance the need for quick data retrieval with cost savings for less frequently accessed data.
Furthermore, retention policies play a crucial role in reducing costs and managing storage size effectively. Aqfer sets policies to delete data that is no longer needed after a certain retention period like 90 days for actionable data and 365 days for static reference data. This automated lifecycle management not only reduces storage costs but also ensures efficient use of storage space, preventing unnecessary data accumulation.
Efficient Compute As Needed
AWS Elastic Kubernetes Service (EKS)
At the scale of the trillions of records that are retained and processed on a daily basis AWS EKS Kubernetes is a service that Aqfer relies on to minimize compute costs while not sacrificing reliability. The ability to specify the number of pods, their sizes, and configuration per individual workload demands is a huge advantage over continually provisioned EC2 resources. AWS CloudWatch logs and Alarms are closely coupled with EKS to give operational observability and quick reaction adjustments with ever-shifting data landscape changes.
Cost Efficient Analytics
AWS Athena
After processing the large-scale data extremely cost-efficiently it only reasons that value should be extracted with the derived output. Aqfer platform is built on top of the AWS Athena service APIs giving CCO programmatic access to query data in a flexible and low latent fashion. Operationally, CCO also utilizes AWS IAM Role Assumption to get direct access to data within Aqfer’s platform, query data as well as transfer results to their internal systems or 3rd parties who have been granted access.
THE RESULTS
Processing Higher Volumes of Data with More Confidence
After successfully implementing Aqfer’s data processing on top of AWS, CCO saw dramatic and immediate improvements to RADAR.
Reduction of Data Volume and Streamlined Data Processing
Over the first 90 days, we processed nearly 1.8 trillion records. Each day, we continue to process roughly 20 billion records as new data continues to come in. Today, these jobs run much more quickly and efficiently when compared to the pre-Aqfer baseline.
For example, RADAR can now run a daily data ingestion job consisting of 20B mobile location records, with a 640GB file size in 121 minutes. Before Aqfer, a job like this would have taken 6-8 hours. Moreover, while a job like this may have started with 20B records, Aqfer streamlines and stitches the data together to reduce the record volume to less than 20% of its original size.
Increased Confidence in and Ownership of Data
Importantly, Aqfer was able to help CCO take control over their data. Previously they relied heavily on third-party vendors with a vested interest in selling more data to CCO. Aqfer’s solution empowered CCO to take charge of the processing and management of their data, ultimately improving their confidence in the quality of the RADAR product.
Gone are the days of data locked away in a vendor’s ecosystem. CCO can now run custom, ad hoc, and self-serve processes for QA, while all data is accessible to the CCO engineering team at all times.
BEYOND DATA
Clear Channel Outdoor Pushes The Boundaries of OOH
CCO is the only OOH company that integrates with data clean rooms, and it shows how they continue to innovate. CCO has revolutionized the OOH industry with RADAR, and Aqfer is proud to power this industry-leading capability. The ability to improve the planning, measurement, and reporting of OOH campaigns is bringing traditional digital ad measurement and attribution techniques into the physical world – something that was previously unattainable in the OOH industry at this scale.
RADAR’s measurement metrics are backed by best-in-class methods for accessing, measuring, and reporting on real-world activities. And, they’ve taken it one step further by matching those metrics to buyer behavior patterns like monitoring in-store visits for retail or automotive for exposed audiences vs control groups.
RADAR gives CCO advertisers previously unheard-of campaign planning and optimization toolkits. They can now hit the right audiences with the right displays in the right locations while minimizing investments with locations that have a lower propensity to deliver the desired target audiences.
Ready to build your own industry-disrupting solution? Reach out to learn more about how Aqfer built RADAR, and how we can help your company push the boundaries of what’s possible in your industry.