When my team and I at Aqfer first dove into the world of marketing data management, we quickly realized we were facing a Goliath of a problem. The sheer volume and complexity of data in this industry were staggering, and existing solutions just weren’t cutting it. I remember thinking there has to be a better way. And of course, there was. It was the spark that ignited our journey to revolutionize storage of marketing data to make it more useful and more efficient.
Learning from the Past
To solve this problem, we first had to understand why current systems were falling short. Row-based storage was great for grabbing entire user profiles quickly, but it fell short when we needed to analyze specific attributes across millions of users. On the flip side, columnar storage was a champ at those broad analyses but stumbled when we needed detailed information on individual users.
It was like having two really good tools, each perfect for a specific job, but neither quite right for the complex, multifaceted needs of AdTech. That’s when it hit us – why choose? Why not create something that combines the strengths of both?
Our Eureka Moment: The Hybrid Approach
This realization led us to develop what we now call our row-columnar hybrid model. At its heart are what we’ve dubbed “multiplexed columns” – think of them as the best of both worlds, packaged into neat, 64-column chunks. This structure allows us to compress data efficiently (saving on storage costs) while still retrieving it lightning-fast, whether we’re pulling specific attributes or entire user profiles.
We didn’t stop there, though. We incorporated advanced compression techniques like:
Run-Length Encoding
A form of lossless data compression. Data runs are stored as a single occurrence of a data value and a count of its consecutive occurrences, rather than as the original run.
Bitmap Indexes
A type of database index structure that uses bitmaps to improve query performance.
These might sound like dry technical terms, but in practice, they’re game-changers. They allow us to squeeze data down to a fraction of its original size without sacrificing speed – crucial in an industry where every millisecond counts.
Real Results: Faster Queries, Lower Costs
Nothing beats the feeling of seeing your creation make a real difference. One of our clients, a major ad tech firm, was struggling with query times and rapidly escalating and unpredictable costs. After implementing our system, they saw query speeds improve by 70% and their data processing costs plummet by 98%. That’s not just a statistic – it’s the difference between making or missing out on real-time bidding opportunities, between generic ads and perfectly targeted campaigns.
What This Means for the Marketing Data Industry
I’m often asked what sets our solution apart in the AdTech world. It really comes down to three key advantages:
Speed
In real-time bidding, milliseconds matter. Our system delivers the speed needed to make split-second decisions.
Precision
We enable more sophisticated audience segmentation, allowing for laser-focused targeting.
Scalability
As campaigns grow and data multiplies, our system scales efficiently, maintaining performance without breaking the bank.
A Better Way
What excites me most about our innovation is its potential for growth. We’ve designed it to be backward-compatible with standard formats like Parquet, making it easy to integrate with existing systems. But we’ve also built in the flexibility to adapt to new data structures and encoding methods as they emerge.
To my fellow tech leaders: I invite you to consider how this approach could transform your operations. In an industry that moves as fast as ours, staying ahead of the curve isn’t just an advantage – it’s a necessity. And I believe that rethinking our approach to data storage and retrieval is the next big leap forward.
Want to keep the conversation going? Reach out – I’d love to discuss the future of data storage at your company, and see if Aqfer is a good fit for you.
About the Author
Daniel Jaye
Chief Executive Officer
Dan has provided strategic, tactical and technology advisory services to a wide range of marketing technology and big data companies. Clients have included Altiscale, ShareThis, Ghostery, OwnerIQ, Netezza, Akamai, and Tremor Media. Dan was the founder and CEO of Korrelate, a leading automotive marketing attribution company, purchased by J.D. Power in 2014. Dan is the former president of TACODA, bought by AOL in 2007, and was the founder and CTO of Permissus, an enterprise privacy compliance technology provider. He was the Founder and CTO of Engage and served as the acting CTO of CMGI. Prior to Engage, he was the director of High Performance Computing at Fidelity Investments and worked at Epsilon and Accenture (formerly Andersen Consulting).
Dan graduated magna cum laude with a BA in Astronomy and Astrophysics and Physics from Harvard University.
