Event processing for an ad network

1 points by xstartup 8y ago ↗ HN

I am not creating a new ad network - it's my friend's ad network with 60B monthly clicks. We are looking to make it better, piece at a time but still need a big picture. Task mostly involves interfacing with a DSP. Logging impressions, generating reports for ad buyers/sellers, ultra-fast redirects. Doing this very cheap, reliable. I am thinking about using Kafa + Go binary (for serving impressions, Kafka producer/consumer). But generating 3 dimensions (group by X, group by Y, group by Z, finally filter) reports in real-time for ad buyers is a bit difficult. Is there any reference architecture for ad networks? Can anyone share their experience with something similar?

1 comment

[ 3.3 ms ] story [ 9.4 ms ] thread

robin_philip 8y ago ↗

We had a similar problem building an ad network. We ended up logging events into a HDFS using Kafka. Backend for serving was built using Java (Go Seems fine too). As for serving reports in real time, I don’t think that would be doable. We need up refreshing the reports every 30 minutes. Building these analytics views can be accomplished using Cassandra, Hive or ElephantDB or any other. People have used apache storm also. You can look up the book Big Data principles by Nathan Marz for discussions on scalable architecture.

Disclosure: I did not code this. I only participated in architecture reviews.