# From Protoype to Production Lessons from building and scaling Reddits Ad Serving Platform
# About
Reddit is frontapge of the internet
# Pieces in Ad tech
Scale: almost all reddit requests will hit the adserver
Speed: sla 30ms (P99)
Auction: run auction real time for each request ; high quality ad; large bid
Pacing: not showing ads from the same advetiser all at once
# Ad Serving Infrastructure:
Prometheus Meos RocksDB: emebddable kv store Kafka Go ApacheThrift : for rpc; legacy systems still need this but would otherwise use grpc Apache Spark Amazon S3 Apache ZooKeeper
Ad Selector: (go)
- 30 ms P99
- a thrift call is made to this service
- job is to get ads and return it back
- makes call to enrichment servicew which gives more info on request/ user / other relevant info needed for request|
- logs data to kafka (for reporting/ billing)
- complex Business Rules
- Auction
Enrichment Service (go)
- embedded rocks DB
- 4ms P99
- does a prefix scan ; does some aggregation and retunr out
Event Tracker: (go)
- 1 ms P99
- acknowlege & log events ; request made by user browser to this service
- needs to be highly relaible
Other Go Services: Reporting, Vault Admin , Ad Event Generation Service
Spark Job Events Stats Spark Streaming
- takes data from kafka
- joining and writing to enrichment service
Ad Stats Spark Streaming:
- count how much ads we have shown for each advertiser
Pacing Saprk Job:
- making sure we are allocating ads in an optimal manner
went from 100 ms to 10 ms
# Lessons Learned
- Choose a framework
Reddit decided to go with Go-kit
reddit go kit service stack
- Changing service
- introduced ad selector that just forwarded to serivce
- introduced new native service that proceessed request , but still used thrid party
- slowly use new service