Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. In this talk, we will share our operating experiences and challenges of running Heron at scale and the approaches that we took to solve those challenges.
Karthik Ramasamy is the engineering manager and technical lead for Real Time Analytics at Twitter. He is the co-creator of Heron and has more than two decades of experience working in parallel databases, big data infrastructure and networking. He cofounded Locomatix, a company that specializes in real time streaming processing on Hadoop and Cassandra using SQL that was acquired by Twitter.
This talk is from the DataEngConf SF in April 2016.