Twitter details Heron, a real-time stream-processing system that outperforms Storm

Datetime:2016-08-23 02:37:27          Topic: Heron           Share
Twisted Agile: We're taking elements Agile dev and shaking it up with savvy best practices for better, faster outcomes. Sign up for our free webinar on June 11 at 10 a.m. PST/11 p.m. EST.

Twitter today is showing off Heron, a new real-time stream processing its engineers have built. Heron offers several advantages over Apache Storm, the stream-processing system Twitter developed in house and later open-sourced years ago.

For one thing, Heron is less demanding on the data center infrastructure that Twitter relies on.

“Since Heron is efficient in terms of resource usage, after migrating all Twitter’s topologies to it we’ve seen an overall 3x reduction in hardware, causing a significant improvement in our infrastructure efficiency,” Twitter engineering manager Karthik Ramasamy wrote in the blog post .

Several tests prove that Heron provides performance gains over Storm, too.

Heron is now the main stream-processing engine inside Twitter now, Ramasamy wrote — but it is backward-compatible with Storm, which has been widely used across Twitter in the past.

Several companies — and cloud providers such as Google Cloud Platform, Microsoft Azure, and Amazon Web Services — have embraced Storm over the years. It’s certainly possible that Heron will follow a similar trajectory.

Ramasamy did not say anything about releasing Heron under an open-source license, although he did say sharing an academic paper on Heron at the 2015 ACM SIGMOD International Conference on Management of Data was a “first step” toward sharing lessons learned “in order to further develop these programs.

For more on Heron, check out the full academic paper here .





About List