Date
1 - 4 of 4
Heron
|
alexis richardson
Moving Heron to new thread.
toggle quoted message
Show quoted text
All - comments on Heron? I think Brian raises some good questions about alignment with our mission (below). On Fri, Mar 17, 2017 at 3:56 AM, Quinton Hoole via cncf-toc
<cncf-toc@...> wrote: .. I don’t know enough about Heron and the related ecosystem to have an on that.... As for Heron, I'm not seeing the demand for Storm/Heron. For example, Google |
|
|
|
Jonathan Boulle <jonathan.boulle@...>
I would echo Brian's sentiments - I also have some concerns about Heron's cloud-native suitability and path to integration with any of the existing or proposed CNCF projects; I think it's still hard for me to understand why we're a more suitable choice than Apache. Alexis Richardson via cncf-toc <cncf-toc@...> schrieb am Fr., 17. März 2017, 11:56: Moving Heron to new thread. |
|
|
|
+Karthik to make sure he sees these emails as a chance to give feedback. If I recall from the call that was a question that BrianG may have asked. Karthik mentioned some things of why CNCF (vs other places) but I'll defer to him to answer any questions as he's the project representative. On Fri, Mar 17, 2017 at 10:00 AM, Jonathan Boulle via cncf-toc <cncf-toc@...> wrote:
--
Chris Aniszczyk (@cra) | +1-512-961-6719 |
|
|
|
Karthik Ramasamy <kramasamy@...>
Thanks Chris. Apologies for responding late.
toggle quoted message
Show quoted text
- Heron is designed to be container friendly. Currently we run Heron as a cgroup container in Mesos/Aurora (which is our production environment). - There are already two PRs which extends Heron to use docker so that a Heron job can be run as a collection of docker instances. - While we did just Storm API for Twitter’s needs, Heron design is extensible - in the sense, we can map any API on top of Heron easily. We have a clear distinction between DAG generation and DAG execution. This is the subject of a paper that got accepted in ICDE 2017. Happy to share a copy if needed. - We are also in the process of implementing exactly once (which requires state storage). Current implementations of exactly once is very messy in other streaming systems since they use Hadoop and very difficult to achieve low latency. If Kubernetes supports container portability (with storage), we will be first one to take advantage of it. I know one company that implements containers with storage portability (robinsystems.com). - Heron also runs in AWS in the Fabric division of Twitter (which was acquired by Google) a couple of months ago. We are in the process of making it natively run in ECS (EC2 docker container service) at AWS due to request by a customer. This is pretty straight forward to implement due to extensible design of Heron. - Finally, we published performance numbers for Heron after some simple optimizations. We can do a latency of 20 ms (but in reality we have pushed it 13 ms) and also high throughput. The blog is https://blog.twitter.com/2017/optimizing-twitter-heron. - Heron is the fastest and low latency engine in the market right now and we have another 4-5x to go. Heron provides the price/performance as of now. Let me know if you have any questions. cheers /karthik
|
|
|