@Alexis,

For taxation overheads as a strawman

- a simple HTTP/1.1 roundtrip latency percentile distribution at 10^n (n = 2..5) qps with a 4k response payload

- CPU time measurement of the proxy under these loads

- RSS size under load

Bonus points for doing the same with HTTP2.

As William mentions having good LB & other network function control improves utilization & latency more holistically throughout the cluster but I do think these numbers will be helpful for folks

On Thu, Oct 6, 2016 at 12:06 PM, William Morgan <william@...> wrote:

We have HTTP/2 in alpha. Gory details here: https://github.com/BuoyantIO/linkerd/issues/174

Performance: a heady topic. That post is still accurate. Note that it's focused on a sidecar / low-mem configuration--as I'm sure you know, the JVM exhibits a, shall we say, "complex" relationship between memory footprint, CPU, throughput, and latency, and changing constraints can dramatically change the resource profile. (This complexity is not ideal for us, of course, but it's a result of a conscious decision to go with production-tested Finagle as opposed to starting with fresh code--just to give you context.)

E.g. the K8s configs in our recent Kubernetes post deploy linkerd as a DaemonSet--allowing us to amortize resource cost per node rather than per process, which gives us more breathing room--at the expense of a Horrible Hack to determine the node-local linkerd.

Finally, our assertion is that, with better load-balancing and flow control, end-to-end tail latencies can actually be *reduced* even when introducing a per-hop cost--though I'll admit that we don't have a nice experimental setup to back this up yet.

HTH,

-William

On Thu, Oct 6, 2016 at 10:59 AM, Brian Grant <briangrant@...> wrote:

Thanks for that additional information, William.

Are there plans for linkerd to support HTTP2?

Is the most recent performance and overhead data available that in https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/?

On Wed, Oct 5, 2016 at 6:38 PM, William Morgan <william@...> wrote:

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

We have HTTP/2 in alpha. Gory details here: https://github.com/BuoyantIO/linkerd/issues/174

Performance: a heady topic. That post is still accurate. Note that it's focused on a sidecar / low-mem configuration--as I'm sure you know, the JVM exhibits a, shall we say, "complex" relationship between memory footprint, CPU, throughput, and latency, and changing constraints can dramatically change the resource profile. (This complexity is not ideal for us, of course, but it's a result of a conscious decision to go with production-tested Finagle as opposed to starting with fresh code--just to give you context.)

E.g. the K8s configs in our recent Kubernetes post deploy linkerd as a DaemonSet--allowing us to amortize resource cost per node rather than per process, which gives us more breathing room--at the expense of a Horrible Hack to determine the node-local linkerd.

Finally, our assertion is that, with better load-balancing and flow control, end-to-end tail latencies can actually be *reduced* even when introducing a per-hop cost--though I'll admit that we don't have a nice experimental setup to back this up yet.

HTH,

-William

On Thu, Oct 6, 2016 at 10:59 AM, Brian Grant <briangrant@...> wrote:

Thanks for that additional information, William.

Are there plans for linkerd to support HTTP2?

Is the most recent performance and overhead data available that in https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/?

On Wed, Oct 5, 2016 at 6:38 PM, William Morgan <william@...> wrote:

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

Thanks for that additional information, William.

Are there plans for linkerd to support HTTP2?

Is the most recent performance and overhead data available that in https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/?

On Wed, Oct 5, 2016 at 6:38 PM, William Morgan <william@...> wrote:

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

I think it would be useful to have some standardized assessment of the CPU, latency & memory footprint tax that products in this space have.

A feature comparison matrix would be nice too.

On Thu, Oct 6, 2016 at 10:59 AM, Brian Grant <briangrant@...> wrote:

Thanks for that additional information, William.

Are there plans for linkerd to support HTTP2?

Is the most recent performance and overhead data available that in https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/?

On Wed, Oct 5, 2016 at 6:38 PM, William Morgan <william@...> wrote:

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

_______________________________________________
cncf-toc mailing list
cncf-toc@...
https://lists.cncf.io/mailman/listinfo/cncf-toc

Thanks for that additional information, William.

Are there plans for linkerd to support HTTP2?

Is the most recent performance and overhead data available that in https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-microservice-sidecars/?

On Wed, Oct 5, 2016 at 6:38 PM, William Morgan <william@...> wrote:

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

(adding Oliver, linkerd's maintainer)

Hi Brian,

Good point about Amalgam8, I should have mentioned them. Also maybe Netflix's Prana?

In my mind Traefik, Vulcand, etc. are largely targeted at ingress / "API gateway" use cases. Whereas linkerd, Envoy, Amalgam8, and Prana are focused explicitly on service-to-service communication. Not 100% familiar with all of those projects so I hope that's not a gross mischaracterization.

Differentiating features / advantages of linkerd:

• Focused on production-tested / used-in-anger code. Sticking close to Finagle, not writing our own load-balancing, etc. This stuff is hard to get right, tons of edge cases, etc. E.g. what happens when service discovery down? When it's up, but it's lying? Etc.
  
• Production users across the globe (see below), lots more on the way.
  
• Plugin architecture, tons of integrations, explicitly easy to drop in custom stuff.
  
• Routing layer (again, used in prod) that is extremely powerful and decouples routing topology from traffic-serving topology.
  
• Focus on modern operational affordances, e.g. retry budgets and deadlines over retry policies and timeouts.
  
• Lots of contributors, active community and friendly Slack!

Current real-life prod users that we know of (and can talk about--some we can't) are Monzo (UK), Quid (US), Douban (CN), CentralApp (EU), NCBI (US). Lots of other companies in various staging / QA environments that we're tracking and helping to productionize. Of course open source is a funny thing and people will occasionally drop into the Slack and say "so, I've been running linkerd in prod for a few weeks and I noticed this one thing..." so we don't have a complete picture.

Let me know if that's helpful. Happy to go into lots more detail.

Also, thanks for having me this morning, was really fun to present and you have all been super friendly and supportive!

-William

On Wed, Oct 5, 2016 at 5:40 PM, Brian Grant <briangrant@...> wrote:

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?

Hi, William. You briefly mentioned other similar proxies, such as Envoy. What do you see as linkerd's differentiating features or advantages compared to the others (Envoy, Traefik, Amalgam8)?

Also, are any of your users using linkerd in production?