Date   

Good read: How Grafana Labs enables horizontally scalable tail sampling in the OpenTelemetry Collector

Mark Carter
 

How Grafana Labs enables horizontally scalable tail sampling in the OpenTelemetry Collector

Published: 18 Jun 2020 RSS

Tracing is a widely adopted solution to provide performance insights into distributed applications. It is a valuable resource for developers to view the service call graph and track service latency at a granular level. It’s also a handy tool for on-call engineers to drill down and debug a problematic service during an outage. There are a number of open source distributed tracing frameworks out in the wild, including Jaeger, Zipkin, and OpenTelemetry. These provide libraries to instrument client applications, as well as components to process, store, and query trace data.

At Grafana Labs, we use Jaeger and OpenTelemetry extensively to track performance of various projects in production including Cortex, Metrictank, and Loki.

Sampling: dealing with large data volumes

Since most of the traces in a healthy application are almost the same, storing all traces from an application can often be overkill as well as increase operational cost. Sampling is a good method to throttle down and store representative traces, and in most implementations, the sampling decision is made upfront and at random (probabilistic). But this form of sampling, often called head-based sampling, can miss out on important traces that we are interested in, such as traces with errors or high latencies, which may never get sampled because they are (hopefully!) rare.

To deal with this problem, the OpenTelemetry team implemented a processor in the collector component that performs tail-based sampling. In this form of sampling, the sampling decision is not necessarily made upfront, but rather based on conditionals evaluated on full traces that are aggregated in-memory at the collector level. Example conditions:

  • http.status_code > 201
  • error = true
  • service = “cortex-gateway” && duration > 10s

This brings much needed flexibility to sample representative traces, as it allows us to leverage complete trace data post application processing. A combination of head and tail based sampling can be used depending on the traffic through the cluster, and the desired storage capacity for traces.

The problem

The OpenTelemetry Collector does not perform tail sampling when scaled horizontally. When multiple instances of the collector are deployed, spans belonging to the same trace could end up with different collectors. Since these instances have no way of communicating with each other, each of the instances would make a tail sampling decision on a partial trace – a subset of spans – that reached the particular instance. This could cause unexpected results where only a subset of spans in a trace that satisfied the sampling conditions is saved to the backend.

The solution

In order to make the tail sampling processor horizontally scalable, we designed a new processor, called the aggregate processor. This processor solves the problem by forwarding spans between the collectors in a way that all spans with the same traceID end up at the same collector. This ensures that each instance of the collector can make a sampling decision on the entire trace, and no partial traces are flushed to the backend.

The aggregate processor works by discovering peer instances of the collector via DNS (this can easily be extended to include plugins for etcd, Consul, etc.) and forwarding spans to peers based on traceIDs. To determine which peer a particular span should be forwarded to, each collector instance performs a fnv hash of the traceID, followed by a jumphash. This addresses two problems:

  • The fnv hash is used to protect against bad instrumentation of the client applications wherein traceID generation does not use the entire 128 bit keyspace.

  • Jumphash ensures that redistribution of traceIDs is minimized when scaling the number of collectors. In simple modulo hash, a lot of spans would be “orphaned” during collector churn, because each collector repartitions the entire keyspace, whereas with jumphash, at most 1/N of the traces will be repartitioned, where N is the number of instances.

By adding the aggregate processor before the tail sampling processor in the processing pipeline, we ensure that all spans of a single trace are forwarded to the same collector instance. Once a trace can be assembled in-memory, conditions as discussed above can be applied to make a sampling decision.

Results

We have been using the collector with the aggregate and tail sampling processor in our internal infrastructure. We are ingesting ~14k spans a second with ~60 spans per trace with success. These are some graphs from our internal deployment of the OpenTelemetry Collector:

Grafana Labs deployment of the OpenTelemetry Collector

(A) Spans received from the application at the thrift endpoint.

(B) Spans forwarded between peers and received at the OpenCensus endpoint.

Another advantage of running the horizontal scaling tail sampling processor is that in case of a spike in incoming traffic, the storage backend is not overloaded because the number of sampled traces remains almost constant. This is depicted in the following figure: We are consistently sampling very few traces even through a spike in traffic.

Trace sampling through a spike in traffic

Check it out!

The aggregate processor is available in our GitHub repository. A Docker image built off of this branch is also available here. Do check it out and let us know what you think!



Sent from my mobile phone 


Great article: Jaeger embraces OpenTelemetry collector - JaegerTracing - Medium

Carter, Mark
 

Jaeger embraces OpenTelemetry collector

In this article we are going to explain Jaeger integration with the OpenTelemetry collector, describe differences between these two and have a look at Kubernetes deployment via Jaeger Operator.

OpenTelemetry collector is a vendor-agnostic service for receiving, processing and exporting telemetry data. In the Jaeger project we have decided to deprecate the Jaeger collector and migrate its functionality to an implementation based on OpenTelemetry collector. This has several benefits:

Roadmap

In the long term we would like to base the Jaeger collector, agent and ingester components on OpenTelemetry collector. These new components will be separate distributions with new image and binary names.

Our goal is to provide a smooth migration from existing binaries by supporting legacy Jaeger configuration (flags, env. vars, Jaeger configuration file). However, there will be a couple of breaking changes:

  • a different set of metrics exposed by these components
  • different health check and metric endpoints
  • not all flags will be supported in the new component e.g. --metrics-backed, --collector.queue-size

Follow this milestone on Github to see what is missing for the first stable release.

In the meantime, you can use collector image with thelatest tag jaegertracing/jaeger-opentelemetry-collector:latest.

Configuration & give it a try!

Upstream OpenTelemetry collector is configurable via a configuration file that is provided as a flag at application startup--config-file=config.yaml. If the configuration file is missing the collector will not start because it does not know what components (receivers, processors, exporters) should be used.

Jaeger’s build of the OpenTelemetry collector is opinionated about the configuration and it always uses a set of default components: Jaeger receiver, processors, and exporter. The exporter is one of Jaeger’s supported storage backends: Elasticsearch, Cassandra, Kafka (buffer). The configuration provided in the file is merged with the default configuration.

Jaeger specific components can be configured by the same flags that were exposed by the Jaeger collector e.g. --es.server-urls. The configuration provided in OpenTelemetry config has higher precedence.

The configuration precedence from the lowest to the highest is as follows:

  1. Jaeger default values <Jaeger config file< environmental variables < flags (Viper’s default precedence order)
  2. OpenTelemetry configuration file

Let’s have a look at an example that configures Jaeger OpenTelemetry collector:

  • Enabled Elasticsearch backend with URL http://elasticsearch:9200, 3 primary shards (default is 5) and 2 replica shards (default 1)
  • Disables batch processor (enabled by default)
  • Enables attribute processor (disabled by default). Note that new components have to be explicitly added to the pipeline.
docker run --rm -it -v ${PWD}:/config \
-e SPAN_STORAGE_TYPE=elasticsearch \ #dsa
jaegertracing/jaeger-opentelemetry-collector \
--config-file=/config/config.yaml \
--es.server.urls=http://localhost:9200 \
--es.num-shards=3

The content of config.yaml:

exporters:
jaeger_elasticsearch:
es:
server-urls: http://elasticsearch:9200
num-replicas: 2
processors:
batch:
disabled: true
attributes:
actions:
- key: user
action: delete
service:
pipelines:
traces:
processors: [attributes]

The storage exporter has to be configured via the same environmental variable SPAN_STORAGE_TYPE as Jaeger collector.

Jaeger Operator

Because the new components can be considered as almost drop-in replacements for existing Jaeger binaries we will be able to directly use them in the Jaeger Operator, by explicitly providing the image name in the CR. The required change in the Jaeger Operator is to expose the OpenTelemetry configuration in the CR. At the moment this is just a design proposal that is being discussed in the issue jaeger-operator/issues/1004.

Following the same configuration as in the previous section, the OpenTelemetry configuration is directly embedded into Jaeger collector node:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
collector:
config: |
exporters:
jaeger_elasticsearch:
es:
server-urls: http://elasticsearch:9200
num-replicas: 2
processors:
batch:
disabled: true
attributes:
actions:
- key: user
action: delete
service:
pipelines:
traces:
processors: [attributes]
storage:
type: elasticsearch
options:
es:
server-urls: http://localhost:9200
num-shards: 3

Conclusion

We have explained how the Jaeger project integrates with OpenTelemetry collector and what are the key differences between these components. Share your feedback with us and try our new collector based on OpenTelemetry.

References



Sent from my mobile phone 


Projects using telemetry

alexettelis@...
 

Hi,
I'm looking for inspiration on how OpenTelemetry is used in medium/large scale projects.
Is there any list of such projects?


OTel winter break

Ted Young
 

As discussed in the last Community meeting, I’ve canceled all SIG meetings for the winter holidays, Dec 23rd - Jan 1st. If you *would* like a meeting scheduled during that time, please let me know and I can add it back for you. 

Happy Holidays!

-T


OpenTelemetry community feedback

Ted Young
 

Hello everyone,

It’s the end of the year, a good time to look back on how far we’ve come, and look forwards to an exciting 2020. OpenTelemetry is always evolving, and as we gear up for the next phase, we’d like to provide an opportunity for anonymous feedback. We’ve created a feedback form to facilitate this. 

Any feedback is fine, but we are particularly interested in how we can improve our processes and community structure. Please submit as many pieces of feedback as you like.


Feedback form: https://forms.gle/m2iY9rYm1Xvc6fUs9

Cheers, and happy holidays,

- Tedsuo and the OTel governance committee


Re: [External] Re: [opentelemetry-community] Kubecon

Hamel, Jordan
 

Any table size needed or anyone here already?


From: cncf-opentelemetry-community@... <cncf-opentelemetry-community@...> on behalf of Morgan McLean via Lists.Cncf.Io <morganmclean=google.com@...>
Sent: Wednesday, November 20, 2019 5:45:06 PM
To: cncf-opentelemetry-community@... <cncf-opentelemetry-community@...>
Subject: [External] Re: [opentelemetry-community] Kubecon
 
I'll be there, thanks for organizing!

On Wed, Nov 20, 2019 at 12:51 PM Brian Putt <puttbrian@...> wrote:
I look forward to meeting everyone at 730pm at the UNION Kitchen & Tap Gaslamp

On Wed, Nov 20, 2019, 09:24 Venkata Bhumireddy <starzbhumi@...> wrote:
Hi,
I am also new and now I am in Charlotte. I can contribute to open telemetry. Please let me know.
I am currently working to bridge open telemetry to open tracing with one configuration. From micrometer to jaeger. Please share me your thoughts on this.

Best Regards,
Venkata Bhumireddy.

On Wed, Nov 20, 2019 at 12:17 PM Vineeth Reddy <vineethbfhs@...> wrote:
Hey!

I am new to open-telemetry, would love to meet you guys. Looking forward in contributing to OT.
Count me in for the meetup. :)

Thanks for the invite.

On Sun, Nov 17, 2019, 11:01 Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Morgan McLean
 

I'll be there, thanks for organizing!


On Wed, Nov 20, 2019 at 12:51 PM Brian Putt <puttbrian@...> wrote:
I look forward to meeting everyone at 730pm at the UNION Kitchen & Tap Gaslamp

On Wed, Nov 20, 2019, 09:24 Venkata Bhumireddy <starzbhumi@...> wrote:
Hi,
I am also new and now I am in Charlotte. I can contribute to open telemetry. Please let me know.
I am currently working to bridge open telemetry to open tracing with one configuration. From micrometer to jaeger. Please share me your thoughts on this.

Best Regards,
Venkata Bhumireddy.

On Wed, Nov 20, 2019 at 12:17 PM Vineeth Reddy <vineethbfhs@...> wrote:
Hey!

I am new to open-telemetry, would love to meet you guys. Looking forward in contributing to OT.
Count me in for the meetup. :)

Thanks for the invite.

On Sun, Nov 17, 2019, 11:01 Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Brian Putt
 

I look forward to meeting everyone at 730pm at the UNION Kitchen & Tap Gaslamp


On Wed, Nov 20, 2019, 09:24 Venkata Bhumireddy <starzbhumi@...> wrote:
Hi,
I am also new and now I am in Charlotte. I can contribute to open telemetry. Please let me know.
I am currently working to bridge open telemetry to open tracing with one configuration. From micrometer to jaeger. Please share me your thoughts on this.

Best Regards,
Venkata Bhumireddy.

On Wed, Nov 20, 2019 at 12:17 PM Vineeth Reddy <vineethbfhs@...> wrote:
Hey!

I am new to open-telemetry, would love to meet you guys. Looking forward in contributing to OT.
Count me in for the meetup. :)

Thanks for the invite.

On Sun, Nov 17, 2019, 11:01 Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Venkata Bhumireddy
 

Hi,
I am also new and now I am in Charlotte. I can contribute to open telemetry. Please let me know.
I am currently working to bridge open telemetry to open tracing with one configuration. From micrometer to jaeger. Please share me your thoughts on this.

Best Regards,
Venkata Bhumireddy.

On Wed, Nov 20, 2019 at 12:17 PM Vineeth Reddy <vineethbfhs@...> wrote:
Hey!

I am new to open-telemetry, would love to meet you guys. Looking forward in contributing to OT.
Count me in for the meetup. :)

Thanks for the invite.

On Sun, Nov 17, 2019, 11:01 Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Vineeth Reddy
 

Hey!

I am new to open-telemetry, would love to meet you guys. Looking forward in contributing to OT.
Count me in for the meetup. :)

Thanks for the invite.

On Sun, Nov 17, 2019, 11:01 Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Reitbauer, Alois
 

Happy to join as well

 

From: <cncf-opentelemetry-community@...> on behalf of "armin.ruech via Lists.Cncf.Io" <armin.ruech=dynatrace.com@...>
Reply to: "cncf-opentelemetry-community@..." <cncf-opentelemetry-community@...>
Date: Wednesday, 20. November 2019 at 08:34
To: "cncf-opentelemetry-community@..." <cncf-opentelemetry-community@...>
Subject: Re: [opentelemetry-community] Kubecon

 

I'll join as well with four of my DT folks :)
When are we starting?

Thanks for organizing!

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313


Re: Kubecon

Armin Ruech
 

I'll join as well with four of my DT folks :)
When are we starting?

Thanks for organizing!


Re: [External] Re: [opentelemetry-community] Kubecon

Hamel, Jordan
 

Still time to join in? +1 if so

thanks

 

From: <cncf-opentelemetry-community@...> on behalf of Brian Putt <puttbrian@...>
Reply-To: "cncf-opentelemetry-community@..." <cncf-opentelemetry-community@...>
Date: Tuesday, November 19, 2019 at 2:02 PM
To: "cncf-opentelemetry-community@..." <cncf-opentelemetry-community@...>
Subject: [External] Re: [opentelemetry-community] Kubecon

 

Sure, feel free to share outward

 

I guess I'll let the Union Kitchen and Tap folks know about a 12ish person group. Let me know if you think we should plan for more.

 

On Tue, Nov 19, 2019, 12:37 Ted Young <ted@...> wrote:

I'm down for wednesday! Should we promote this elsewhere? Not everyone reads the mailing list, lol.

 

On Mon, Nov 18, 2019 at 11:35 AM Bruno Garcia <bruno@...> wrote:

Thanks for the invite! Count me in. 

On Mon, Nov 18, 2019, 10:14 Amelia Mango <amelia@...> wrote:

[Independent of festivities on Wednesday evening]

 

Anyone in town for OPS/Kubecon is also welcome to swing by The Whiskey House (https://thewhiskeyhousesd.com/) tonight from 5:30-8pm. LightStep and Google are co-hosting and there will be drinks and a taco bar :-)

 

On Sun, Nov 17, 2019 at 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:

+1 Wednesday

 

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:

Ya Wednesday works for me

 

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:

I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

 

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:

Now that Kubecon has arrived, thoughts on meeting up?

 

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

 

Thanks,

 

Brian

 

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:

Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

 

It would be great to get some time during kubecon itself to have an OTel meetup.

 

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Brian Putt
 

Sure, feel free to share outward

I guess I'll let the Union Kitchen and Tap folks know about a 12ish person group. Let me know if you think we should plan for more.


On Tue, Nov 19, 2019, 12:37 Ted Young <ted@...> wrote:
I'm down for wednesday! Should we promote this elsewhere? Not everyone reads the mailing list, lol.

On Mon, Nov 18, 2019 at 11:35 AM Bruno Garcia <bruno@...> wrote:
Thanks for the invite! Count me in. 

On Mon, Nov 18, 2019, 10:14 Amelia Mango <amelia@...> wrote:
[Independent of festivities on Wednesday evening]

Anyone in town for OPS/Kubecon is also welcome to swing by The Whiskey House (https://thewhiskeyhousesd.com/) tonight from 5:30-8pm. LightStep and Google are co-hosting and there will be drinks and a taco bar :-)

On Sun, Nov 17, 2019 at 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Ted Young
 

I'm down for wednesday! Should we promote this elsewhere? Not everyone reads the mailing list, lol.


On Mon, Nov 18, 2019 at 11:35 AM Bruno Garcia <bruno@...> wrote:
Thanks for the invite! Count me in. 

On Mon, Nov 18, 2019, 10:14 Amelia Mango <amelia@...> wrote:
[Independent of festivities on Wednesday evening]

Anyone in town for OPS/Kubecon is also welcome to swing by The Whiskey House (https://thewhiskeyhousesd.com/) tonight from 5:30-8pm. LightStep and Google are co-hosting and there will be drinks and a taco bar :-)

On Sun, Nov 17, 2019 at 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Bruno Garcia
 

Thanks for the invite! Count me in. 


On Mon, Nov 18, 2019, 10:14 Amelia Mango <amelia@...> wrote:
[Independent of festivities on Wednesday evening]

Anyone in town for OPS/Kubecon is also welcome to swing by The Whiskey House (https://thewhiskeyhousesd.com/) tonight from 5:30-8pm. LightStep and Google are co-hosting and there will be drinks and a taco bar :-)

On Sun, Nov 17, 2019 at 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Morgan McLean
 

+1 for Wednesday and for the 5:30 to 8:00 PM at the Whiskey House that Amelia mentioned

On Mon, Nov 18, 2019 at 10:14 AM Bogdan Drutu via Lists.Cncf.Io <bdrutu=google.com@...> wrote:
+1 to meet :) you pick the time that works best for everyone :)

On Sun, Nov 17, 2019, 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Bogdan Drutu
 

+1 to meet :) you pick the time that works best for everyone :)


On Sun, Nov 17, 2019, 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Amelia Mango
 

[Independent of festivities on Wednesday evening]

Anyone in town for OPS/Kubecon is also welcome to swing by The Whiskey House (https://thewhiskeyhousesd.com/) tonight from 5:30-8pm. LightStep and Google are co-hosting and there will be drinks and a taco bar :-)


On Sun, Nov 17, 2019 at 4:59 PM Yuri Shkuro via Lists.Cncf.Io <ys=uber.com@...> wrote:
+1 Wednesday

On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.


Re: Kubecon

Yuri Shkuro
 

+1 Wednesday


On Sun, Nov 17, 2019 at 4:52 PM Brian Putt <puttbrian@...> wrote:
Ya Wednesday works for me

On Sun, Nov 17, 2019, 13:46 Tyler Benson <tyler@...> wrote:
I think the expo hall is open pretty late on Tuesday so can I suggest Wednesday instead?

On Sun, Nov 17, 2019 at 11:01 AM Brian Putt <puttbrian@...> wrote:
Now that Kubecon has arrived, thoughts on meeting up?

I'll suggest Tuesday at 730pm at the UNION Kitchen & Tap Gaslamp

Thanks,

Brian

On Wed, Sep 4, 2019 at 12:54 PM Ted Young <ted@...> wrote:
Shameless plug: I'm throwing the Observability Summit as a co-located event on Nov 18th. If you're going to Kubecon, you should attend! https://observabilitysummit.com. Lots of good discussion happening there.

It would be great to get some time during kubecon itself to have an OTel meetup.

On Wed, Sep 4, 2019 at 8:11 AM Bruno Garcia <bruno@...> wrote:

I'm attending KubeCon and would really like to meet and discuss that.