Date   

Re: Thoughts on KubeCon

justinsb@...
 

I really like the checkbox (or slider) suggestion - I think it could be very helpful all around.  I've helped review some submissions, and I try to apply consistent criteria.  I based my criteria on the instructions, but there's certainly room for interpretation particularly when it comes to how to weight criteria.  So it would probably both help me review faster and score more accurately if it was "+1 for OSS, +1 for end-user, -2 for content itself = average" type thing, and this could likely both let the track assemblers reweight these criteria to give balance to a track, and also provide some simple actionable feedback for submitters (e.g. "your strongest factor was that the topic was interesting, but the weakest factor was that people worried it was going to turn into a sales pitch").


Re: Thoughts on KubeCon

Andrew Martin
 
Edited

Firstly I am more than happy with the conference tracks, I always learn a huge amount and take away a lot more for study. 

A few thoughts from discussing this thread with organisers of other major (non-LF) conferences:

- possibly having review spreadsheet checkboxes for "abstract doesn't clarify the talk's content", or "good abstract but a better one was selected, please apply again", "huge number of submissions in this area" etc could ease the burden of personalised, individual review. Small extra detail in rejection responses may serve to encourage first-time submitters/serial rejectees
- helpmeabstract.com could help as one of these checkbox recommendations for unclear or unspecific abstracts
- in some cases the natural bias of single blind submissions is desirable, and a good abstract is not necessarily correlated with a quality presentation. My personal review flow is to skim and roughly grade abstracts, then when a subject's in high contention scroll right for further information. In some cases an SME's authorship with scant abstract will outweigh a well-written abstract, as the expected depth of the SME's presentation is preferable (especially for new or emerging technologies). And observationally: in small fields such as Kubecon's per-technology tracks, double blind doesn't ensure anonymity due to writing and abstract style. YYMV

Thanks,


Andy


Re: Thoughts on KubeCon

Michael Hausenblas <mhausenb@...>
 

As an organiser of various conferences (and as someone who didn't get a
talk accepted at KubeCon this year), can I just point out that Liz and
Janet have done a fantastic job in orchestrating something as complex as
this, and I want to thank them for this.
+1000

As much as I like to contribute via reviewing, I certainly don’t envy
the chairs and no matter how rude certain people on this thread are or
think it’s OK to use whatever tone they feel entitled to, first and
foremost I have 100% trust in the chairs to carry out their duties in
an objective manner.

Cheers,
Michael

--
Michael Hausenblas, Developer Advocate
OpenShift by Red Hat
Mobile: +353 86 0215164 | Twitter: @mhausenblas
http://openshift.com | http://mhausenblas.info

-----Original Message-----
From: Daniel Bryant <db@...>
Reply: Daniel Bryant <db@...>
Date: 10 October 2018 at 10:50:59
To: CNCF TOC <cncf-toc@...>
Subject:  Re: [cncf-toc] Thoughts on KubeCon

As an organiser of various conferences (and as someone who didn't get a
talk accepted at KubeCon this year), can I just point out that Liz and
Janet have done a fantastic job in orchestrating something as complex as
this, and I want to thank them for this.

I feel that some of the mails on this list are making quite strong
accusations without much evidence or understanding of the process, and not
only does this dilute the feedback process, but it is also potentially
making the task of recruiting (and being) a co-chair much less appealing
for future events.

With the sheer volume of players (vendors, communities and individuals) in
the CNCF space, the creation of a conference program truly is a *complex*
problem, and clearly not everyone will be happy with the result.

I personally would be keen to see a public retrospective of the conference,
but this should be done after the event, and take in attendee feedback as
the highest priority. Questions like "what topics were missing at KubeCon?"
and "did you see all the technologies you expected?" will provide answers
more indicative than all of us trying to guess whether the selected talks
are representative of what the community wants to see.

Best wishes,

Daniel



On Tue, Oct 9, 2018 at 9:19 PM Matt Farina wrote:

Liz, I appreciate you’re taking time here with more details.

I want to note a difference between malicious intent and people drawing
conclusions of it. Those are different people in different situations. I,
personally, don’t think you had any malicious intent. I can see how some
might draw the conclusion, though. I can only imagine what intent people
drew from my decisions in the past. That’s why I keep thinking with a
forward eye.

Matt, BTW you left out nuclio serverless platform
I said including because I’m sure I missed more than one serverless
platform. There are oh so many of them right now.

Again I go back to my point that a lot of people (and not just from one
company)
submitting on a topic suggests that at this moment in time it's of broad
interest to the community.

There different types of people with different goals. Lots of submissions
means there’s are people, vendors in our case, who want to talk about
something to users. But, does that mean it’s useful material for the
end-users? Should end-users get to see the diversity?

I note the vendors here because Dan shared details on how few percent of
end-users even submitted to the conference. On the whole the general
sessions are vendors.

One of the Knative talks is an end-user story from T-Mobile
I really want to see if “work in progress” software is being used by
T-Mobile in production. I’m curious now :)

Kubeless was mentioned in two abstracts, one of which is an accepted
talk and the other really didn't get a great score.

Can you point me to the session on this? Searching the schedule for
kubeless turned up speaders with it in their bios but not in the sessions
themselves. I’m curious to see what we have and what track that landed on.

just for stats, for the ServerlessNYC event in 2 weeks (1 day we
organized)
we got about 50 submission, only 2 on Knative (one accepted)
What does this say about CloudNativeCon and it’s interaction with
developers and the outside world (where the end-users are)? Are our
submissions too much on our island of ideas and topics?

Let's say that JaneSchmoeNativeTool IS better than GNativeTool.
Even if Jane Schmoe DOES get a talk at Kubecon, will it really make a
difference to adoption under these circumstances?
This is a wonderful question. Consider the room of attendees going to this
track. If they don’t hear about other options will they ever try them?
Should that room full of attendees hear about a diverse set of solutions?

Back to this question, in practice presentations at conferences like this
can have an impact. That’s part of the reason why there was a push for
knative sessions here.

To circle back to something Dan said,

The reality is than an organization like CNCF has many constituencies,
including our members, the TOC, our project maintainers, our end users,
developers considering using or contributing to our projects, and others.
knative is really an interesting case for this system. It’s not a CNCF
project. Does that mean it’s not one of our projects when our is the CNCF?
That means having so much of it isn’t in support of our project maintainers
or of getting devs to consider using or contributing to our projects… right?

It’s work in progress rather than being released so it’s not ready for
most end-users. There are alternative solutions that are 1.0 released and
ready for production. So, how does knative benefit end-users this year at
the con?

Yet, a group of vendors is pushing it to make it hot.

Just thinking out loud. knative is a really good test to think out loud on.



--
Matt Farina
mattfarina.com







Re: Thoughts on KubeCon

Daniel Bryant
 

As an organiser of various conferences (and as someone who didn't get a talk accepted at KubeCon this year), can I just point out that Liz and Janet have done a fantastic job in orchestrating something as complex as this, and I want to thank them for this. 

I feel that some of the mails on this list are making quite strong accusations without much evidence or understanding of the process, and not only does this dilute the feedback process, but it is also potentially making the task of recruiting (and being) a co-chair much less appealing for future events.

With the sheer volume of players (vendors, communities and individuals) in the CNCF space, the creation of a conference program truly is a *complex* problem, and clearly not everyone will be happy with the result. 

I personally would be keen to see a public retrospective of the conference, but this should be done after the event, and take in attendee feedback as the highest priority. Questions like "what topics were missing at KubeCon?" and "did you see all the technologies you expected?" will provide answers more indicative than all of us trying to guess whether the selected talks are representative of what the community wants to see.

Best wishes,

Daniel



On Tue, Oct 9, 2018 at 9:19 PM Matt Farina <matt@...> wrote:
Liz, I appreciate you’re taking time here with more details.

I want to note a difference between malicious intent and people drawing conclusions of it. Those are different people in different situations. I, personally, don’t think you had any malicious intent. I can see how some might draw the conclusion, though. I can only imagine what intent people drew from my decisions in the past. That’s why I keep thinking with a forward eye.

> Matt, BTW you left out nuclio serverless platform

I said including because I’m sure I missed more than one serverless platform. There are oh so many of them right now.

> Again I go back to my point that a lot of people (and not just from one company)
> submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

There different types of people with different goals. Lots of submissions means there’s are people, vendors in our case, who want to talk about something to users. But, does that mean it’s useful material for the end-users? Should end-users get to see the diversity?

I note the vendors here because Dan shared details on how few percent of end-users even submitted to the conference. On the whole the general sessions are vendors.

> One of the Knative talks is an end-user story from T-Mobile

I really want to see if “work in progress” software is being used by T-Mobile in production. I’m curious now :)

> Kubeless was mentioned in two abstracts, one of which is an accepted talk and the other really didn't get a great score. 

Can you point me to the session on this? Searching the schedule for kubeless turned up speaders with it in their bios but not in the sessions themselves. I’m curious to see what we have and what track that landed on.

> just for stats, for the ServerlessNYC event in 2 weeks (1 day we organized)
> we got about 50 submission, only 2 on Knative (one accepted)

What does this say about CloudNativeCon and it’s interaction with developers and the outside world (where the end-users are)? Are our submissions too much on our island of ideas and topics?

Let's say that JaneSchmoeNativeTool IS better than GNativeTool.
> Even if Jane Schmoe DOES get a talk at Kubecon, will it really make a
> difference to adoption under these circumstances?

This is a wonderful question. Consider the room of attendees going to this track. If they don’t hear about other options will they ever try them? Should that room full of attendees hear about a diverse set of solutions?

Back to this question, in practice presentations at conferences like this can have an impact. That’s part of the reason why there was a push for knative sessions here.

To circle back to something Dan said,

> The reality is than an organization like CNCF has many constituencies,
> including our members, the TOC, our project maintainers, our end users,
> developers considering using or contributing to our projects, and others.

knative is really an interesting case for this system. It’s not a CNCF project. Does that mean it’s not one of our projects when our is the CNCF? That means having so much of it isn’t in support of our project maintainers or of getting devs to consider using or contributing to our projects… right?

It’s work in progress rather than being released so it’s not ready for most end-users. There are alternative solutions that are 1.0 released and ready for production. So, how does knative benefit end-users this year at the con?

Yet, a group of vendors is pushing it to make it hot.

Just thinking out loud. knative is a really good test to think out loud on.



-- 
Matt Farina
mattfarina.com




Re: Thoughts on KubeCon

Matt Farina
 

Liz, I appreciate you’re taking time here with more details.

I want to note a difference between malicious intent and people drawing conclusions of it. Those are different people in different situations. I, personally, don’t think you had any malicious intent. I can see how some might draw the conclusion, though. I can only imagine what intent people drew from my decisions in the past. That’s why I keep thinking with a forward eye.

> Matt, BTW you left out nuclio serverless platform

I said including because I’m sure I missed more than one serverless platform. There are oh so many of them right now.

> Again I go back to my point that a lot of people (and not just from one company)
> submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

There different types of people with different goals. Lots of submissions means there’s are people, vendors in our case, who want to talk about something to users. But, does that mean it’s useful material for the end-users? Should end-users get to see the diversity?

I note the vendors here because Dan shared details on how few percent of end-users even submitted to the conference. On the whole the general sessions are vendors.

> One of the Knative talks is an end-user story from T-Mobile

I really want to see if “work in progress” software is being used by T-Mobile in production. I’m curious now :)

> Kubeless was mentioned in two abstracts, one of which is an accepted talk and the other really didn't get a great score. 

Can you point me to the session on this? Searching the schedule for kubeless turned up speaders with it in their bios but not in the sessions themselves. I’m curious to see what we have and what track that landed on.

> just for stats, for the ServerlessNYC event in 2 weeks (1 day we organized)
> we got about 50 submission, only 2 on Knative (one accepted)

What does this say about CloudNativeCon and it’s interaction with developers and the outside world (where the end-users are)? Are our submissions too much on our island of ideas and topics?

Let's say that JaneSchmoeNativeTool IS better than GNativeTool.
> Even if Jane Schmoe DOES get a talk at Kubecon, will it really make a
> difference to adoption under these circumstances?

This is a wonderful question. Consider the room of attendees going to this track. If they don’t hear about other options will they ever try them? Should that room full of attendees hear about a diverse set of solutions?

Back to this question, in practice presentations at conferences like this can have an impact. That’s part of the reason why there was a push for knative sessions here.

To circle back to something Dan said,

> The reality is than an organization like CNCF has many constituencies,
> including our members, the TOC, our project maintainers, our end users,
> developers considering using or contributing to our projects, and others.

knative is really an interesting case for this system. It’s not a CNCF project. Does that mean it’s not one of our projects when our is the CNCF? That means having so much of it isn’t in support of our project maintainers or of getting devs to consider using or contributing to our projects… right?

It’s work in progress rather than being released so it’s not ready for most end-users. There are alternative solutions that are 1.0 released and ready for production. So, how does knative benefit end-users this year at the con?

Yet, a group of vendors is pushing it to make it hot.

Just thinking out loud. knative is a really good test to think out loud on.



-- 
Matt Farina
mattfarina.com




Re: Thoughts on KubeCon

Steven Dake
 

Nick,

 

In my experience, airtime at a major conference is a huge factor in developer growth for a project.  Of the several projects I’ve started personally (heat, magnum, kolla), we saw 3x core developer growth within 2 weeks of any given 40 minute speaking slot at a major conference.  Growth was pretty equal across the board independent of the project.

 

Selecting talks in a track is equivalent to selecting which projects grow needed corporate and individual contributor developers.

 

Note I’m biased.  I personally am super bullish on knative, but I also strongly feel KubeCon should not make kings as per governance documentation.

 

I have also served as a track chair and understand the difficulty involved that Liz has pointed out.

 

Both perspectives should be evaluated.

 

Cheers

-steve

 

 

From: <cncf-toc@...> on behalf of Nick Chase <nchase@...>
Date: Tuesday, October 9, 2018 at 12:29 PM
To: "cncf-toc@..." <cncf-toc@...>
Subject: Re: [cncf-toc] Thoughts on KubeCon

 

I'm going to say that I agree with you here, but I'm going to play devil's advocate for just a moment.

Let's say that JaneSchmoeNativeTool IS better than GNativeTool.  Even if Jane Schmoe DOES get a talk at Kubecon, will it really make a difference to adoption under these circumstances?

Not saying we don't have a problem, just saying that KubeCon talks are a necessary, but not sufficient solution.

----  Nick

 

On 10/9/2018 3:21 PM, Ruben Orduz wrote:

For the sake of my point let's remove "Google" altogether and replace with "HugeVendor" not to be pointing fingers since they are doing what any vendor would do in their position of influence.

 

HugeVendor invests/works on GNativeTool. They push it hard through their marketing channels (developer and otherwise), get 5K github stars just by inertia, suddenly you have a large number of people submitting talks about it to all conferences. Meanwhile JaneSchmoe, inc. has been quietly working hard on her version of NativeTool for years and since she has neither the marketing budget nor the acumen to pull the market one way or the other, her product is less "popular" while perhaps having the upper hand in terms of technical and business value.

 

My point is not that HugeVendor is submitting an inordinate amount of proposals about a particular tech, what I'm saying is they have the gravitas and the market power to blow everything else out of the water and the committees are perceiving that as actual community adoption/opinion.

 

Best,

Ruben

 

On Tue, Oct 9, 2018 at 3:07 PM Liz Rice <liz@...> wrote:

Only four of the submissions on Knative were from Google! Perhaps it goes to show that a lot of other people are also interested in this technology? Again I go back to my point that a lot of people (and not just from one company) submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

 

I'm not going to dig out all the numbers on Istio but it was the same kind of thing. We can't pick talks that aren't submitted!

 

 

 

On Tue, Oct 9, 2018 at 7:43 PM Ruben Orduz <ruben@...> wrote:

I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

 

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

 

Best,

Ruben 

 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:

Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

 

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

 

But, there are a couple other ways to look at this situation…

 

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

 

Second, there is from the perspective of people proposing sessions.

 

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

 

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

 

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

 

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

 

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

 

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

 

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

 

This all leads me to other questions...

 

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

 

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

 

If some of the intent and goal components could be ironed out it would help future decision makers.

 

 

 

-- 
Matt Farina
mattfarina.com

 

 



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

 

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 

 

 

 

 

On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon

 

 

Here is a summary of the discussion so far:

 

 

-- 

Liz Rice

@lizrice  | lizrice.com +44 (0) 780 126 1145

 

--

Liz Rice

@lizrice  | lizrice.com | +44 (0) 780 126 1145

 

--
Nick Chase, Head of Technical and Marketing Content, Mirantis
Editor in Chief, Open Cloud Digest Author, Machine Learning for Mere Mortals


Re: Thoughts on KubeCon

Yaron Haviv <yaronh@...>
 

just for stats, for the ServerlessNYC event in 2 weeks (1 day we organized) we got about 50 submission, only 2 on Knative (one accepted)

also while most submissions covered real user stories based on experience, those two were entirely Theoretical

this further emphasize that the current decision criteria favor some big pocket companies, and KC users are less likely to to learn about other proven or more innovative approaches without the same marketing/brand power

Yaron


From: Ruben Orduz <ruben@...>
Sent: Tuesday, October 9, 2018 9:43:09 PM
To: Matt Farina
Cc: Liz Rice; Yaron Haviv; Dan Kohn; CNCF TOC
Subject: Re: [cncf-toc] Thoughts on KubeCon
 
I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice
@lizrice  | lizrice.com +44 (0) 780 126 1145


Re: Thoughts on KubeCon

Yaron Haviv <yaronh@...>
 

Liz,

i assume you know how things work, a company doesn't need to make all the submission it self to push an agenda

your role is to provide diversity, not just in speakers but also in topics, its just insane were we got to, even worse that it haven't raised any red flag when u made a decision to make one unproven tech dominate 80% of the track

i can add to Ruben examples the over focus on KubeFlow at EU which was pretty much at it's infancy at that point (btw guess which co is behind KF)

Yaron



From: Liz Rice
Sent: Tuesday, October 9, 10:07 PM
Subject: Re: [cncf-toc] Thoughts on KubeCon
To: Ruben Orduz
Cc: Matt Farina, Yaron Haviv, Dan Kohn, CNCF TOC


Only four of the submissions on Knative were from Google! Perhaps it goes to show that a lot of other people are also interested in this technology? Again I go back to my point that a lot of people (and not just from one company) submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

I'm not going to dig out all the numbers on Istio but it was the same kind of thing. We can't pick talks that aren't submitted!



On Tue, Oct 9, 2018 at 7:43 PM Ruben Orduz <ruben@...> wrote:
I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

knativeOpenFaaSKubelessFissionBrigadeVirtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:
Dan,
 
looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process
how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?
 
Yaron
 
From: cncf-toc@... <cncf-toc@...On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
Subject: Re: [cncf-toc] Thoughts on KubeCon

 
Here is a summary of the discussion so far:
 
--
Dan Kohn <dan@...>
Executive Director, Cloud Native Computing Foundation https://www.cncf.io

-- 
Liz Rice
@lizrice  | lizrice.com | +44 (0) 780 126 1145

--
Liz Rice
@lizrice  | lizrice.com | +44 (0) 780 126 1145



Re: Thoughts on KubeCon

Nick Chase
 

I'm going to say that I agree with you here, but I'm going to play devil's advocate for just a moment.

Let's say that JaneSchmoeNativeTool IS better than GNativeTool.  Even if Jane Schmoe DOES get a talk at Kubecon, will it really make a difference to adoption under these circumstances?

Not saying we don't have a problem, just saying that KubeCon talks are a necessary, but not sufficient solution.

----  Nick


On 10/9/2018 3:21 PM, Ruben Orduz wrote:
For the sake of my point let's remove "Google" altogether and replace with "HugeVendor" not to be pointing fingers since they are doing what any vendor would do in their position of influence.

HugeVendor invests/works on GNativeTool. They push it hard through their marketing channels (developer and otherwise), get 5K github stars just by inertia, suddenly you have a large number of people submitting talks about it to all conferences. Meanwhile JaneSchmoe, inc. has been quietly working hard on her version of NativeTool for years and since she has neither the marketing budget nor the acumen to pull the market one way or the other, her product is less "popular" while perhaps having the upper hand in terms of technical and business value.

My point is not that HugeVendor is submitting an inordinate amount of proposals about a particular tech, what I'm saying is they have the gravitas and the market power to blow everything else out of the water and the committees are perceiving that as actual community adoption/opinion.

Best,
Ruben

On Tue, Oct 9, 2018 at 3:07 PM Liz Rice <liz@...> wrote:
Only four of the submissions on Knative were from Google! Perhaps it goes to show that a lot of other people are also interested in this technology? Again I go back to my point that a lot of people (and not just from one company) submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

I'm not going to dig out all the numbers on Istio but it was the same kind of thing. We can't pick talks that aren't submitted!



On Tue, Oct 9, 2018 at 7:43 PM Ruben Orduz <ruben@...> wrote:
I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice

--
Liz Rice
@lizrice  | lizrice.com+44 (0) 780 126 1145

--
Nick Chase, Head of Technical and Marketing Content, Mirantis
Editor in Chief, Open Cloud Digest Author, Machine Learning for Mere Mortals


Re: Thoughts on KubeCon

Ruben Orduz <ruben@...>
 

For the sake of my point let's remove "Google" altogether and replace with "HugeVendor" not to be pointing fingers since they are doing what any vendor would do in their position of influence.

HugeVendor invests/works on GNativeTool. They push it hard through their marketing channels (developer and otherwise), get 5K github stars just by inertia, suddenly you have a large number of people submitting talks about it to all conferences. Meanwhile JaneSchmoe, inc. has been quietly working hard on her version of NativeTool for years and since she has neither the marketing budget nor the acumen to pull the market one way or the other, her product is less "popular" while perhaps having the upper hand in terms of technical and business value.

My point is not that HugeVendor is submitting an inordinate amount of proposals about a particular tech, what I'm saying is they have the gravitas and the market power to blow everything else out of the water and the committees are perceiving that as actual community adoption/opinion.

Best,
Ruben

On Tue, Oct 9, 2018 at 3:07 PM Liz Rice <liz@...> wrote:
Only four of the submissions on Knative were from Google! Perhaps it goes to show that a lot of other people are also interested in this technology? Again I go back to my point that a lot of people (and not just from one company) submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

I'm not going to dig out all the numbers on Istio but it was the same kind of thing. We can't pick talks that aren't submitted!



On Tue, Oct 9, 2018 at 7:43 PM Ruben Orduz <ruben@...> wrote:
I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice

--
Liz Rice
@lizrice  | lizrice.com+44 (0) 780 126 1145


Re: Thoughts on KubeCon

Liz Rice
 

Only four of the submissions on Knative were from Google! Perhaps it goes to show that a lot of other people are also interested in this technology? Again I go back to my point that a lot of people (and not just from one company) submitting on a topic suggests that at this moment in time it's of broad interest to the community. 

I'm not going to dig out all the numbers on Istio but it was the same kind of thing. We can't pick talks that aren't submitted!



On Tue, Oct 9, 2018 at 7:43 PM Ruben Orduz <ruben@...> wrote:
I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice

--
Liz Rice
@lizrice  | lizrice.com+44 (0) 780 126 1145


Re: Thoughts on KubeCon

Ruben Orduz <ruben@...>
 

I'm aware this is a bit a political minefield here, but I'm concerned the committee(s) are unintentionally choosing winners here (same for KubeCon EU Købnhavn). What I mean is this: "popularity" of a topic or tech can be driven/influenced by movers and shakers in the field. Google pushes for a tool they are working on will get much more traction than a competing tool from a small third party. A dramatic example of this phenomenon is having a whole track dedicated to Istio even though it was as yet a somewhat unproven technology on the field and far from production-ready for enterprise customers who tend to wait until a tech is more stable before deploying it. Several other service meshy-techs felt shunned by this. 

I'm getting the same feeling about knative here. Seeing the over abundance of talk proposals about it, it was perceived as a good gauge of community interest, which again, a behemoth is behind pushing it so that's no surprise.

I would posit we need to be more careful to unintentionally pick favorites based on popularity, specially when there's a huge asymmetry in terms of marketing power and community outreach among competitors in any given tech.

Best,
Ruben 

On Tue, Oct 9, 2018 at 1:46 PM Matt Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice
@lizrice  | lizrice.com +44 (0) 780 126 1145


Re: Thoughts on KubeCon

Yaron Haviv <yaronh@...>
 

liz,

its not the number of submission which should count, i guess if "theoretically" there was a large company w special interest in Knative they can make hundred submission to promote it.

We saw the same phenomena in EU, some companies rule the agenda, making it hard for smaller members to demonstrate their innovation. some of the sessions i attended you could easily see that the driving decision wasn't how qualified the speaker was or how interesting/relevant the session.

Matt, BTW you left out nuclio serverless platform

Yaron



From: Matthew Farina
Sent: Tuesday, October 9, 8:46 PM
Subject: Re: [cncf-toc] Thoughts on KubeCon
To: Liz Rice
Cc: Yaron Haviv, Dan Kohn, CNCF TOC


Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

knativeOpenFaaSKubelessFissionBrigadeVirtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina

On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:
Dan,
 
looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process
how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?
 
Yaron
 
From: cncf-toc@... <cncf-toc@...On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
Subject: Re: [cncf-toc] Thoughts on KubeCon

 
Here is a summary of the discussion so far:
 
--
Dan Kohn <dan@...>
Executive Director, Cloud Native Computing Foundation https://www.cncf.io

-- 
Liz Rice
@lizrice  | lizrice.com | +44 (0) 780 126 1145




Re: Thoughts on KubeCon

Liz Rice
 

Eughhh, reading this through I fear it might come over as defensive, but since I've collected the information, I'm going ahead regardless. Let's look at the *actual* situation on this serverless track:
  • The "Jupyter" talk is titled "Running Serverless HPC Workloads on top of Kubernetes and Jupyter Notebooks". Which sounds pretty serverless to me. The serverless tech in question is Fn, per the abstract.
  • One of the Knative talks is an end-user story from T-Mobile
  • One of the non-Knative talks is an end-user story about a bank in Paraguay working with OpenFaaS. 
  • One of the Knative sessions is a BOF, and because it seems so hot, a BOF seems appropriate. (And before anyone asks, no, no-one suggested a broader BOF about Serverless.)
  • There were a few submissions on Fission, but the highest rated was by a speaker who had a much better submission on a different topic (which did get accepted), and the others really didn't score that well (given our high bar). 
  • Kubeless was mentioned in two abstracts, one of which is an accepted talk and the other really didn't get a great score. 
  • I can't find any mention of Brigade or Virtual Kubelet in any of these submissions. 
  • I can't meaningfully search for "serverless" the technology in this spreadsheet :-) 
Did we worry about whether this was too much Knative? Yes, we did consider it. Is it the perfect Serverless track? Probably not. Do I think the choices we made are reasonable, given the submissions and the reviewer feedback we had? Absolutely! Was there "malicious intent"? Fight me! :-)

I'm not going to go into this level of detail about anything else, I promise.


On Tue, Oct 9, 2018 at 6:46 PM Matthew Farina <matt@...> wrote:
Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice

--
Liz Rice
@lizrice  | lizrice.com+44 (0) 780 126 1145


Re: Thoughts on KubeCon

Matt Farina
 

Liz, thanks for sharing those details. I know this is a tough job. Thanks for putting up with this extra work of the questioning and people poking at the ideas here. Anything I’m suggesting is more about clarifying for future conferences and trying to be explicit where we may not have been before.

I completely understand the desire to identify hot technologies. With 2/3 of the proposed talks on one technology it speaks to a level of hotness.

But, there are a couple other ways to look at this situation…

First, there is as a track attendee. 4 of 6 presentations on the same technology is not exciting and does not give me a diverse view. For someone not in the know it gives the impression that the space is not very diverse and that the main piece of technology is “work-in-progress” (the label on knative). Is this the impression we want conference attendees to have?

Second, there is from the perspective of people proposing sessions.

For Kubernetes there are currently numerous serverless technologies including, but not limited to:

  • knative
  • OpenFaaS
  • Kubeless
  • Fission
  • Brigade
  • Virtual Kubelet (works with serverless containers like ACI, Fargate, etc but is not FaaS)

Jupyter bills itself as a web application and notebook. It’s getting a lot of buzz but I’ve not heard of it being billed as Serverless.

There are also tools like serverless that can work with numerous technologies including kubeless (on this list) that are workflow solutions.

In addition there are things the CNCF serverless working group has been working on like cloudevents (which has an intro and deep dive out of band from the serverless track).

The serverless track then has 4 of 6 session on knative, 1 of 6 on something else (Jupyter), and sessions on other serverless technologies being rejected.

Can we all see how decisions here could be interpreted with malicious intent and how it could put a negative view on the conference and decision making process? Whether it happened that way or not, people could come to malicious conclusions.

This all leads me to other questions...

Do we want end-user presentations in this space? Since knative is hot but not ready for production some other technology would be used by them. But, it’s useful today and not “hot”. How do we encourage end-users to present here? Is “hot” or useful today more important?

Is a goal diversity? If so, mirroring presentations with only those that are hot doesn’t provide for diversity.

If some of the intent and goal components could be ironed out it would help future decision makers.



-- 
Matt Farina
mattfarina.com



On Oct 9, 2018, at 1:26 PM, Liz Rice <liz@...> wrote:

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon


 

Here is a summary of the discussion so far:



-- 
Liz Rice
@lizrice  | lizrice.com +44 (0) 780 126 1145


Re: Thoughts on KubeCon

Liz Rice
 

Matt, thank you for your thoughtful response. I like your list and your focus on identifying solutions for things that need to be improved. 

Yaron, by my very quick reckoning in a rather complicated spreadsheet: of ~60 submissions under Serverless, ~40 of them mentioned Knative. If number of submissions has some rough correlation to "what the community is currently interested in" (and I believe it does) then Knative is currently very hot, and we have tried to reflect this in the agenda. There's actually a seventh talk from the Serverless list that we accepted into the Observability track because we felt it straddled both topics. 




On Tue, Oct 9, 2018 at 5:15 PM Yaron Haviv <yaronh@...> wrote:

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...


Subject: Re: [cncf-toc] Thoughts on KubeCon

 

Here is a summary of the discussion so far:

--
Liz Rice
@lizrice  | lizrice.com+44 (0) 780 126 1145


Re: Thoughts on KubeCon

Yaron Haviv <yaronh@...>
 

Dan,

 

looking at the schedule, the fact that out of total 6 sessions in the Serverless track there are 4 talks about Knative raises a serious question about the bias of this process

how come the only other two sessions are on OpenFaaS and Jupyter (serverless? really)  and other efforts in the space are left in the cold ?

 

Yaron

 

From: cncf-toc@... <cncf-toc@...> On Behalf Of Dan Kohn
Sent: Monday, October 8, 2018 23:35
To: cncf-toc@...
Subject: Re: [cncf-toc] Thoughts on KubeCon

 

Here is a summary of the discussion so far:

 


Re: Thoughts on KubeCon

Dan Kohn <dan@...>
 

On Tue, Oct 9, 2018 at 11:14 AM Matt Farina <matt@...> wrote:
 
if TLF is there for the benefit of it’s members than shouldn’t most feel they are on a level playing field in this 501c6?
 
If TLF is a 501c6 for the benefit of it’s members than shouldn’t we look at a setup that benefits all of the members.

These were just two asides in a long and thoughtful comment, but I did want to address that employment with a CNCF member has never played a factor in what talks have been accepted to KubeCon + CloudNativeCon.

The LF (and the CNCF as its project) is a 501(c)(6), but it's not correct that we just exist for the benefit of our members. The reality is than an organization like CNCF has many constituencies, including our members, the TOC, our project maintainers, our end users, developers considering using or contributing to our projects, and others. Those are also some of the constituencies we're aiming to satisfy with KubeCon + CloudNativeCon.
--
Dan Kohn <dan@...>
Executive Director, Cloud Native Computing Foundation https://www.cncf.io
+1-415-233-1000 https://www.dankohn.com


Re: Thoughts on KubeCon

Matt Farina
 

If we don’t have problems we’re trying to solve or things we’re trying to improve, how can we review the proposed solutions against the issues? Wouldn’t it be easy to start having color of the bike shed conversations?

With that in mind I wanted to break out some of the issues I saw in the comments so far. And then get into some solutions.

1) We have too low of an acceptance rate for talks

There are so many proposed talks and only a small percentage are accepted. This makes the whole space around the talks highly competitive. Especially since speaking helps with project uptake, kicking off new projects, and career advancement (because let’s be honest, people use them for this).

Can we appreciate that this is a problem born from success? Kudos to the people who’ve organized the conferences to get us here.

With a low rate of acceptance we also end up with a high rate of rejection. This leads to hurt feelings and wondering what it takes to get in.

2) There is concern that one or a couple vendors will have an outsized presence compared to their contemporaries.

If we’re honest, many try to game systems to their advantage. Sometimes we’re even more upset that a competitor did a better job at it than us. But, if TLF is there for the benefit of it’s members than shouldn’t most feel they are on a level playing field in this 501c6?

3) We want more end user talks

The best form of advertising is word of mouth. One end user sharing with another end user. End users talking also helps vendors and project developers hear what works well and what needs improving. There are many reasons end user talks are good for the ecosystem.

4) We want to ensure a high quality bar on the KubeCon/CloudNativeCon talks

If the number of talks accepted is low we want to make sure the quality is high.

What can we do to improve these? Here are some ideas from the conversations (and that I’ve pulled from other conferences)…

A) Have camps in addition to cons. In local cities enable people to self organize camps. This provides 3 benefits:

  1. It opens up content to people in the local areas, many of whom won’t attend a con. That expands the message radius.
  2. Camps provide more speaking opportunities in the cloud native space. People can still get their message out in talks.
  3. It provides a sort of “minor leagues”, to use a sports analogy, where speakers can work on their skills and test out ideas. More practice leads to better talks later which helps the quality bar.

WordCamps, by the WordPress community, provide a nice example of what these can look like. OpenStack, Drupal, and others have done this well, too.

B) Limit the number of general session spots per vendor (excluding project intros, deep dives, lightening talks, etc).

Fairness is hard to judge. If TLF is a 501c6 for the benefit of it’s members than shouldn’t we look at a setup that benefits all of the members. For example, if the number was limited to 20 general sessions per vendor in the current con only one company would have been impacted and if limited to 15 only 2 companies would have been impacted.

If members are concerned with one or two companies having and outsized impact on general sessions there are ways to handle that. This combined with (A) provides a way to limit the general sessions at KubeCon/CloudNativeCon while still providing an opportunity for those vendor presentations to be heard.

C) When proposed sessions are submitted collect more information.

What should this information be? Should it be academic level information? Someone who can write up a complex and long idea isn’t necessarily someone who is a good speaker or presenter. Instead of trying to turn this into an academic conference, which I do enjoy, I would prefer keeping the focus on end-user enablement. Enabling end users helps our vendors and projects go further and be more successful.

Instead, I would collect:

  • A list of benefits people who attend will walk away with. This puts the focus on the audience and enabling them
  • A list of previous places the speaker has already presented along with asking, optionally, for links to video. This will help people evaluate someones speaking capability
  • A field for additional comments for the reviewers

This additional information will help reviewers make decisions. For example, if two people want to give a very similar talk their ability to speak to a crowd may come into play in judging the quality of the final presentation. This does help already proven speakers which is why the camps and the next suggestion are important.

The 900 character field has a place. We need to have something to share in the schedule that goes just beyond skimming.

D) Teach people to present

Being a good speaker is hard. When I first started speaking it was rough. I might only put in a few hours into prepping and then just wing it. When I heard of people putting in 40 hours of prep for a 1 hour presentation and I was shocked. But, their presentations were always better than mine. I eventually read books and learned to speak and learned that many things are not obvious.

So, what about offering a free online course on presenting targeted at speaking at cons and camps. Other conferences have done this so it’s not a new idea. This can, also, help with the quality level by helping the people with a good idea learn how to present it well.

E) Provide feedback on all submissions

The first time I was a reviewer at an academic style conference I was surprised that I had to rate and give feedback on all 30 of the sessions I had to review and each of them was 3 pages long. It was a fair amount of work. As someone who had talks not get accepted to the same conference I found the feedback to be invaluable.

This will help people know what happened. It’s more work for reviewers. But, it gives reviewers a different angle to review the presentation and it’s kinder to people who submitted.

F) Actively encourage more end-users to speak

When we see someone who has a good story to tell we should identify them and encourage them to speak about it. We could even organize around this type of thing and talk about it.




Sorry this got to be a little long. This is me restructuring our conversation and throwing a few things at the wall to see if anything sticks. Feel free to pick it apart…. or not.

-- 
Matt Farina
mattfarina.com




Re: Thoughts on KubeCon

Michael Hausenblas <mhausenb@...>
 

Background: I was in academia/research for 12y+, submitted hundreds of
papers, reviewed even more and was serving on many dozens of PC,
organizing workshops, serving as general chair and program chair,
yadayada … yawn …

The number one thing I liked about industry conferences, especially
after I moved from research to industry (struggling to get my PhD and
master student’s papers accepted) was that: 1. industry conferences
focus on sharing knowledge, lessons learned while academia focuses on
where you made a mistake (or: I’ve done that same research 20 years
ago, where’s the improvement), and 2. the lack of structural and
formal review processes.

Let me be very clear on this: blind, double blind, triple blind, feel
free to do whatever you *think* makes sense. The only thing I’m rather
certain would help if we’d get rid of the compartementalization, that
is, rather than reviewing my little corner at KubeCon (serverless,
machine learning, what have you), let *all* reviewers access *all*
submissions. This model works very well for O’Reilly (where I’ve been
reviewing for Strata and Velocity for years) and gives you way more
objective results, since it cancels out the bias across the reviews
and the reviewers.


Cheers,
Dr. Michael Hausenblas
(sorry, couldn’t resist ;)

--
Michael Hausenblas, Developer Advocate
OpenShift by Red Hat
Mobile: +353 86 0215164 | Twitter: @mhausenblas
http://openshift.com | http://mhausenblas.info

-----Original Message-----
From: Dan Kohn <dan@...>
Reply: Dan Kohn <dan@...>
Date: 8 October 2018 at 21:35:44
To: cncf-toc@... <cncf-toc@...>
Subject:  Re: [cncf-toc] Thoughts on KubeCon

Here is a summary of the discussion so far:

https://docs.google.com/document/d/1sDXfk5MHAmHZVdIx1t4PREo_SSXKcloCOUYjZIo4jBs/
--
Dan Kohn
Executive Director, Cloud Native Computing Foundation https://www.cncf.io
+1-415-233-1000 https://www.dankohn.com



4661 - 4680 of 7194