|
I'm strongly in favour of additional, community level mini conferences, at the scale of promcon for example. Having those could justify some rules aimed at improving the mega conferences.
On Thu, 4 Oct 2018, 06:36 Mark Coleman, < mark@...> wrote: I agree that asking for more talk information up front would be useful.
I also agree that we should be considering what is important to a conference attendee.
Chiradeep, there are plans to run one day events but I don’t have more information to hand.
Perhaps Dan can help?
Is it feasible at all to hold more Kubecons? Minikubecons? The mini version would be more like a day-long meetup, I guess, but with a little bit of backing from CNCF (marketing). Wouldn’t expect people to fly over from all over the world
to attend though.
This is like autoscaling for conferences. For emerging /exciting areas it is natural that there are a lot of submissions, and for things to change a lot. By the time a conference presentation actually happens, a lot of information could
be obsolete / uninteresting. Mini-conferences would have a shorter time between CFP and conference day. The CFP acceptance rate, I expect, would be closer to 100%.
--
Chiradeep Vittal
The point of double-blind is not to increase diversity, it is to improve quality.
Turns out bias leads people to select based on things other than the quality of the work.
On Wed, Oct 3, 2018 at 4:21 PM Alex Clemmer <alex@...> wrote:
I think it's important that tactical measures (e.g., double-blind, vendor talk limits, etc.)
should be in the service of a general goal. IMO the first responsibility of conference organizers is to the conference attendees. A primary goal might look lie: make an engaging and useful conference,
that fosters community development of cloud native software.
There is always a push among talk proposers to have a completely "objective" admissions process, but (1) an "objective" admissions process may well make for a worse conference, and (2) it is impossible for the committee to not impose some editorial
view in the talks they select. So you may as well embrace it.
To the issue at hand: IME the best reason to have double-blind reviews is that it increases the amount of diversity, both in submissions and in balance of accepts. I reckon this is for much the same way that blind auditions helped diversity in orchestras.
What I very seriously doubt is that double-blind reviews will have the affect Bryan seems to think it will. In practice, most talks proposed about interesting technology (linkerd/Istio/Envoy, say) will be inextricably linked to the vendors that produce the
technology, and the authoritative credibility they carry. Likewise most "customer success" talks will be linked to the customer itself. It will be harder on balance to judge talks on their merit without some idea of the "believability" of these authors. This
is in contrast to scientific papers, where the submission process is geared towards papers that explore some idea abstract the people involved, excepting talks from industry, which will carry more weight because they
have operational expertise, even though they are therefore less anonymous.
There may be other reasons to do double-blind review, but personally I can't think of them.
On Wed, Oct 3, 2018 at 2:20 PM Nick Chase <nchase@...> wrote:
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”,
“previously presented”, “duplicated submission”, “off topic" etc) to help submitters to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative
data point, all 10 of my submissions to KubeCon China and US were rejected, and none of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
I agree that asking for more talk information up front would be useful.
I also agree that we should be considering what is important to a conference attendee.
Chiradeep, there are plans to run one day events but I don’t have more information to hand.
Perhaps Dan can help?
toggle quoted message
Show quoted text
Is it feasible at all to hold more Kubecons? Minikubecons? The mini version would be more like a day-long meetup, I guess, but with a little bit of backing from CNCF (marketing). Wouldn’t expect people to fly over from all over the world
to attend though.
This is like autoscaling for conferences. For emerging /exciting areas it is natural that there are a lot of submissions, and for things to change a lot. By the time a conference presentation actually happens, a lot of information could
be obsolete / uninteresting. Mini-conferences would have a shorter time between CFP and conference day. The CFP acceptance rate, I expect, would be closer to 100%.
--
Chiradeep Vittal
The point of double-blind is not to increase diversity, it is to improve quality.
Turns out bias leads people to select based on things other than the quality of the work.
On Wed, Oct 3, 2018 at 4:21 PM Alex Clemmer <alex@...> wrote:
I think it's important that tactical measures (e.g., double-blind, vendor talk limits, etc.)
should be in the service of a general goal. IMO the first responsibility of conference organizers is to the conference attendees. A primary goal might look lie: make an engaging and useful conference,
that fosters community development of cloud native software.
There is always a push among talk proposers to have a completely "objective" admissions process, but (1) an "objective" admissions process may well make for a worse conference, and (2) it is impossible for the committee to not impose some editorial
view in the talks they select. So you may as well embrace it.
To the issue at hand: IME the best reason to have double-blind reviews is that it increases the amount of diversity, both in submissions and in balance of accepts. I reckon this is for much the same way that blind auditions helped diversity in orchestras.
What I very seriously doubt is that double-blind reviews will have the affect Bryan seems to think it will. In practice, most talks proposed about interesting technology (linkerd/Istio/Envoy, say) will be inextricably linked to the vendors that produce the
technology, and the authoritative credibility they carry. Likewise most "customer success" talks will be linked to the customer itself. It will be harder on balance to judge talks on their merit without some idea of the "believability" of these authors. This
is in contrast to scientific papers, where the submission process is geared towards papers that explore some idea abstract the people involved, excepting talks from industry, which will carry more weight because they
have operational expertise, even though they are therefore less anonymous.
There may be other reasons to do double-blind review, but personally I can't think of them.
On Wed, Oct 3, 2018 at 2:20 PM Nick Chase <nchase@...> wrote:
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”,
“previously presented”, “duplicated submission”, “off topic" etc) to help submitters to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative
data point, all 10 of my submissions to KubeCon China and US were rejected, and none of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
Chiradeep Vittal <chiradeep.vittal@...>
Is it feasible at all to hold more Kubecons? Minikubecons? The mini version would be more like a day-long meetup, I guess, but with a little bit of backing from CNCF (marketing). Wouldn’t expect people to fly over from all over the world
to attend though.
This is like autoscaling for conferences. For emerging /exciting areas it is natural that there are a lot of submissions, and for things to change a lot. By the time a conference presentation actually happens, a lot of information could
be obsolete / uninteresting. Mini-conferences would have a shorter time between CFP and conference day. The CFP acceptance rate, I expect, would be closer to 100%.
--
Chiradeep Vittal
From: <cncf-toc@...> on behalf of Bob Wise <bob@...>
Date: Thursday, October 4, 2018 at 8:23 AM
To: "alex@..." <alex@...>
Cc: "nchase@..." <nchase@...>, "quinton.hoole@..." <quinton.hoole@...>, "bryan@..." <bryan@...>, "anthony@..." <anthony@...>, CNCF TOC <cncf-toc@...>
Subject: Re: [cncf-toc] Thoughts on KubeCon
The point of double-blind is not to increase diversity, it is to improve quality.
Turns out bias leads people to select based on things other than the quality of the work.
toggle quoted message
Show quoted text
On Wed, Oct 3, 2018 at 4:21 PM Alex Clemmer < alex@...> wrote:
I think it's important that tactical measures (e.g., double-blind, vendor talk limits, etc.)
should be in the service of a general goal. IMO the first responsibility of conference organizers is to the conference attendees. A primary goal might look lie: make an engaging and useful conference,
that fosters community development of cloud native software.
There is always a push among talk proposers to have a completely "objective" admissions process, but (1) an "objective" admissions process may well make for a worse conference, and (2) it is impossible for the committee to not impose some editorial
view in the talks they select. So you may as well embrace it.
To the issue at hand: IME the best reason to have double-blind reviews is that it increases the amount of diversity, both in submissions and in balance of accepts. I reckon this is for much the same way that blind auditions helped diversity in orchestras.
What I very seriously doubt is that double-blind reviews will have the affect Bryan seems to think it will. In practice, most talks proposed about interesting technology (linkerd/Istio/Envoy, say) will be inextricably linked to the vendors that produce the
technology, and the authoritative credibility they carry. Likewise most "customer success" talks will be linked to the customer itself. It will be harder on balance to judge talks on their merit without some idea of the "believability" of these authors. This
is in contrast to scientific papers, where the submission process is geared towards papers that explore some idea abstract the people involved, excepting talks from industry, which will carry more weight because they
have operational expertise, even though they are therefore less anonymous.
There may be other reasons to do double-blind review, but personally I can't think of them.
On Wed, Oct 3, 2018 at 2:20 PM Nick Chase <nchase@...> wrote:
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”,
“previously presented”, “duplicated submission”, “off topic" etc) to help submitters to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative
data point, all 10 of my submissions to KubeCon China and US were rejected, and none of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
Yuan Chen <yuan.chen@...>
I second that. As someone who has had a long history with CS academic conferences (as both a reviewer and author), I was really surprised by the fact that we only needed to write a very short abstract (up to 900 characters)! I was wondering
how a reviewer could make a decision based on such limited information.
Also, as the effort required to write a proposal was not that much, there were a larger number of submissions. To me, the quality or outcome (those accepted proposals) should matter most, not the number of submissions.
Interestingly, we were asked to provide a lot of information about our background and experience. I couldn’t help thinking the reviewers care more about an author’s background and experience than the submission itself.
Would it be helpful to try something like an extended abstract, which can provide more information and technical content? We can use a template (e.g., problem statement, solution and results), maybe 1-2 pages.
Also, I would like to have received feedbacks on my submissions.
Thanks,
-Yuan
Principal Architect, Infrastructure
JD.com Silicon Valley R&D Center
From: <cncf-toc@...> on behalf of Bob Wise <bob@...>
Date: Wednesday, October 3, 2018 at 7:50 PM
To: "alena@..." <alena@...>
Cc: Dan Kohn <dan@...>, "skamille@..." <skamille@...>, Brian Grant <briangrant@...>, "bryan@..." <bryan@...>, CNCF TOC <cncf-toc@...>
Subject: Re: [cncf-toc] Thoughts on KubeCon
Since the number of submissions is really high, might be ok to require a more in-depth submission to provide enough context for the double-blind assessment. Fewer but better submissions seems like it would be a fine tradeoff.
toggle quoted message
Show quoted text
On Wed, Oct 3, 2018 at 5:36 PM Alena Prokharchyk < alena@...> wrote:
I'm not sure going with double blind for Kubecon talk submissions is a good idea. In academic conferences, the paper itself is a good enough justification as it includes all the information needed
to make a fair judgement. Kubecon submissions are short abstracts, and can't be judged the same way. Speaker's presentation skills, the projects he/she is involved in, the presentations given in the past should be taken into consideration. Unless we ask to
include slides and transcript of the presentation as a part of the submission, there is not enough basis to do double blind voting.
A disclaimer: some of my talks were accepted to kubecon, some were rejected. As a speaker (and I don't consider myself to be a particularly good one) I'd really like to know the reasons behind
both decisions.
From:
cncf-toc@... <cncf-toc@...> on behalf of Dan Kohn <dan@...>
Sent: Wednesday, October 3, 2018 5:29:27 PM
To: Camille Fournier
Cc: Brian Grant; Bryan Cantrell;
cncf-toc@...
Subject: Re: [cncf-toc] Thoughts on KubeCon
On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier <skamille@...> wrote:
What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
The point of double-blind is not to increase diversity, it is to improve quality. Turns out bias leads people to select based on things other than the quality of the work.
On Wed, Oct 3, 2018 at 4:21 PM Alex Clemmer < alex@...> wrote: I think it's important that tactical measures (e.g., double-blind, vendor talk limits, etc.) should be in the service of a general goal. IMO the first responsibility of conference organizers is to the conference attendees. A primary goal might look lie: make an engaging and useful conference, that fosters community development of cloud native software.There is always a push among talk proposers to have a completely "objective" admissions process, but (1) an "objective" admissions process may well make for a worse conference, and (2) it is impossible for the committee to not impose some editorial view in the talks they select. So you may as well embrace it.To the issue at hand: IME the best reason to have double-blind reviews is that it increases the amount of diversity, both in submissions and in balance of accepts. I reckon this is for much the same way that blind auditions helped diversity in orchestras.What I very seriously doubt is that double-blind reviews will have the affect Bryan seems to think it will. In practice, most talks proposed about interesting technology (linkerd/Istio/Envoy, say) will be inextricably linked to the vendors that produce the technology, and the authoritative credibility they carry. Likewise most "customer success" talks will be linked to the customer itself. It will be harder on balance to judge talks on their merit without some idea of the "believability" of these authors. This is in contrast to scientific papers, where the submission process is geared towards papers that explore some idea abstract the people involved, excepting talks from industry, which will carry more weight because they have operational expertise, even though they are therefore less anonymous.There may be other reasons to do double-blind review, but personally I can't think of them.On Wed, Oct 3, 2018 at 2:20 PM Nick Chase < nchase@...> wrote:
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”, “previously presented”, “duplicated submission”, “off topic" etc) to help submitters
to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative data point, all 10 of my submissions to KubeCon China and US were rejected, and none
of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
Since the number of submissions is really high, might be ok to require a more in-depth submission to provide enough context for the double-blind assessment. Fewer but better submissions seems like it would be a fine tradeoff.
On Wed, Oct 3, 2018 at 5:36 PM Alena Prokharchyk < alena@...> wrote:
I'm not sure going with double blind for Kubecon talk submissions is a good idea. In academic conferences, the paper itself is a good enough justification as it includes all the information needed to make a fair judgement. Kubecon submissions are short
abstracts, and can't be judged the same way. Speaker's presentation skills, the projects he/she is involved in, the presentations given in the past should be taken into consideration. Unless we ask to include slides and transcript of the presentation as a
part of the submission, there is not enough basis to do double blind voting.
A disclaimer: some of my talks were accepted to kubecon, some were rejected. As a speaker (and I don't consider myself to be a particularly good one) I'd really like to know the reasons behind both decisions.
From: cncf-toc@... <cncf-toc@...> on behalf of Dan Kohn <dan@...>
Sent: Wednesday, October 3, 2018 5:29:27 PM
To: Camille Fournier
Cc: Brian Grant; Bryan Cantrell; cncf-toc@...
Subject: Re: [cncf-toc] Thoughts on KubeCon
On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier < skamille@...> wrote:
What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
Quinton Hoole wrote: > I also think that it would be super-useful for submission rejection notices> to be accompanied by a few brief reviewer notes (e.g. “too much > marketing pitch”, “not open source”, “previously presented”, “duplicated > submission”, “off topic" etc) to help submitters to improve their chances > in future (and perhaps also clarify any possible misperceptions by > reviewers, as the submissions are by necessity brief).
I want to echo what Quinton suggested. I’ve been a reviewer at a number of conferences and this is something some conferences do and I’ve had to do. I’ve been on the recieving end of this feedback and it’s been useful.
Alex Clemmer wrote: > IMO the first responsibility of conference organizers is to the conference > attendees.
Who are the types of people we want to be attendees and what is it they need and want? Have we collected this information anywhere?
Brian Grant wrote: > There are vastly more talks submitted by project contributors than by > end users. Perhaps that should be an ask to our end-user community — > submit more talks.
What does this say about our community? Are there not enough end users? Are they there but not engaged enough?
|
|
|
I agree with Alena that single blind may make the most sense for Kubecon. Academic venues that use double blind usually do so (in part) to try to cut down on nepotism, etc. The acknowledged loss is that sometimes knowing who the presenter is can add information about the value of the work.
I do think that single blind probably makes more sense in this case, because I'm presuming that it isn't that PC members from vendor A are accepting all vendor A talks. (In academic conferences, this would be a conflict regardless so people at the same institution cannot review each others' submissions.)
Justin
On Wed, Oct 3, 2018 at 8:36 PM Alena Prokharchyk < alena@...> wrote:
I'm not sure going with double blind for Kubecon talk submissions is a good idea. In academic conferences, the paper itself is a good enough justification as it includes all the information needed to make a fair judgement. Kubecon submissions are short
abstracts, and can't be judged the same way. Speaker's presentation skills, the projects he/she is involved in, the presentations given in the past should be taken into consideration. Unless we ask to include slides and transcript of the presentation as a
part of the submission, there is not enough basis to do double blind voting.
A disclaimer: some of my talks were accepted to kubecon, some were rejected. As a speaker (and I don't consider myself to be a particularly good one) I'd really like to know the reasons behind both decisions.
From: cncf-toc@... <cncf-toc@...> on behalf of Dan Kohn <dan@...>
Sent: Wednesday, October 3, 2018 5:29:27 PM
To: Camille Fournier
Cc: Brian Grant; Bryan Cantrell; cncf-toc@...
Subject: Re: [cncf-toc] Thoughts on KubeCon
On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier < skamille@...> wrote:
What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
No I mean, of the total number of submissions made by end users, what percentage were accepted? Given that the overall rate was 13%
On Wed, Oct 3, 2018, 8:29 PM Dan Kohn < dan@...> wrote: On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier < skamille@...> wrote: What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
Alena Prokharchyk
I'm not sure going with double blind for Kubecon talk submissions is a good idea. In academic conferences, the paper itself is a good enough justification as it includes all the information needed to make a fair judgement. Kubecon submissions are short
abstracts, and can't be judged the same way. Speaker's presentation skills, the projects he/she is involved in, the presentations given in the past should be taken into consideration. Unless we ask to include slides and transcript of the presentation as a
part of the submission, there is not enough basis to do double blind voting.
A disclaimer: some of my talks were accepted to kubecon, some were rejected. As a speaker (and I don't consider myself to be a particularly good one) I'd really like to know the reasons behind both decisions.
toggle quoted message
Show quoted text
From: cncf-toc@... <cncf-toc@...> on behalf of Dan Kohn <dan@...>
Sent: Wednesday, October 3, 2018 5:29:27 PM
To: Camille Fournier
Cc: Brian Grant; Bryan Cantrell; cncf-toc@...
Subject: Re: [cncf-toc] Thoughts on KubeCon
On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier < skamille@...> wrote:
What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
On Wed, Oct 3, 2018 at 8:14 PM Camille Fournier < skamille@...> wrote: What percentage of end user talks were accepted?
27.8% of talks are from end users.
|
|
|
David Baldwin <dbaldwin@...>
It could also help if there was an option for a session with a sponsorship. We were told no when asked and after we missed the submission deadline. It wasn’t clear if there are other options for us to get in.
I know other conferences enable sessions with sponsorships sometimes depending on the level. Doesn’t address your need for more user sessions unless we co-present with our customers which is an option and there are some who are willing.
David
David Baldwin
Product Management
Splunk Inc.
Voice & Text: 510-301-4524
From: <cncf-toc@...> on behalf of "Brian Grant via Lists.Cncf.Io" <briangrant=google.com@...>
Reply-To: "briangrant@..." <briangrant@...>
Date: Wednesday, October 3, 2018 at 8:12 PM
To: Bryan Cantrill <bryan@...>
Cc: "cncf-toc@..." <cncf-toc@...>
Subject: Re: [cncf-toc] Thoughts on KubeCon
Please remember that "vendors" are also in many cases the primary contributors to CNCF projects.
I talked to one of the co-chairs. There are vastly more talks submitted by project contributors than by end users. Perhaps that should be an ask to our end-user community -- submit more talks.
toggle quoted message
Show quoted text
On Wed, Oct 3, 2018 at 12:59 PM Bryan Cantrill < bryan@...> wrote:
One per vendor might be too acute, as some vendors are doing much more than others. But having
some system that limits the number of submissions per vendor (and therefore force the vendors to adopt some process to determine their best submissions) would probably help -- and would also help address the too-low acceptance rate...
On Wed, Oct 3, 2018 at 11:54 AM Anthony Skipper <anthony@...> wrote:
I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill <bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
|
|
|
What percentage of end user talks were accepted?
Please remember that "vendors" are also in many cases the primary contributors to CNCF projects.
I talked to one of the co-chairs. There are vastly more talks submitted by project contributors than by end users. Perhaps that should be an ask to our end-user community -- submit more talks.
On Wed, Oct 3, 2018 at 12:59 PM Bryan Cantrill < bryan@...> wrote: One per vendor might be too acute, as some vendors are doing much more than others. But having some system that limits the number of submissions per vendor (and therefore force the vendors to adopt some process to determine their best submissions) would probably help -- and would also help address the too-low acceptance rate...
- Bryan
On Wed, Oct 3, 2018 at 11:54 AM Anthony Skipper < anthony@...> wrote: I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill < bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
- Bryan
|
|
|
Please remember that "vendors" are also in many cases the primary contributors to CNCF projects.
I talked to one of the co-chairs. There are vastly more talks submitted by project contributors than by end users. Perhaps that should be an ask to our end-user community -- submit more talks.
On Wed, Oct 3, 2018 at 12:59 PM Bryan Cantrill < bryan@...> wrote: One per vendor might be too acute, as some vendors are doing much more than others. But having some system that limits the number of submissions per vendor (and therefore force the vendors to adopt some process to determine their best submissions) would probably help -- and would also help address the too-low acceptance rate...
- Bryan
On Wed, Oct 3, 2018 at 11:54 AM Anthony Skipper < anthony@...> wrote: I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill < bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
- Bryan
|
|
|
I think it's important that tactical measures (e.g., double-blind, vendor talk limits, etc.) should be in the service of a general goal. IMO the first responsibility of conference organizers is to the conference attendees. A primary goal might look lie: make an engaging and useful conference, that fosters community development of cloud native software.
There is always a push among talk proposers to have a completely "objective" admissions process, but (1) an "objective" admissions process may well make for a worse conference, and (2) it is impossible for the committee to not impose some editorial view in the talks they select. So you may as well embrace it.
To the issue at hand: IME the best reason to have double-blind reviews is that it increases the amount of diversity, both in submissions and in balance of accepts. I reckon this is for much the same way that blind auditions helped diversity in orchestras.
What I very seriously doubt is that double-blind reviews will have the affect Bryan seems to think it will. In practice, most talks proposed about interesting technology (linkerd/Istio/Envoy, say) will be inextricably linked to the vendors that produce the technology, and the authoritative credibility they carry. Likewise most "customer success" talks will be linked to the customer itself. It will be harder on balance to judge talks on their merit without some idea of the "believability" of these authors. This is in contrast to scientific papers, where the submission process is geared towards papers that explore some idea abstract the people involved, excepting talks from industry, which will carry more weight because they have operational expertise, even though they are therefore less anonymous.
There may be other reasons to do double-blind review, but personally I can't think of them.
On Wed, Oct 3, 2018 at 2:20 PM Nick Chase < nchase@...> wrote:
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”, “previously presented”, “duplicated submission”, “off topic" etc) to help submitters
to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative data point, all 10 of my submissions to KubeCon China and US were rejected, and none
of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”, “previously presented”, “duplicated submission”, “off topic" etc) to help submitters
to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative data point, all 10 of my submissions to KubeCon China and US were rejected, and none
of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
I recognize that it's not always that cut-and-dried, BTW; I've been on the selection team for several conferences and sometimes it's just a matter of "there were 10 slots and you ranked #11". But not always.
|
|
|
"Quinton Hoole" <quinton.hoole@...> wrote:
> I also think that it would be super-useful for submission rejection
> notices to be accompanied by a few brief reviewer notes (e.g. “too
> much marketing pitch”, “not open source”, “previously presented”,
> “duplicated submission”, “off topic" etc) to help submitters to
> improve their chances in future (and perhaps also clarify any
Big +1
> possible misperceptions by reviewers, as the submissions are by
> necessity brief). As just one illustrative data point, all 10 of my
> submissions to KubeCon China and US were rejected, and none of the
> rejections seem explainable by any of the “how to improve the odds”
> guidelines. So I have no idea what to do differently in future.
-Doug
|
|
|
toggle quoted message
Show quoted text
From: cncf-toc@... <cncf-toc@...> on behalf of Shannon Williams <shannon@...>
Sent: Wednesday, October 3, 2018 4:19:09 PM
To: Anthony Skipper
Cc: bryan@...; Alexis Richardson via cncf-toc
Subject: Re: [cncf-toc] Thoughts on KubeCon
+1
Best Regards,
Shannon Williams
+1 (650) 521-6902
On Oct 3, 2018, at 11:54 AM, Anthony Skipper < anthony@...> wrote:
I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill < bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
- Bryan
|
|
|
toggle quoted message
Show quoted text
Thanks for the insightful and thought-provoking blog post Bryan. I missed the call yesterday, but co-incidentally had been noodling with similar thoughts recently, as, anecdotally, I’m also not convinced that we have the best submission review outcomes
today. I think that introducing the the partially double-blind review process would be a great step forward, and may well obviate the need for complicated per-vendor limits.
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”, “previously presented”, “duplicated submission”, “off topic" etc) to help submitters
to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative data point, all 10 of my submissions to KubeCon China and US were rejected, and none
of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
Q
One per vendor might be too acute, as some vendors are doing much more than others. But having
some system that limits the number of submissions per vendor (and therefore force the vendors to adopt some process to determine their best submissions) would probably help -- and would also help address the too-low acceptance rate...
- Bryan
On Wed, Oct 3, 2018 at 11:54 AM Anthony Skipper < anthony@...> wrote:
I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill < bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
- Bryan
|
|
|
Thanks for the insightful and thought-provoking blog post Bryan. I missed the call yesterday, but co-incidentally had been noodling with similar thoughts recently, as, anecdotally, I’m also not convinced that we have the best submission review outcomes
today. I think that introducing the the partially double-blind review process would be a great step forward, and may well obviate the need for complicated per-vendor limits.
I also think that it would be super-useful for submission rejection notices to be accompanied by a few brief reviewer notes (e.g. “too much marketing pitch”, “not open source”, “previously presented”, “duplicated submission”, “off topic" etc) to help submitters
to improve their chances in future (and perhaps also clarify any possible misperceptions by reviewers, as the submissions are by necessity brief). As just one illustrative data point, all 10 of my submissions to KubeCon China and US were rejected, and none
of the rejections seem explainable by any of the “how to improve the odds” guidelines. So I have no idea what to do differently in future.
Q
toggle quoted message
Show quoted text
One per vendor might be too acute, as some vendors are doing much more than others. But having
some system that limits the number of submissions per vendor (and therefore force the vendors to adopt some process to determine their best submissions) would probably help -- and would also help address the too-low acceptance rate...
- Bryan
On Wed, Oct 3, 2018 at 11:54 AM Anthony Skipper < anthony@...> wrote:
I would agree with double blind. But a max of 1 talk per vendor might also go a long way.
On Wed, Oct 3, 2018 at 2:47 PM Bryan Cantrill < bryan@...> wrote:
On the call yesterday, Dan asked me to send out my thoughts on double-blind reviewing. My e-mail quickly turned into a blog entry:
Something that I probably didn't highlight well enough in there is Kathryn McKinley's excellent piece on double-blind review:
There are certainly lots of ways to attack this problem, but I view double-blind as an essential piece -- but probably not sufficient on its own.
- Bryan
|
|
|
