|
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
|
|
|
I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
|
|
|
It does seem like a hard cut-off is going to hurt people without some further action. I can see two ways the CNCF might be able to help:
- Getting the word out. I suspect that a lot of Helm chart creation is done by folks outside of the "Helm community" and maybe the message is not reaching the wider cloud world. This isn't a criticism of the Helm community or the way it communicates, it's an indicator of how widely used the project is. I think CNCF could help push a message to this wider audience
- I'm not clear from the GH thread whether there is already a CNCF owned location that the old locations can be redirected to, but I'd be supportive of CNCF funds being used for that purpose. Is it more-or-less as simple as redirecting URLs from google-funded resources to community-funded resources?
Does Artifact Hub have a role to play here?
Finally, I can't be the first person to think this thought, but is there some possible way to contact chart owners and tell them about the imminent demise of the current buckets?
Liz
On Wed, 14 Oct 2020 at 07:45, Arvind Gupta < agupta@...> wrote: I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
|
|
|
On Wed, Oct 14, 2020 at 12:51 AM Liz Rice < liz@...> wrote: It does seem like a hard cut-off is going to hurt people without some further action. I can see two ways the CNCF might be able to help:
- Getting the word out. I suspect that a lot of Helm chart creation is done by folks outside of the "Helm community" and maybe the message is not reaching the wider cloud world. This isn't a criticism of the Helm community or the way it communicates, it's an indicator of how widely used the project is. I think CNCF could help push a message to this wider audience
The affected charts are mainly the ones in the official community repo:
There is a notice in the README.
PRs to the charts are still being merged, though at a lower rate than earlier in the year:
- I'm not clear from the GH thread whether there is already a CNCF owned location that the old locations can be redirected to, but I'd be supportive of CNCF funds being used for that purpose. Is it more-or-less as simple as redirecting URLs from google-funded resources to community-funded resources?
There is not a CNCF-owned location that I'm aware of.
Regarding redirects: "The URL to the stable repository has been embedded in Helm since v2.0.0-alpha.2. It's directly to a Google Cloud bucket"
No redirector was put in place.
The project containing the buckets could be moved to a CNCF account.
Does Artifact Hub have a role to play here?
Finally, I can't be the first person to think this thought, but is there some possible way to contact chart owners and tell them about the imminent demise of the current buckets?
Liz
On Wed, 14 Oct 2020 at 07:45, Arvind Gupta < agupta@...> wrote: I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
|
|
|
Chris Aniszczyk
It would be great to have input from the current Helm maintainers on their issue and public deprecation plan here. +Matt Farina
Also, this is a big budget item and will likely involve the GB, where the TOC should mostly focus on the technical deprecation plan and how to do this for the overall ecosystem with minimal impact. As I understand it, the storage cost of the charts is relatively cheap but the bandwidth costs are considerable and would make up a huge chunk of the budget and honestly something usually sponsored by large cloud providers or a CDN, similar to how we handle the Kubernetes project or even other projects like say jquery.
A simple solution would be to reuse or extend the existing credits that exist for Kubernetes [1] and firm up an ownership plan with the Helm maintainers on when deprecation can happen with a firm date, but this requires collaboration with the Helm maintainers [2] and the wider community [3]. We can also try to offer some time at the upcoming KubeCon to further make the community aware of the upcoming deprecation, but these things are always tricky for when it comes to widely used software.
On Wed, Oct 14, 2020 at 12:51 AM Liz Rice < liz@...> wrote: It does seem like a hard cut-off is going to hurt people without some further action. I can see two ways the CNCF might be able to help:
- Getting the word out. I suspect that a lot of Helm chart creation is done by folks outside of the "Helm community" and maybe the message is not reaching the wider cloud world. This isn't a criticism of the Helm community or the way it communicates, it's an indicator of how widely used the project is. I think CNCF could help push a message to this wider audience
The affected charts are mainly the ones in the official community repo:
There is a notice in the README.
PRs to the charts are still being merged, though at a lower rate than earlier in the year:
- I'm not clear from the GH thread whether there is already a CNCF owned location that the old locations can be redirected to, but I'd be supportive of CNCF funds being used for that purpose. Is it more-or-less as simple as redirecting URLs from google-funded resources to community-funded resources?
There is not a CNCF-owned location that I'm aware of.
Regarding redirects: "The URL to the stable repository has been embedded in Helm since v2.0.0-alpha.2. It's directly to a Google Cloud bucket"
No redirector was put in place.
The project containing the buckets could be moved to a CNCF account.
Does Artifact Hub have a role to play here?
Finally, I can't be the first person to think this thought, but is there some possible way to contact chart owners and tell them about the imminent demise of the current buckets?
Liz
On Wed, 14 Oct 2020 at 07:45, Arvind Gupta < agupta@...> wrote: I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
|
|
|
Thanks Chris. Just noting that we do have the Special Issues process where $$$ are involved to take technical budget issues to the GB, though it sounds unlikely that this would be an affordable item for the community to sustain. (I am wondering if there is a stop-gap measure to fund the storage for a short period of time while the deprecation message gets through to users, to stop it falling off a cliff - but there is only any point in doing that if we can effectively encourage migration off of these resources.)
On Wed, Oct 14, 2020 at 2:20 PM Chris Aniszczyk < caniszczyk@...> wrote: It would be great to have input from the current Helm maintainers on their issue and public deprecation plan here. +Matt Farina
Also, this is a big budget item and will likely involve the GB, where the TOC should mostly focus on the technical deprecation plan and how to do this for the overall ecosystem with minimal impact. As I understand it, the storage cost of the charts is relatively cheap but the bandwidth costs are considerable and would make up a huge chunk of the budget and honestly something usually sponsored by large cloud providers or a CDN, similar to how we handle the Kubernetes project or even other projects like say jquery.
A simple solution would be to reuse or extend the existing credits that exist for Kubernetes [1] and firm up an ownership plan with the Helm maintainers on when deprecation can happen with a firm date, but this requires collaboration with the Helm maintainers [2] and the wider community [3]. We can also try to offer some time at the upcoming KubeCon to further make the community aware of the upcoming deprecation, but these things are always tricky for when it comes to widely used software.
On Wed, Oct 14, 2020 at 12:51 AM Liz Rice < liz@...> wrote: It does seem like a hard cut-off is going to hurt people without some further action. I can see two ways the CNCF might be able to help:
- Getting the word out. I suspect that a lot of Helm chart creation is done by folks outside of the "Helm community" and maybe the message is not reaching the wider cloud world. This isn't a criticism of the Helm community or the way it communicates, it's an indicator of how widely used the project is. I think CNCF could help push a message to this wider audience
The affected charts are mainly the ones in the official community repo:
There is a notice in the README.
PRs to the charts are still being merged, though at a lower rate than earlier in the year:
- I'm not clear from the GH thread whether there is already a CNCF owned location that the old locations can be redirected to, but I'd be supportive of CNCF funds being used for that purpose. Is it more-or-less as simple as redirecting URLs from google-funded resources to community-funded resources?
There is not a CNCF-owned location that I'm aware of.
Regarding redirects: "The URL to the stable repository has been embedded in Helm since v2.0.0-alpha.2. It's directly to a Google Cloud bucket"
No redirector was put in place.
The project containing the buckets could be moved to a CNCF account.
Does Artifact Hub have a role to play here?
Finally, I can't be the first person to think this thought, but is there some possible way to contact chart owners and tell them about the imminent demise of the current buckets?
Liz
On Wed, 14 Oct 2020 at 07:45, Arvind Gupta < agupta@...> wrote: I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
--
|
|
|
On Wed, Oct 14, 2020 at 7:02 AM Liz Rice < liz@...> wrote: Thanks Chris. Just noting that we do have the Special Issues process where $$$ are involved to take technical budget issues to the GB, though it sounds unlikely that this would be an affordable item for the community to sustain. (I am wondering if there is a stop-gap measure to fund the storage for a short period of time while the deprecation message gets through to users, to stop it falling off a cliff - but there is only any point in doing that if we can effectively encourage migration off of these resources.)
This has been an issue since at least April 2019. The deprecation message is not getting through.
On Wed, Oct 14, 2020 at 2:20 PM Chris Aniszczyk < caniszczyk@...> wrote: It would be great to have input from the current Helm maintainers on their issue and public deprecation plan here. +Matt Farina
Also, this is a big budget item and will likely involve the GB, where the TOC should mostly focus on the technical deprecation plan and how to do this for the overall ecosystem with minimal impact. As I understand it, the storage cost of the charts is relatively cheap but the bandwidth costs are considerable and would make up a huge chunk of the budget and honestly something usually sponsored by large cloud providers or a CDN, similar to how we handle the Kubernetes project or even other projects like say jquery.
A simple solution would be to reuse or extend the existing credits that exist for Kubernetes [1] and firm up an ownership plan with the Helm maintainers on when deprecation can happen with a firm date, but this requires collaboration with the Helm maintainers [2] and the wider community [3]. We can also try to offer some time at the upcoming KubeCon to further make the community aware of the upcoming deprecation, but these things are always tricky for when it comes to widely used software.
On Wed, Oct 14, 2020 at 12:51 AM Liz Rice < liz@...> wrote: It does seem like a hard cut-off is going to hurt people without some further action. I can see two ways the CNCF might be able to help:
- Getting the word out. I suspect that a lot of Helm chart creation is done by folks outside of the "Helm community" and maybe the message is not reaching the wider cloud world. This isn't a criticism of the Helm community or the way it communicates, it's an indicator of how widely used the project is. I think CNCF could help push a message to this wider audience
The affected charts are mainly the ones in the official community repo:
There is a notice in the README.
PRs to the charts are still being merged, though at a lower rate than earlier in the year:
- I'm not clear from the GH thread whether there is already a CNCF owned location that the old locations can be redirected to, but I'd be supportive of CNCF funds being used for that purpose. Is it more-or-less as simple as redirecting URLs from google-funded resources to community-funded resources?
There is not a CNCF-owned location that I'm aware of.
Regarding redirects: "The URL to the stable repository has been embedded in Helm since v2.0.0-alpha.2. It's directly to a Google Cloud bucket"
No redirector was put in place.
The project containing the buckets could be moved to a CNCF account.
Does Artifact Hub have a role to play here?
Finally, I can't be the first person to think this thought, but is there some possible way to contact chart owners and tell them about the imminent demise of the current buckets?
Liz
On Wed, 14 Oct 2020 at 07:45, Arvind Gupta < agupta@...> wrote: I would recommend setting up a URL forwarding from old repos to new repos, so that charts are maintained at new locations but old URL references continue to work.
Thanks, -Arvind
TOC Folks,
This has the potential for causing a lot of pain to users.
The buckets are scheduled to be deleted on Nov 13, 2020, per the original deprecation plan. As that date is fast approaching, it is becoming clear that doing so could cause major issues. Per Brian's comment on the bug " The usage of the charts storage buckets hasn't decreased at all. It's higher than when the deprecation was announced last October. There have been about 3.5 accesses per second on average over the last 12 hours."
But the hard part to fight is the countless posts/examples on stackoverflow/other sites, and the many vendors who apparently haven't migrated yet.
Given that and the bucket usage metrics, the fear is that the number of people still using the old repos may be under-estimated, and shutting down the buckets could cause many unanticipated outages. The long lead time was supposed to be about getting users off, but that result does not appear to be achieved yet.
As TOC, should we consider alternative options such as transitioning ownership of these resources to CNCF to ensure a longer runway for deprecation?
Regards,
Saad Ali
--
|
|
|
All,
I'm really unclear on why CNCF would devote resources to maintaining a
set of outdated Helm charts if the Helm project itself has decided not
to do so. Wouldn't it be more sensible for us to use the CNCF marketing
resources to make sure that people know about the impending scheduled
removal?
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
Josh,
You ask a good question.
There are two things worth noting...
- Many of the charts people use have been slower to move out of the stable and incubator repos than expected. We are still finding them new homes. So, not everything is deprecated. There are many useful things there.
- There is more use of the stable and incubator repos than I ever expected. I'm told the use is growing even as things move off it. When I recently saw some numbers I was amazingly surprised. I had expected it to be lower. Note, I don't have access to this data. The buckets are Google owned/managed so only they have access to this. I think Vic is the only one of the Helm maintainers with access to this.
Many people will have some pain when the stable and incubator repos go away.
We do have some announcements (I'm hoping next week) and a path forward we're working out that will help make this easier.
- Matt Farina
On Wed, Oct 14, 2020, at 1:18 PM, Josh Berkus wrote:
All,
I'm really unclear on why CNCF would devote resources to maintaining a
set of outdated Helm charts if the Helm project itself has decided not
to do so. Wouldn't it be more sensible for us to use the CNCF marketing
resources to make sure that people know about the impending scheduled
removal?
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
On 10/14/20 10:52 AM, Matt Farina wrote:
There are two things worth noting...
1. Many of the charts people use have been slower to move out of the
stable and incubator repos than expected. We are still finding them
new homes. So, not everything is deprecated. There are many useful
things there.
2. There is more use of the stable and incubator repos than I ever
expected. I'm told the use is growing even as things move off it.
When I recently saw some numbers I was amazingly surprised. I had
expected it to be lower. Note, I don't have access to this data. The
buckets are Google owned/managed so only they have access to this. I
think Vic is the only one of the Helm maintainers with access to this.
Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund extended storage for these charts, or not? (assuming this is technically possible). I can see why the answer might be "not"; we have break the links to the old locations sometime, and why not now? The request to the TOC came from a concerned contributor, not from the project leadership, as far as I can tell. For my part, I would not want to see the TOC messing with a project's official deprecation plans without a formal project request. -- -- Josh Berkus Kubernetes Community Red Hat OSPO
|
|
|
From what I have been told, it's not technically possible to transfer the project.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote: From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
Sorry for the confusion. I've asked multiple people at Google about this and I was told it wasn't possible. Thank you for not only clearing up the confusion but pointing out the project-migration docs.
This still doesn't deal with the funding issue. We've not gone to the GB on funding (it would be well over the limit to just do so GB is needed) because the amount would be high enough that it impacts other CNCF activities like security audits and contractors working on things. We don't want to see that happen.
Now, if Helm could use some of the unused GCP credits for k8s the funding wouldn't be a problem. There are more than enough unused credits. Is transferring the bucket to the CNCF k8s account and setting it up to use some of those credits possible? We would shrink down what the buckets served and it would enable the k8s SIG Service catalog Helm repo to stay alive as it's managed through the same project.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
This highlights part of the problem. If we look at the current Helm governance this isn't a list of the right people. Sadly, my access didn't even let me see who the other people were or that my access was "Storage Admin". IAM is a finicky thing.
The lack of ownership is more than a storage problem. For example, not having access to setup a custom domain with HTTPS. You need more than storage admin for that. This is what I meant by a pain point. Or, being able to manage who has access. Of the charts maintainers, I'm the only one on that list. Adam and Michelle aren't. What if I'm unavailable and someone else needs to step in? The reality is that "storage admin" isn't enough to really manage something well. That's what I meant.
- Matt Farina
On Wed, Oct 14, 2020, at 2:53 PM, Brian Grant wrote:
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
To be clear, what I was suggesting was not intended to “mess with anyone’s plans” - I was simply trying to lay out that if Helm finds itself in a difficult position here, we (TOC) are here to help, be that lobbying for additional support in marketing or even budget, if that’s appropriate. Helm is a graduated project; that’s supposed to mean minimal risk for end users of the project; a risk has been identified and I think it is entirely appropriate for TOC to offer help & support in an attempt to mitigate that risk.
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote: From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
On Wed, Oct 14, 2020 at 12:27 PM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
Sorry for the confusion. I've asked multiple people at Google about this and I was told it wasn't possible. Thank you for not only clearing up the confusion but pointing out the project-migration docs.
This still doesn't deal with the funding issue. We've not gone to the GB on funding (it would be well over the limit to just do so GB is needed) because the amount would be high enough that it impacts other CNCF activities like security audits and contractors working on things. We don't want to see that happen.
Now, if Helm could use some of the unused GCP credits for k8s the funding wouldn't be a problem. There are more than enough unused credits. Is transferring the bucket to the CNCF k8s account and setting it up to use some of those credits possible? We would shrink down what the buckets served and it would enable the k8s SIG Service catalog Helm repo to stay alive as it's managed through the same project.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
This highlights part of the problem. If we look at the current Helm governance this isn't a list of the right people. Sadly, my access didn't even let me see who the other people were or that my access was "Storage Admin". IAM is a finicky thing.
The lack of ownership is more than a storage problem. For example, not having access to setup a custom domain with HTTPS. You need more than storage admin for that. This is what I meant by a pain point. Or, being able to manage who has access. Of the charts maintainers, I'm the only one on that list. Adam and Michelle aren't. What if I'm unavailable and someone else needs to step in? The reality is that "storage admin" isn't enough to really manage something well. That's what I meant.
Right. There are Googlers who can grant additional permissions, but moving the Project would fix this completely.
Vic had another suggestion: is it possible to redirect via Helm index files?
- Matt Farina
On Wed, Oct 14, 2020, at 2:53 PM, Brian Grant wrote:
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
Brian/Vic - are you saying continue serving index.yaml from GCS, but modify it to point to .tgz files hosted elsewhere? such as: entries:
alpine:
- created: 2016-10-06T16:23:20.499814565-06:00
description: Deploy a basic Alpine Linux pod
digest: 99c76e403d752c84ead610644d4b1c2f2b453a74b921f422b9dcb8a7c8b559cd
name: alpine
urls:
- https://otherdomain.com/charts/alpine-0.2.0.tgz
version: 0.2.0
?
This might cut down on transfer costs, but I think the index is actually larger in size than the .tgz's.
Is that what you mean by "redirect via index files"?
- Josh
On Wed, Oct 14, 2020 at 12:27 PM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
Sorry for the confusion. I've asked multiple people at Google about this and I was told it wasn't possible. Thank you for not only clearing up the confusion but pointing out the project-migration docs.
This still doesn't deal with the funding issue. We've not gone to the GB on funding (it would be well over the limit to just do so GB is needed) because the amount would be high enough that it impacts other CNCF activities like security audits and contractors working on things. We don't want to see that happen.
Now, if Helm could use some of the unused GCP credits for k8s the funding wouldn't be a problem. There are more than enough unused credits. Is transferring the bucket to the CNCF k8s account and setting it up to use some of those credits possible? We would shrink down what the buckets served and it would enable the k8s SIG Service catalog Helm repo to stay alive as it's managed through the same project.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
This highlights part of the problem. If we look at the current Helm governance this isn't a list of the right people. Sadly, my access didn't even let me see who the other people were or that my access was "Storage Admin". IAM is a finicky thing.
The lack of ownership is more than a storage problem. For example, not having access to setup a custom domain with HTTPS. You need more than storage admin for that. This is what I meant by a pain point. Or, being able to manage who has access. Of the charts maintainers, I'm the only one on that list. Adam and Michelle aren't. What if I'm unavailable and someone else needs to step in? The reality is that "storage admin" isn't enough to really manage something well. That's what I meant.
Right. There are Googlers who can grant additional permissions, but moving the Project would fix this completely.
Vic had another suggestion: is it possible to redirect via Helm index files?
- Matt Farina
On Wed, Oct 14, 2020, at 2:53 PM, Brian Grant wrote:
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
Vic Iglesias <viglesias@...>
I had 2 suggestions wrt mucking with index.yaml:
Use index.yaml to point all chart downloads to a specially crafted chart that causes a hard failure on the users end but provides an error message that points them at remediation steps. Pros Gives existing users an error message that helps them with the migration instead of a 404 with no guidance.
Cons Needs to be tested thoroughly. We don't want Helm to delete any user resources when it encounters this Chart. It must be an early failure with no side effects. Still some bandwidth costs even if the chart is very small.
Use the index.yaml to redirect all chart downloads to Chart Center. Pros Easy change for maintainers No change for users
Cons Requires JFrog to foot the bandwidth bill for chart tarballs
Requires Google to foot the bill for index.yaml Creates a crutch for users on the deprecated versions of Helm that will eventually need to be removed as well. We're kinda just kicking the can down the road.
On Wed, Oct 14, 2020 at 1:24 PM Josh Dolitsky < jdolitsky@...> wrote: Brian/Vic - are you saying continue serving index.yaml from GCS, but modify it to point to .tgz files hosted elsewhere? such as: entries:
alpine:
- created: 2016-10-06T16:23:20.499814565-06:00
description: Deploy a basic Alpine Linux pod
digest: 99c76e403d752c84ead610644d4b1c2f2b453a74b921f422b9dcb8a7c8b559cd
name: alpine
urls:
- https://otherdomain.com/charts/alpine-0.2.0.tgz
version: 0.2.0
?
This might cut down on transfer costs, but I think the index is actually larger in size than the .tgz's.
Is that what you mean by "redirect via index files"?
- Josh
On Wed, Oct 14, 2020 at 12:27 PM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
Sorry for the confusion. I've asked multiple people at Google about this and I was told it wasn't possible. Thank you for not only clearing up the confusion but pointing out the project-migration docs.
This still doesn't deal with the funding issue. We've not gone to the GB on funding (it would be well over the limit to just do so GB is needed) because the amount would be high enough that it impacts other CNCF activities like security audits and contractors working on things. We don't want to see that happen.
Now, if Helm could use some of the unused GCP credits for k8s the funding wouldn't be a problem. There are more than enough unused credits. Is transferring the bucket to the CNCF k8s account and setting it up to use some of those credits possible? We would shrink down what the buckets served and it would enable the k8s SIG Service catalog Helm repo to stay alive as it's managed through the same project.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
This highlights part of the problem. If we look at the current Helm governance this isn't a list of the right people. Sadly, my access didn't even let me see who the other people were or that my access was "Storage Admin". IAM is a finicky thing.
The lack of ownership is more than a storage problem. For example, not having access to setup a custom domain with HTTPS. You need more than storage admin for that. This is what I meant by a pain point. Or, being able to manage who has access. Of the charts maintainers, I'm the only one on that list. Adam and Michelle aren't. What if I'm unavailable and someone else needs to step in? The reality is that "storage admin" isn't enough to really manage something well. That's what I meant.
Right. There are Googlers who can grant additional permissions, but moving the Project would fix this completely.
Vic had another suggestion: is it possible to redirect via Helm index files?
- Matt Farina
On Wed, Oct 14, 2020, at 2:53 PM, Brian Grant wrote:
On Wed, Oct 14, 2020 at 11:50 AM Matt Farina < matt@...> wrote:
From what I have been told, it's not technically possible to transfer the project.
Transferring the GCP Project to a CNCF GCP Organization and Account would make it easier for you to manage, and has to be done if it's going to continue to exist.
You, Michelle Noorali, and Adam Reese have "Storage Admin" access. There's also a chart uploader service account.
To add some context about cost and history...
We have seen numbers on what it would cost to host these through Google Cloud. The numbers, if I understood what I saw right, is the public cost rather than the internal cross charge cost. It's a larger number than I ever expected and a cost I'm very hesitant to ask the CNCF to cover as an unexpected expense. Especially during these times.
If there were a bunch of unused Google Cloud Credits laying around I would be happy to use them. But, that doesn't appear to be the case.
Just know, the cost to the CNCF would be substantial.
One of the problems is the hard coded URLs in the Helm client. This was all setup in early 2016 when Jack Greenfield and Ville Aikas were major Helm contributors on behalf of Google. The bucket setup (as with all the Helm Google Cloud services) were setup through Google corp accounts. Googlers controlled them and do to this day.
The lack of ownership by the project has been a pain point. But, older Helm v2 clients are stuck with it (can't do a redirect) and many of them are still in use. The version use across Helm versions is rather large.
What we want is to deprecate the charts repository (headed toward a distributed route) while still keeping things available for people. That includes old versions of charts.
I hope that context helps.
- Matt Farina
On Wed, Oct 14, 2020, at 2:10 PM, Josh Berkus wrote:
On 10/14/20 10:52 AM, Matt Farina wrote:
>
> There are two things worth noting...
>
> 1. Many of the charts people use have been slower to move out of the
> stable and incubator repos than expected. We are still finding them
> new homes. So, not everything is deprecated. There are many useful
> things there.
> 2. There is more use of the stable and incubator repos than I ever
> expected. I'm told the use is growing even as things move off it.
> When I recently saw some numbers I was amazingly surprised. I had
> expected it to be lower. Note, I don't have access to this data. The
> buckets are Google owned/managed so only they have access to this. I
> think Vic is the only one of the Helm maintainers with access to this.
>
> Many people will have some pain when the stable and incubator repos go away.
My question is: is the *Helm project* asking for the CNCF to fund
extended storage for these charts, or not? (assuming this is technically
possible). I can see why the answer might be "not"; we have break the
links to the old locations sometime, and why not now?
The request to the TOC came from a concerned contributor, not from the
project leadership, as far as I can tell.
For my part, I would not want to see the TOC messing with a project's
official deprecation plans without a formal project request.
--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO
|
|
|
