Date   

Pre-proposal idea: introduce a new flag for helm uninstall "--fail-on-error" and a new release status "failed-uninstall"

mikeshngdev@...
 

Hi all,

Following the HIP guide. I am looking for feedback regarding my idea below:

The current Helm uninstall implementation is marking release record as uninstalled even if Helm detected errors during the uninstall process. See:

https://github.com/helm/helm/blob/v3.4.2/pkg/action/uninstall.go#L122-L127

https://github.com/helm/helm/blob/v3.4.2/pkg/action/uninstall.go#L144-L150


If the release record is marked as uninstalled then re-attempting to helm uninstall doesn't actually delete any resources (just purge the release record) . See:

https://github.com/helm/helm/blob/v3.4.2/pkg/action/uninstall.go#L81-L91

I think it's very useful for scripts or automation tools to be able to re-attempt a helm uninstall that resulted in error.

That's why I am sharing my idea of adding a new flag to helm uninstall "--fail-on-error".
If this flag is enabled, and the helm uninstall is successful then it doesn't change any behavior.

If this flag is enabled, and the helm uninstall completed with errors then update the release record to "failed-uninstall" status.

This allows the next helm uninstall attempt to actually perform resources delete.

For reference, I initially suggested https://github.com/helm/helm/issues/9201 but after thinking about it I think this "--fail-on-error" idea is much clearer and cleaner.

Thank you for reading and please provide some feedback. I am willing to drive the implementation if needed.

Mike Ng


Re: Proposed dates for 3.6 release

Marc Khouzam
 

There was no objections to the proposed dates for 3.6.  So with today's 3.5 release the 3.6 dates can be announced as:

3.6 RC1: Monday May 17th, 2021
3.6 final release: Wednesday May 26th, 2021

Thanks

On Thu, Jan 7, 2021 at 13:27 Marc Khouzam <marc.khouzam@...> wrote:
Hi,

helm 3.5 will be released next week Wednesday, January 13th as previously announced.
At the same time we should announce when the 3.6 release is planned.
On the dev call today it was suggested:

3.6 RC1: Monday May 17th, 2021
3.6 final release: Wednesday May 26th, 2021

That date is about a month following the release of Kubernetes 1.21 to give maintainers
enough time to adapt helm to any unexpected changes to Kubernetes's Go API.

Does anyone have concerns about those dates?  (there is a holiday very close following which may affect maintainers availability).

Let's do a lazy consensus, so if there are no objections by next Tuesday, we'll announce the above dates the next day.

Thank you

Marc



Is there any way to enforce constraints

Loritsch, Berin
 

Example: I am creating a chart for a single page app server.  In it I want the ability to add a reference to the TLS secret.  The TLS secret would be provided by the person using the chart.

In my chart I want to enforce that the provided Secret is the type:  kubernetes.io/tls 

I want that constraint enforced prior to deploying the chart. 

--
Berin Loritsch

DOMEX Architect


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708


Proposed dates for 3.6 release

Marc Khouzam
 

Hi,

helm 3.5 will be released next week Wednesday, January 13th as previously announced.
At the same time we should announce when the 3.6 release is planned.
On the dev call today it was suggested:

3.6 RC1: Monday May 17th, 2021
3.6 final release: Wednesday May 26th, 2021

That date is about a month following the release of Kubernetes 1.21 to give maintainers
enough time to adapt helm to any unexpected changes to Kubernetes's Go API.

Does anyone have concerns about those dates?  (there is a holiday very close following which may affect maintainers availability).

Let's do a lazy consensus, so if there are no objections by next Tuesday, we'll announce the above dates the next day.

Thank you

Marc



Re: Helm Test: What kind of tests can we perform on a helm chart?

Abhishek Tamrakar
 

Hi,
It depends what you are doing with the helm charts? Mostly people install apps, many install pre-requisites (roles, clusterroles, crds, etc.) apart from the applications (pods).
In case of application, a basic sanity of the application can be planned and could be written as part of tests/<template> in your helm chart wherever possible.
I don't think any other tests are necessary, because post deployment you can follow standard testing process which may include smoke tests, performance tests, penitration/security tests, etc.


Helm Test: What kind of tests can we perform on a helm chart?

@Tejaswini_Vadlamudi
 

Hi,
I was looking for the list of tests that can be executed or recommended to execute on a helm chart. Does anyone have a good handy list that can be seen as a good starting point?
One observation, most of the helm hub helm charts don't have tests. I'm not sure if they are late in implementing or if have any valid reason to NOT deliver tests.

Thanks!
Teja


Helm v3.4, Helm v2.17, and Stable/Incubator Updates

Matt Farina
 

Helm v3.4.0 and v2.17.0 were both released. Along with that the details on the new location for the stable and incubator chart repositories are available. Details are in the blog post at https://helm.sh/blog/new-location-stable-incubator-charts/


Helm Feature Proposal: Fail-Faster Testing

黃健毅 <johnny.jy.ooi@...>
 

Good morning/afternoon/evening (delete as applicable)

I've been directed to this mailing list by Martin Hickey and Matt Fisher due to a proposal I have briefly mentioned at https://github.com/helm/helm/issues/8861

I will put more details in this email

_____

Background

When deploying a Helm chart, you normally need to wait for the resources to be deployed by Helm, then use something like "kubectl rollout deployments/{deploymentname}" to wait for the rollout to finish before running "helm test" on the chart -- as if you ran the Helm test before the rollout was completed, and your test accesses the service via the cluster service (ClusterIP/NodePort/LoadBalancer), you would get a split between the old and new versions of the chart.

And herein lies the problem. If you have a chart that might have any number of features such as:

1) has a large number of replicas
2) has a slow readiness process (e.g. due to cluster-like syncing behaviour such as ElasticSearch)

Then the rollout state will take a long time -- during which, you can't run "helm test" because of the "split brain" situation and potentially inconsistent results.

If you wait until the rollout finishes, then run the test, and find the test has failed (e.g. for a complete breakage with the update), you then need to roll back the deployment -- this then takes the same amount of time to rolling update the pods back to the previous version as it did to update them to the new version. And when you have spent 30 minutes updating the pods, the last thing you want to have to do is have to do it again.

However, from the end user/customer perspective, the service will be down 100% in between the broken update being applied, and the rollback succeeding (if it does succeed)

____

Example

Let's assume we have a four-pod service based on nginx, with our website baked into the custom image used by the pods, represented here as "O"s

OOOO

We apply the update, and the deployment has a surge of two pods, so two additional pods come up -- we'll represent these as "n"s

OOOOnn

The two new pods come good -- represented as "N"s, and Kubernetes terminates two existing pods -- represented as "X"s. At this point, if the update was bad, we now have 50% of traffic going to the two old pods (and being successful), and 50% going to the new pods (and failing)

OOXXNN

Kubernetes continues on. The terminated pods disappear, and it spins up two new pods

OONNnn

The new pods come good and it terminates the final two old pods

XXNNNN

NNNN

We now have four new pods, but at this state, 100% of the traffic going through this service is failing. We run "helm test" which fails, and we rollback back to the good state

____

Proposal

The proposal of this change is to allow (opt-in) the testing to be applied earlier, preferably during the pod rollout, so that if the test fails, the rollout is cancelled while it is still in progress, reducing the downtime impact and minimising the rollback time. Using the example above, the flow would look like this:

4 pods as before:

OOOO

2 new pods are created

OOOOnn

Those two pods come good, and Kubernetes terminates two existing pods

OOXXNN

Helm detects the new pods are now good, and actions tests on one of them (or both, if the "--parallel" switch is used) -- the pod being testing is marked with T

OOTN

Test fails on the pod. So Helm cancels the rollout and triggers a rollback of the changes.

At this precise moment in time, we have two old pods and two new pods, so traffic is split 50/50 between the pods. So instead of your service being broken for 100% of traffic, your service is broken for 50% and rollback only requires creation to two pods (with the old configuration), allowing a faster rollback and reduced downtime.

If we had 100 replicas instead of four, we could find a broken update 2% into the update (assuming a surge of 2 pods), rather than after 100% into the update, and save the time of having to roll back 100 pods,

____


Happy to respond to any questions or suggestions to this proposal.

Regards

Johnny Ooi



Helm's 5th Birthday

Matt Farina
 

Today, Oct 19th, is Helm's 5th birthday. Along with that we have an announcement about GitHub and the stable/incubator charts... https://helm.sh/blog/helm-turns-five/


Helm Hub Migrated To Artifact Hub

Matt Farina
 

The Helm Hub has now migrated to the Artifact Hub. Details are available in the blog announcement at https://helm.sh/blog/helm-hub-moving-to-artifact-hub/

If there are any questions or comments there are some details in the announcement on the best places to share those. Or, you can respond here and I'll try to help.

- Matt Farina


Re: Why do we need Helm Tests? What does they solve?

Loritsch, Berin
 

My understanding is that Terratest was designed around testing deployments and predates any testing code in Helm.  I'm not on the terratest email distribution list so I don't know if they are even aware of helm's new testing support.


On Thu, Oct 1, 2020 at 6:12 AM <vtejaswini1@...> wrote:
@Matt:  I am trying to understand different test cases that can be tested via the helm test. I now agree that we need these tests but I was unsure how can help/aid my chart consumers. Most of the helm charts today in helm hub are testing connections to their ports. I was thinking if we can use "helm tests" to notice if the deployment was successful or not i.e. to understand if microservices are deployed properly as per deployment configuration and ready to serve traffic. Am I thinking correctly? Can we use helm tests to test readiness and liveness checks of microservices or is it a bad idea (rely on monitoring instead)?

@Berin: Thanks for the suggestions. I had a quick look at Terratest. If we have helm tests, we can either run "helm test" or "ct lint". I think Terratest is a go library support for writing test code and also offers support for deployment, validation of charts, and infrastructure readiness. Does Terratest run "helm test" as part of its validation or does that something else?

Thanks,
Teja



--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708


FYI, Helm Hub to Artifact Hub

Matt Farina
 

The Helm project has decided to deprecate the Helm Hub and redirect people to the Artifact Hub. We have been tracking this work for some time.

We are now at the point where the redirect will go into place, docs will go up, and a blog announcement will be doing out. This will be happening soon. This is a heads up that it's coming soon.

Regards,
Matt Farina


Small governance change

Matt Farina
 

I wanted to share that we have made a small governance change to Helm. There was a section in the governance that pertained to the initial election of org maintainers. That event happened a couple years ago. It is not relevant to ongoing or future work and we have git which records the history of the governance.

So, that section was removed following a vote.


Note, because it was the removal of no-longer relevant governance (the equivalent of dead code) I didn't email it out. If it were something that would make a relevant change I would have emailed the list about it.

Regards,
Matt Farina


Re: Why do we need Helm Tests? What does they solve?

@Tejaswini_Vadlamudi
 

@Matt:  I am trying to understand different test cases that can be tested via the helm test. I now agree that we need these tests but I was unsure how can help/aid my chart consumers. Most of the helm charts today in helm hub are testing connections to their ports. I was thinking if we can use "helm tests" to notice if the deployment was successful or not i.e. to understand if microservices are deployed properly as per deployment configuration and ready to serve traffic. Am I thinking correctly? Can we use helm tests to test readiness and liveness checks of microservices or is it a bad idea (rely on monitoring instead)?

@Berin: Thanks for the suggestions. I had a quick look at Terratest. If we have helm tests, we can either run "helm test" or "ct lint". I think Terratest is a go library support for writing test code and also offers support for deployment, validation of charts, and infrastructure readiness. Does Terratest run "helm test" as part of its validation or does that something else?

Thanks,
Teja


Re: Why do we need Helm Tests? What does they solve?

Loritsch, Berin
 

Keep in mind that there is a tool called Terratest https://github.com/gruntwork-io/terratest that is designed to test your deployments.  It works with a wide array of deployment tools, including Kubernetes.  Might be worth seeing what they provide.


On Wed, Sep 9, 2020 at 7:37 PM <vtejaswini1@...> wrote:
Thank you Matt, you always share prompt and correct reflections!! I'm very happy with the above explanation, sounds reasonable. :)
I agree that chart consumers (automation/human beings) are different actors compared to chart owners/developers. If chart owners don't involve in the chart consumer's helm deployments and tests, chart owners of course need to provide some sort of sanity tests.

Best Regards,
Teja 



--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708


Re: Why do we need Helm Tests? What does they solve?

@Tejaswini_Vadlamudi
 

Thank you Matt, you always share prompt and correct reflections!! I'm very happy with the above explanation, sounds reasonable. :)
I agree that chart consumers (automation/human beings) are different actors compared to chart owners/developers. If chart owners don't involve in the chart consumer's helm deployments and tests, chart owners of course need to provide some sort of sanity tests.

Best Regards,
Teja 


Re: Why do we need Helm Tests? What does they solve?

Matt Farina
 

Teja,

This is a good question.

There are a couple things that come to mind for uses of Helm tests.

Helm tests enable a chart author to package tests in a chart and ship them to a chart consumer. The chart consumer (sometimes called the Helm user) can then install a chart and test that it's running. This is useful as Helm is a package manager and the one creating the chart is often not the one installing and using the chart.

Helm is used across many versions of Kubernetes. That includes supported and unsupported versions. When a chart is packaged up by a chart creator it may be tested against the version of Kubernetes they use but does that mean it works against the version of Kubernetes a chart user is dealing with? Testing passed to the users in a common way provides a means to help figure it out.

One tool to help with testing is the Chart Testing tool (https://github.com/helm/chart-testing). It's used for the CI testing in the stable charts repo.

Regards,
Matt


On Mon, Sep 7, 2020, at 9:11 AM, vtejaswini1@... wrote:
Hi,
I want to understand why we need helm tests? I have my CI automation which deploys all of my charts and checks if the feature is working or not.
I don't understand why I need "helm tests" in the helm chart directories. Also, is it the only way-forward or do we have any other tools that can help us for the time being?

Thanks,
Teja


Why do we need Helm Tests? What does they solve?

@Tejaswini_Vadlamudi
 

Hi,
I want to understand why we need helm tests? I have my CI automation which deploys all of my charts and checks if the feature is working or not.
I don't understand why I need "helm tests" in the helm chart directories. Also, is it the only way-forward or do we have any other tools that can help us for the time being?

Thanks,
Teja


Stable and Incubator chart repo deprecation

Matt Farina
 

This has been shared previously but we've come to realize that not everyone has noticed.

On November 13th of this year the stable and incubator chart repos will be marked obsolete and will be going away.

There are two elements to this. The first is that they will be obsolete and we will no longer be accepting changes to them. Any charts that want to be around long term will need to migrate to a new location. If you have questions on doing this or running your own repository, we are happy to help. We have documented how to run a repo on GitHub and have released tools, like the chart releaser, to help.

Those charts can be made discoverable through the Helm Hub and Artifact Hub. The Artifact Hub is listed because the Helm Hub is migrating there. You can find the migration issue at https://github.com/helm/hub/issues/439.

The other element is that the chart repositories will be going away. While the git repository will still be available the Helm repository will be garbage collected and no longer available. This can happen on November 13th or any day after that. I would plan on Nov 13th to be on the safe side.

jFrog Chart Center does have a cache of the charts. I have been told this will remain after the stable and incubator repos are gone.

If you have any questions about this please let us know. This was initially announced months ago but I realize the message has been lost with everything else going on. We are putting a renewed focus on it in an attempt to make a clean of a transition as possible.

Regards,
Matt Farina


Helm Hub move to Artifact Hub

Matt Farina
 

The Helm Hub has made it far easier to find charts that are distributed around the Internet. General purpose search engines don't make it easy and the Helm Hub, being Kubernetes and charts specific, provided an improvement.

The Helm Hub is built on Monocular which was originally designed for purposes other than running a large many repo site. It has worked well but its limitations have started to showcase themselves. For example, the repository listing on the charts search page does not visually scale well and the pageload times increase with each new chart added to the site. We knew that some changes needed to happen for the Helm Hub to scale better.

The Artifact Hub is a Kubernetes artifact search site. It handles Helm charts and other cloud native artifacts. This site is self service for both publishers and consumers and designed to scale. The Artifact Hub was recently accepted as a CNCF Sandbox project and was born out of work in the CNCF.

Instead of investing in Helm Hub software that does a better job scaling, we are going to migrate over to use the Artifact Hub. This means the Helm Hub site will eventually redirect to the Artifact Hub and the Helm Hub codebase will be archived.

The issue tracking this migration is at https://github.com/helm/hub/issues/439

Some steps in the migration have already begun. This includes making all of the existing Helm Hub repositories available in the Artifact Hub. Those that were not already listed on the Artifact Hub were migrated. If you own one of the repositories that has been migrated and would like to take over management please file an issue on the Helm Hub and we can transfer the ownership.

We are happy to answer any questions related to this.

21 - 40 of 368