Re: [HIP] helm lock
Just out of curiosity does anyone know how helm would behave if we used ValidatingAdmissionController webhook to deny the helm release to be modified ? I bet you could rough it out with OPA/Gatekeeper to say “do not allow modifications to secrets with the helm labels and a lock annotation. I would imagine Helm would error out before upgrading the chart, which would give us close to the locking functionality without even touching helm yet.
If we could confirm that behavior, then we could even look at getting Helm to understand a webhook denial to give a better looking error, and then maybe write a helm plugin to perform the lock/unlock (add/remove the annotation), plus maybe write a targeting helm release locker webhook, or publish the rego required for OPA.
Sent: Tuesday, March 2, 2021 10:53 AM
To: Justin Garrison <justinleegarrison@...>
Cc: cncf-helm@...
Subject: Re: [cncf-helm] [HIP] helm lock
> I wasn't aware you could do that. Is it common practice for someone to replace a chart for an in place deployment?
A common use case is to migrate from a public chart repository (like one published on ArtifactHub) to an internal chart repository. The workflow is very similar to forking a git repository. A chart
can be changed after it is deployed. You can test this out yourself by installing one chart and call
helm upgrade with another chart.
The situation surrounding replacing one chart with an entirely different chart is a side-effect of this behaviour; templates can be removed and replaced at-will during an upgrade. It's important to keep in mind that Helm simply renders the templates, compares the rendered manifests with the previous release, and creates/updates/deletes resources as requested by the user.
Matthew Fisher
Caffeinated Software Engineer
Microsoft Canada
From: Justin Garrison <justinleegarrison@...>
Sent: Monday, March 1, 2021 1:02 PM
To: Matt Fisher <Matt.Fisher@...>
Cc: cncf-helm@... <cncf-helm@...>
Subject: Re: [cncf-helm] [HIP] helm lock
Thanks for the link to the PR. That seems a bit different than what I had in mind.
Those are great questions to include in the proposal. Here's some basic answers and a question for clarification.
- How would this prevent users from manually editing/updating resources underneath the chart using kubectl, kustomize, etc? Is this considered in-scope or out-of-scope? This is out of scope for locking. Charts should be explicit about what versions of dependencies they need and if a dependency cannot be met the upgrade should fail (same way apt and yum work). If I lock a yum package that requires a specific version of glibc yum will prevent me from upgrading glibc based on explicit version dependencies.
- How would other tools like Helmfile interact with this feature? How can other tools use this feature? What happens if they don't? Maybe I don't know enough about how Helmfile's work but I thought it called `helm` based on the specification in the file. Native version locking would still prevent upgrades so no changes should be needed to Helmfiles.
- Where do I find out which packages are marked as "held" or "unheld"? It would be great to have a `list` command like apt and yum but the lock information could also be displayed as part of other helm list commands
- If I lock `nginx-ingress-vX.X.X`, does that apply to only one release, one namespace, or cluster-wide? Can I apply this lock across different scopes? A lock should be only per deployed chart. In helm 3 those are namespace scoped but you can also have some charts deployed multiple times under different names so a lock should only apply to one of them. This is probably the biggest difference between apt/yum because those work at a machine level but helm can have multiple versions of the same chart installed.
- How does the locking mechanism work? If I lock to kubernetes/nginx-ingress-vX.X.X but I call `helm upgrade my-ingress bitnami/nginx-ingress-vX.X.X`, what happens? What happens if I replace it with a different chart (e.g. apache)? I wasn't aware you could do that. Is it common practice for someone to replace a chart for an in place deployment? I had assumed a chart could not be changed after it was deployed. If people do this and have a lock in place then there are unknown results. I suppose it would be the same if I had 2 repos on linux that both provided an nginx package and they both had different version strings. It would be up to me as an admin to know the difference and lock the correct version.
--
Justin Garrison
justingarrison.com
On Thu, Feb 25, 2021 at 2:23 PM Matt Fisher <Matt.Fisher@...> wrote:
Hey Justin!
I like this idea. We've had similar proposals presented for resource update locks like https://github.com/helm/helm/pull/5290, though there were several design flaws preventing that from working in a practical sense. Still, it could be worth reading up on its use cases to see if other cases align with your proposal.
In your proposal, it might be worth answering questions such as:
- How would this prevent users from manually editing/updating resources underneath the chart using kubectl, kustomize, etc? Is this considered in-scope or out-of-scope?
- How would other tools like Helmfile interact with this feature? How can other tools use this feature? What happens if they don't?
- Where do I find out which packages are marked as "held" or "unheld"?
- If I lock `nginx-ingress-vX.X.X`, does that apply to only one release, one namespace, or cluster-wide? Can I apply this lock across different scopes?
- How does the locking mechanism work? If I lock to
kubernetes/nginx-ingress-vX.X.X but I call `helm upgrade my-ingress bitnami/nginx-ingress-vX.X.X`, what happens? What happens if I replace it with a different chart (e.g. apache)?Hope this helps.
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIMAAAAdCAYAAACJ8d9VAAAAAXNSR0IArs4c6QAAAAlwSFlzAAAOxAAADsQBlSsOGwAAABl0RVh0U29mdHdhcmUATWljcm9zb2Z0IE9mZmljZX/tNXEAAA/ESURBVGhD7VoLlF1Ved57n32ed2Yyk8lzkhCYmLBGU0YSI2BajTQgjTWlVnHASDWCoCwbTQuhATINGslaVKCgGJdUAYtdq6wCwRXRKC5ilQA1JFMC4bVCAnmn88i9c8/77NPvP3fuzJ3JJHNduvDSNXvWXTP3nP349/9/+/sfe2ThC63XmJy/J1BMsSqarXONF6Ofdtz//v86/5nja/HNVnGajjWUi5Qz9EoD9u3itgtfHav/+Pu3XwOSqfQKvU77oB6Nac+SdKZgUYFHHvN2YcSaXINkSTg2joTg2fBiGvwEv8bB8PbbeswVZcp5MQEtuHF1YKjHlOjp28xWLC3kvULckETVgwHD4zGlGu/wR9GA/KOsOr5oTWpgHAw1aZY/nFCdnZ0Cn9Y4ZtMZ48fb29/92p49e5LRVqg5MHR3d+u+7+utra1q714mli5t9SH82H4Iuzt48KBTGrdXWJYVNTc3R7TpdevW8ZtvvvkC32dHGhqMvX84Vdf2THnX/cBNN627LYzU+9I0tTnnQVdX1yLDMHa/I8Bg2/XX1zdMvLKvrxBNmiT03z6/65UZ06d+sq+vLzyd6n0/vGnK1JbP9PYW4slTWmQcRQ+j/y0EhH9ce/PdhmF90TTD3tANlxuOsb22zfj7SxeG4bsNzfixbhgTlVJMCMGklFYhX7AMY/T5a44ZOGezLcs8OwhAavgiuX72oWPHPuwYxs9OpSLXdScwrl1r6frMLOExTVaMo7Po71WrVknbqbtU6lIzTWNSfyG/BI//34MhitnqXM6aCFCwOI53MM7uE0xMZrZ+5OBBt6Gp2ViKVGCJV9Q2NDeLo6SrmgMDZAqjKGL0oQa6R/6RrmAGOyUYAPu/NHRzpud5DHSYnQL8ypiEXIXrh5vCIFwTJ/EBTbDNv/+5q+0ZyNXaTsMCYgToI5EaWwXX8JtBqSfKbY5tfTCOoWM7/Wb5eS2CYVBmYoYMFJx/FKd/puM4B0aaAcERv2HN2s8RALB5ZMoYNKI5lrEB47+7cOHCnmrjj9o29+ml8/0W3bILDXQwkjgJLEu+OWLENPoO1gD/GsUaB0NmTxeb2Q8ktIEdmvzA/wSe3TVSDWvWdJ7DRfIhUCHjgu9DEWTWSMZzXTULczhdXS9Nqq+3DgRB0D9yHijmHBykdqVEnRCqXwnW1TJlykuIVWIAaRLGz4AswCV7C3jrCUO1EHOQW9pjGGKwiNat1NRcqBZhLvQnScQRFvvPAshHRjMh5ralZf2pCuM5jAmqwexTKnwO/fMj+9M+mFDnAfaT8E4JIffHpbn7qG9bW5uGAPGs5uZ4pkqZRWCAvAJ7W4jDUh/H4Ahq8L5JQgkFHZzoXNeVh0C/Wk0yw8DZpvL1EypJzmC6zIHvOna37b5n/p75w9MiHv2tbecklFpE/83Y6HUjyYHz+PaUW5eqNGQnThSX4aT8sqxo1w0Xa1K7JU35EqnrpqZpLEmy5CU4dOTIKseyvguLXq5ScTsV3aM4/BoK9w0Y81UARC/09//IMMxPI05x8nl3LQS9CqCcapl6tgSBVGnG0SCINjU01G0EEP3y2v39/sWGad2exuk5pmVnLo6a76WP4ddfD8noThNCXwfy6xBSNiH6gbtPIUvENGm97obx3Y4h70EWZUD0/5Sa1qZiJSleQAMoxH+oBFLzuMyaOuSgdwb0tYX8cIZbvNU1jTOjugIkY+gLoSXK0Ry4M6QuMN/YrVyODr1IjN0by2iaDYrrUlw9jf4XCaG9/4wdrYuZw35VoaQJKdc+kRk/Zb/G86cw7iujzG9pQpgJtIl+pdOB5vr+cqkb/y6l7tALrMe80PNgTLupaaLZ0xNkASkTTMfJMsllYe9Xw/CzKUg1TYsVi0VYhrG+E4WHHSe3jIxfMqgX4eBRiqslYAtD1zt7+grz3regfQW5KhjqXCm1RxDh52gM4h2XVmpoaLAC328dlDFNpzAht9iOs4D6EWA83w3AUSbFU3CN7zI0/W7X9SbisHy99fm5jXihx8VS38zihgn5h6xEQKB4gvSGfejlwyN5qrrTmP1vnFSHBhN7BR6IwhKg4HAUqfokHrsMMCSMzCA5ViMBhVI94MIHsKmLoHieeOpyjBsEA7C83DTMWbQxtB/BYCcodhjZKIiiPlm/lKrpgoFJpnGhbwJ4nCSJwQbxb1VCwZTcCZ3MPnHixKfJxgNzYaiiPswyrdlwtQfDKPxeFIYO08QB142vc3LOskzJSQIfnG4ABT+BlQzf964AkFf5oGXbsi/fubNri2UZD8Vx+jknl8tFUQgQxj8BgX8ZNK7ni/0XY+xZoHxBoEm96BvotqA0tzqkVPz1RBrbLBZP9v3gKgBqBVE+sVtr17ynBRdfdovFudjzGgBtcgz6iKPga9jzQewlQwQwcisCyhaAOw78YC101k3vZO6+fSsIkbnsyqGqxuFbkkXvWplc39F4NmOXVDVosNO8efH6J2+tbowUNlPx49BDN8DQjF0sh1FI+N5sAs6u1HVJpxObiR9TSi7Whg7+GGvIlThZ00mROHE7TUO7WJgD8zLxCgZvNevq68gIZXhp8KrQXyEV7G+Q6j6bqSzO4bom/z8xWIUazuNX6izjvorFn/O8UJeG+aUMiSm/Gu8ewrFsSVMFLOHiL2VPwTh7S/l/tja77LLL+JrOzjmMpx0EGMqOsPcrHcd6sjS3gR+2zfOjiRi7DDrRwC4dli0/T0Da8XzXtQD6ZLBdZBja9/GewJA114tWASgtyCZw3xxvQjhVoOeSbw6uYboxn0VedWBwbM5c9kQ6g29bd7R+HdNvslkpCzx9K7PUcfYvt7Lrsw2P1RDMmQiOCkUvfBTMchWM1+LH0cccQzwIX/9e0PmSASrcTAEXCk/IQ0/foNDMakg7LhlIvViq0jsGAVYxvCLQzKSHO2GB7/3ccUwAodSk9OYnSpxBfwMor81smfJDBJ3DhEg19h0Y9As6kAs4zGtsbJSed/RVeBEWxCG8gLYaft9ncfgQ9tFDg9evX5+6YbgITIRzisg/8LfncuYAEIamh+z3AtDLBp6cR9kVPhPwPcMw9MMB9gkAzCAY8FQjvdE7hBSN6FYCA1P+p1jO+DMW2GPpsfSeROtjPn52MMFXswasedra4MC05ePVkz6OJ1WBAVF9ZgRdY/cHQbgSRSPBYkU1hwdhzCsc25YwGJJp9gD1g3H5YEBwit3A6AOgT88sGTAKEVDurG7zNJR3VfaFomdbtpXtDuB8EUA42Q3G8VsoilE2QTFIfU9Pz2x4rk0IEz5r2/b0MIym6Zq4Gx747/0wvuf1V1++a/58BMpKzCa3l7lMbfi6QzLIl7AHhEMCW+cTNm7cmOvtBUHxzL6/U5OARz/rhzUDil+qaGkDOgmPIXYBQ+RZHg+qYYZBrq2q9zBB2tvbt4P2noMpzofCFyP4WowOHyElxUmyc0bLlKdHnsYqdpJ1oROCuxCGE1nVECh9WLwMW2l0wony0U51PQ/ReZIFaviUlnNeALtdirjjNjy/kHwe7Dlb141/PrN17nsx12eAbr2UHlJiJU515BTeV7J6htiqNjOiU02mliM3khWKuLgflHg+FO/EiboT2p9LNI9g9t+oFvC7bh7a8knR8Ks6WBM3euzFauYoUetQgysrUCUvM7RK55QDv2FzWVZzGsTTM5emWB5JQFY8cxzjOfz6c9Qs/gqu/TqY8CLKPE3DWAEXQZXBfTRvFviqBPHZKOaS8ZkiFbIEtPTYjTfe6MJNNPpVhekkhQ+hSgfhHQEGEtQyxGN+4G2A7Zo1TSyizXu+34cA8xEKpqptUGzmSaC6XQDCXKpYgmmuwqNfjJyDfPvYQNN2gaZPIB6YgIi+fUdXF92jDPftofo4fLZBTI7VdlN8Ujk33lCJfLPvR5txsbQ8ywhjsVBK9RuwVgo5udC0D6Po9B7HEcNAmybpZTpqGqUKbPrfFGtgj6dN34lsSrEW1xFIEtW/s8CAzR6F+A/rhn4tcnhm2TZlAY+BbkeWWk+LizLNo1TyAAKvT1IWAGV/CpTdzyz5TXA4qnHWRMQr1x46cmw/Stn3ZgnpKZrj8Dddnz8BMHQAVBTmfM/349XA3FMQkaHY9HGsua4UsFFuzzbRVIcPH98Ids8rQ/6gybaP9/b2It9HfEkmgifCRC7w83LRC54xdesCpJ91iDN+6PvqH1Ch3IFg2gKjrEQqeQ0xRxY3SP6D7Kz7WOU0jkKggpoxgZQC2UYHxt+JEsb8WmQG6A73a1mBaLh/5mnyQBgEFJULXFEjfhUPVdqI6ijlsRRzVbwbnBPZVKYmnLAtxWJwb66+7kvw28y0zM8jGL0ChwVpa1w/obGpvqe35zbqSwAqB3L09SRcKG2t53uLbdOehcuwswCyR1kaHfV8JArI9YkRUA9gyP8fRCbyKI1H1Ns2sbFpWU9P71f7gsKBlMsJWGMWrQOw7zdNmTEVT/lqyLcVe66HizwXNY4nURE47AcxLmn1ptJ+NRaE0YY6h9LdUgOqkLWSqFyDG+MA1qDYKk2ehVwUe1GyewvmQtrLmmsODBQMDaQ9OPkK1bMh1YMFnsEJ/hWKMEsKhcKOlulTnjopjSulTPQZPM2Vcw5lE4y98cZrfzdnztlFoYkvgh3q0OgfQGwaWiy6KSLDfbR6DJQhkispuWLesmQA1htIDZf5UfAthHIfosogDDeV3DhVDQGyPMrY32lqavingTIwWeswGaO+vr4RTNBICKV3MPwLSEVXQs7jJdAaz+Td8GOY8A6c6AWQkR5PJzeJ612WRNGRSAV3AAi3V4IU71VJVHCgLDFOuaHueCf29xe5nNNGzwCmZiqA1hwYUCD5NqpiP4WMYGd9V+Um6G8p2dUo185nqXxlpD+H4n4dhvFHUanjUnJccpUa4oxbozD6V1Ciqkwjs/SNsRvy+fD7CE6XhlE8Bw4XJXp1APbf3jKQpaDu/wiqfa9CqbT+qGkx4oTdnW2dF97yQudSz/XOw5GejmJGEd57n5eIXzQ74pVBIJCRbf0Gzw22KBb/CYw8DfetvVKwF48fP/r4jBkzhl2kNTjGtu3d3R9or2+6JCm6C2DgqTjxVHh7HS5jKw7JW5V6amlpyh86dOyzvhfT/y+HANybw9Z2OKqm0YWeW+zgmnYm4g7koer5mgMDTsQeyxJ7RoKg/B109zopAcY+qQvFFRhL/4qPNkQphsEBKr6rkiorB+Nf4V7Gd/rQORl8VWYdzLsfQBsA16mD1fV71iscr61wxVsr53dGGUI3n5bF4TKMzG2UG4Aw6tYvaG6m/ACBZhZsVrSTTUj/FQZ5B8r2RsY4I9vALepd2fPSnVrtMcOomhh/+LZooOaY4W3Z9fgio2pgHAzjwBjUwDgYxsEwqIH/AzVqglIzFa2qAAAAAElFTkSuQmCC
Matthew Fisher
Caffeinated Software Engineer
Microsoft Canada
From: cncf-helm@... <cncf-helm@...> on behalf of Justin Garrison via lists.cncf.io <justinleegarrison=gmail.com@...>
Sent: Wednesday, February 24, 2021 5:21 PM
To: cncf-helm@... <cncf-helm@...>
Subject: [cncf-helm] [HIP] helm lock
I would like to propose a process to "lock" a helm release to a specific version so it is safe from `helm upgrade` and other situations that may cause unintended breaking changes.
There are a variety of ways to prevent breaking users with helm charts. Here are a couple I've investigated but both have limits.
- Publish charts to different repos. This adds a lot of burden on the publisher to make sure all compatible versions are in a specific repo and users have to switch repos to upgrade.
- Use deprecation messages to require user action before upgrading.
These methods should still be used if the maintainer wants to make sure user upgrades are safe, but it would also be good for consumers of the chart to not accidentally upgrade charts. There may be cases where a repo has stable and unstable versions. In that case it would be best if the user can "lock" their version to a stable release.
I envision a helm lock could work similar to yum versionlock or apt mark. A user can install a specific version of the chart and then run `helm lock nginx-ingress-vX.X.X` and any additional helm upgrades will ignore versions beyond that version. It may also be desired to allow a user to `helm install $CHART --lock` which will automatically lock to the currently available version or they can specify `--version $VERSION` to request a version.
Ideally this lock would be stored in the release's secret metadata and helm will use it if it is available. Without the metadata `helm upgrade` works the same way as it always has.
Users can also `unlock` their releases and locks could appear in helm list or other places metadata about a release is surfaced.
Please let me know what you think so I can write up a HIP if this feature would benefit the community. If there is prior discussion about this I would also be interested in reading it.
--
Justin Garrison
justingarrison.com