|
Helm Vulnerability: Client Loading and Packaging Chart Directory Containing Malicious Symlinked Content
Matt Farina
Part of the process for Helm to become a graduated CNCF project is to complete an independent and third party security audit with the results being published. As part of the audit of Helm 3 a security issue was found that also impacts Helm v2. Cure53 performed the audit and found the issue. More about the audit will be covered in a future post. The vulnerability found impacts all versions of Helm between Helm >=2.0.0 and < 2.15.2. Helm commands that deal with loading a chart as a directory or packaging a chart provide an opportunity for a maliciously designed chart to include content not intended in the chart or to execute a denial of service (DOS) on the computer performing the packaging via the use of symlinks. No version of Tiller is known to be impacted. This is a client-only issue. The following Helm commands may unsafely handle malformed charts: `helm package`, `helm install`, `helm upgrade`, and `helm dependency build`. We are unaware of any public exploits caused by this issue. Details Helm charts can include symlinks. This feature provides a means of symlinking chart dependencies together when stored in a filesystem. Two types of symlinks can cause vulnerabilities. 1. A symlink to a specially crafted file on the targets system. For example, a file containing sensitive information. When someone runs `helm package` this symlinked file will be copied into the archive without any notification. When the packaged file is moved elsewhere, such as to a Helm repository, the sensitive file will be sent along. 2. A symlink to a special file, such as a device driver. For example, a symlink to `/dev/urandom`. This would cause a command like `helm install` or `helm package` to continuously read from `/dev/urandom` as it tries to create the payload. No Tiller version is impacted. This vulnerability does not render clusters vulnerable to attack. Tiller does not load chart directories. Workarounds A process work around can be used to mitigate the vulnerability. Do not load chart directories or package charts whose contents you do not trust or in an environment with sensitive information. Fix Update to Helm >= 2.15.2. As of Helm 2.15.2, Helm will log to output all of the symlinked files referenced when packaging a chart and it will return an error while failing to load chart directories containing symlinks to irregular files (e.g., device or unix socket).
|
|
|
|
Re: Troubleshooting a Helm installation
Josh Dolitsky
I might also recommend bypassing your remote Tiller entirely to reduce a lot of ops headache. I've been using the helm-tiller plugin which starts a Tiller server locally for the lifetime of your command. You just prefix all of your helm commands with "helm tiller run" - for example "helm tiller run helm list". This should ideally pick up your existing releases. Beyond that, you could also try to start using Helm 3 (rc1), and convert your releases using the helm-2to3 plugin with the --tiller-out-cluster flag. Josh
On Wed, Oct 30, 2019 at 7:37 AM Joe Thompson <kensey@...> wrote: On Oct 30, 2019, at 05:26, bert.laverman@... wrote:
|
|
|
|
Re: Troubleshooting a Helm installation
Joe Thompson <kensey@...>
On Oct 30, 2019, at 05:26, bert.laverman@... wrote:
First place I’d start would be watching the Tiller pod logs, just to see if there’s anything obvious there (`kubectl get logs -f [tiller pod]` while you’re trying to use the Helm client). But from Tiller’s perspective I suspect everything is fine, and this is the client complaining *about* Tiller. But it might also be Tiller, complaining that the client cert is bad in some way (e.g. not signed by a trusted CA). So the next step is, check the certificates Tiller is using to identify itself and the CA it’s using to verify client certificates (Tiller mounts a Secret with the necessary data, so you should see that Secret and its mount info referenced in the pod spec if you inspect it with `kubectl get pod -o yaml [tiller pod]`, then you can inspect and decode the secret and see if what’s there is correct). Make sure the CA that generated your `helm` client certs is one Tiller trusts, and that the CA that generated the Helm certs is one the client trusts. Past that point there are some more advanced things you can do like dumping the Tiller cert directly with `openssl s_client`, but I’ll bet that after checking the above you’ll have already found the problem and know how to fix it. — Joe
|
|
|
|
Troubleshooting a Helm installation
bert.laverman@...
Hello Helm!
I have a Helm installation set up to install some of our DevOps toolchain, which generally works fine and as such tends to get "ignored" on the Ops side. As a result helm commands are only run rarely (once every few months). My current task started with "Let's add a PostGreSQL installation, and then ran into "Bad Certificate" errors. Unfortunately there is no Helm-for-Helm, and the documentation describes how to setup/install Helm securely pretty well, but not so much how to find out why it stopped working. All applications I have installed work fine, but Helm itself refuses to do anything. My last bit of work was on refreshing the certificates (which naturally ran out unnoticed), and I could reinstall our Ingress Controller after that, but 4 weeks onwards it refuses to budge. "helm list --tls --debug" simply responds with [debug] Created tunnel using local port: '55563' [debug] SERVER: "127.0.0.1:55563" [debug] Key="/Users/bertlaverman/.helm/key.pem", Cert="/Users/bertlaverman/.helm/cert.pem", CA="/Users/bertlaverman/.helm/ca.pem" Error: remote error: tls: bad certificate This does not really tell me anything beyond that I have some kind of SSL related issue. Which certificate is in trouble, Helm or Tiller, is unknown, nor what exactly is Helm's beef with it. Who can help me with some basic troubleshooting guidelines? I don't mind cleaning the resulting story up for addition to the docs, but for now I'm kind of looking at having to spend days on trying to find out what happened here. Cheers, Bert
|
|
|
|
Helm v3.0.0-rc.1 has been released
Matt Fisher <matt.fisher@...>
Helm v3.0.0-rc.1 has just been released. The release notes can be found here:
https://github.com/helm/helm/releases/tag/v3.0.0-rc.1
This is the first release candidate for Helm 3. It's been a long time coming. Thank you to everyone who has been helping out opening issues
or providing fixes for Helm 3.
Right now, we are actively looking for users willing to help test Helm 3 before the first release is published. We would love to hear
your feedback as well as any issues you may find with this release candidate.
If you find any issues, please don't hesitate to open a ticket in the Helm issue queue. Also, please feel free to ask any questions/concerns you have with the release candidate in the #helm-dev Slack
channel.
Thank you for all your help and support!
Matthew Fisher Caffeinated Software Engineer
Microsoft Canada
|
|
|
|
Helm 2.15.0 Released
Matt Fisher <matt.fisher@...>
Helm 2.15.0 was released last Friday. The 2.15.0 release of Helm introduces several improvements to commands like `helm test`. Other commands
- `helm search`, `helm repo list`, and `helm install` - received the `--output` flag for machine-readable output.
In addition to these new features (and many more!), many bugs and edge cases in Helm continue to fixed by members of the community. Several parts of the codebase have been refactored for easier maintainability, usability, and better testing.
As Helm moves towards Helm 3's first release, Helm 2 is now transitioning into "maintenance mode". Helm 2.15.0 will be the final feature release for Helm 2.
During the transition, we want to take into account any migration issues for users due to the upcoming holiday shopping season and tax season. We also want to clarify what actions may occur after the support contract ends for Helm 2, so that users will
not be surprised or caught off guard.
For Helm 2, we will continue to accept bug fixes and fix any security issues that arise, but no new features will be accepted. All feature development will be moved over to Helm 3.
6 months after Helm 3's public release, Helm 2 will stop accepting bug fixes. Only security issues will be accepted.
12 months after Helm 3's public release, support for Helm 2 will formally end. Download links for the Helm 2 client through Google Cloud Storage, the Docker image for Tiller stored in Google Container Registry, and the Google Cloud buckets for the stable
and incubator chart repositories may no longer work at any point. Client downloads through get.helm.sh will continue to work, and we will distribute a Tiller image that will be made available at an alternative location which can be updated with `helm init
--tiller-image`.
With Helm 2 transitioning into maintenance mode, now was a good opportunity to switch branches over.
The Helm team would like to thank the community for
nearly 3 years of contributions and
over 5,000 commits towards Helm 2 since its initial release.
Onwards and upwards!
Matthew Fisher Caffeinated Software Engineer
Microsoft Canada
|
|
|
|
Re: Last v2 feature release
Matt Fisher <matt.fisher@...>
Good question. At this time, we do not have a cut date for 2.15. We may have to come up with a planned merge window for this release so contributors are aware of the deadline.
However, I imagine the cut window would be fairly short - likely within the next two weeks. Users have reported issues using Helm with Kubernetes 1.16 and there's
a PR open to fix that. Once that's merged, we will likely be pressured to release 2.15 for that fix.
Matthew Fisher Caffeinated Software Engineer
Microsoft Canada
From: cncf-helm@... <cncf-helm@...> on behalf of Ihor Dvoretskyi via Lists.Cncf.Io <ihor=cncf.io@...>
Sent: Thursday, September 26, 2019 10:48 AM To: Taylor Thomas <taylor@...> Cc: cncf-helm@... <cncf-helm@...> Subject: Re: [cncf-helm] Last v2 feature release What is the estimated cut date for this?
On Thu, Sep 26, 2019, 8:18 PM Taylor Thomas <taylor@...> wrote:
|
|
|
|
Re: Last v2 feature release
What is the estimated cut date for this?
On Thu, Sep 26, 2019, 8:18 PM Taylor Thomas <taylor@...> wrote:
|
|
|
|
Last v2 feature release
Taylor Thomas
Hello all Helmettes! We have decided that Helm 2.15 will be the last feature release for Helm 2. This means that once it is cut, we will be moving the current master branch to be "release-2" and "dev-v3" will become the new master. Any currently open PRs with features will need to be targeted for v3 after that point. PLEASE NOTE: This is the last _feature_ release. We will still be releasing bug and security fixes as needed. Thank you for all of the feedback and for helping us maintain a great community! -Taylor
|
|
|
|
Detecting Helm v2 and v3 in scripts
Matt Farina
If you are going to work with both Helm v2 and Helm v3 in your scripts this email is where we are looking for your input. `helm version` in v2 and the forthcoming v3 work a little different. This includes the `-c` flag for client side version only (not available in v3 where it causes an error). You can get a rundown at https://github.com/helm/helm/issues/6322 Marc Khouzam noted that the acceptance-testing repo uses the following to detect the versions...
This relies on the error response to detect the versions. There is a proposed PR to add and ignore the `-c` flag to v3 in over to get the version when it's used. This means a script would need to parse the version to tell the difference between v2 and v3. But, it would provide the ability to read and detect the version with more accurate (e.g., detect minor versions) The PR to add this flag is at https://github.com/helm/helm/pull/6368 If you are into scripting with helm which path would you prefer? Do you need to detect minor versions? If you have any feedback please feel free to leave it on the PR or in response to this message.
|
|
|
|
Re: Helm in-depth of Q/A for InfoQ
Matt Butcher <matt.butcher@...>
Sure! Please contact me directly and I will get things rolling.
Matt Butcher
From: cncf-helm@... <cncf-helm@...> on behalf of via Lists.Cncf.Io <raghavan.srinivas=gmail.com@...>
Sent: Tuesday, September 17, 2019 9:53 AM To: Raghavan Srinivas <rags@...> Cc: cncf-helm@... <cncf-helm@...> Subject: Re: [cncf-helm] Helm in-depth of Q/A for InfoQ Matt/all:
Can someone help me please? Especially if you attended the summit. I can work with you even if you did not attend but consider yourself as a core Helm contributor.
Thanks!
Rags
|
|
|
|
Re: Helm in-depth of Q/A for InfoQ
Rags Srinivas <raghavan.srinivas@...>
Matt/all:
toggle quoted messageShow quoted text
Can someone help me please? Especially if you attended the summit. I can work with you even if you did not attend but consider yourself as a core Helm contributor. Thanks! Rags
On Sep 10, 2019, at 3:41 AM, Raghavan Srinivas <rags@...> wrote:
|
|
|
|
Re: Helm in-depth of Q/A for InfoQ
Raghavan "Rags" N. Srinivas <rags@...>
Hi all: Now that the Helm summit is over, can someone please get to the helm for this in-depth Q/A? I don't want to lose momentum. Thanks! Rags
On Wed, Sep 11, 2019 at 9:44 AM Raghavan "Rags" N. Srinivas <rags@...> wrote:
|
|
|
|
Re: Migrating Helm 2 to Helm 3
Rimas
And my blog post how to migrate to Helm v3 is published https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/ as well.
|
|
|
|
Migrating Helm 2 to Helm 3
Martin Hickey
Hi Folks,
The plugin for migrating Helm v2 configuration and releases in-place to Helm v3 is now available: https://github.com/helm/helm-2to3 Please check it out and give it a spin. Note: That the plugin is in beta like Helm 3 and should not be used in production yet. Regards, Martin
|
|
|
|
Re: Helm in-depth of Q/A for InfoQ
Raghavan "Rags" N. Srinivas <rags@...>
Hi all: Just a gentle ping. Realize everyone must be busy with the Helm summit which is why I am pushing to do this right after. Matt, any chance you can prod someone please? :-) Thanks! Rags
On Tue, Sep 10, 2019 at 6:41 AM Raghavan Srinivas <rags@...> wrote:
|
|
|
|
Re: Helm in-depth of Q/A for InfoQ
Raghavan Srinivas <rags@...>
Hello Helm maintainers:
I reached out to Matt regarding an in-depth q and a for infoq. Since he does not have the bandwidth can anyone else step in here (his suggestion to reach out to list). It will take a max of 3 to 4 hours of your time spread over
a few days.
Thanks!
Rags
Get Outlook for iOS
From: Raghavan Srinivas <rags@...>
Sent: Monday, September 9, 2019 10:25:30 AM To: Matt Fisher <Matt.Fisher@...> Subject: Helm in-depth of Q/A for InfoQ Matt:
Hope you’r having a great Helm Summit. Too bad, I am not in Amsterdam ☺
I part time write for InfoQ (as you probably know) and would like to do an indepth Helm Q/A similar to
https://www.infoq.com/profile/Rags-Srinivas/#allActivity
Would you have some bandwith (will take just about 2-3 hours of your time max to answer about 6-8 questions)? I am thinking of pushing this out this week or latest next and only after you green light the preview that I’ll send out.
Hopefully this works for you? LMK if I need to reach out to anyone else as well? If you’re not the right person (don’t know who else can it be for Helm ☺) then LMK as well.
Thanks!
Rags
|
|
|
|
Add my Udemy course to Related Projects and Documentation section
usample12@...
Can I add my Udemy course (https://www.udemy.com/course/helm-the-kubernetes-package-manager-course) to Related Projects and Documentation section (https://helm.sh/docs/related/). Can anyone guide me through the steps ?
|
|
|
|
Install apps using helm3 on multiple clusters with --kube-context
siddhesh.divekar@...
Hi,
In helm2 "helm init" would install tiller using current cluster context. Can I do this in parallel on multiple clusters using "--kube-context". We are just starting to use helm so leaning towards using helm3. With helm3 since tiller is gone, can I use --kube-context option to install packages on multiple clusters in parallel ? Will there be any conflicts with helm3 ? Thanks, Siddhesh
|
|
|
|
Re: How to handle experiment of Helm charts in OCI repos when 3.0.0 comes out
Matt Farina
+ Steve
On Wed, Aug 21, 2019, at 11:42 PM, Matt Farina wrote:
|
|
|
