|
Hi,
I work with a project ( https://troubleshoot.sh/) that includes a binary which executes tests on a cluster (or a host) prior to installation of an app. These tests generally include items which prove the environment meets the app's requirements for installation and operation, and happen prior to installation so that folks don't have to spend time diagnosing failed installs - the output just tells them what's missing and they can go fix it before trying again.
As our folks use Helm more and more, we're wondering if there's a variation on 'helm test' that might run before installation, rather than afterwards? And if not, is there appetite for a contributor to come along and add a command (e.g. `helm preflight`) to run a different set of tests prior to install? Of course we could use a plugin for that, however this feels like something which would be more about an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a feature like that.
Thanks Xav
-- Xav Paice (they/them) Engineering Manager
|
|
|
On Tue, Mar 28, 2023 at 6:53 PM Chris Berry < bez625@...> wrote: Hi Xav,
Could you not use a pre-install hook to achieve this for your projects?
Hi, thanks for the reply.
Yeah, I've looked at this but have a couple of concerns: - The pre-install hooks are really for setting things up prior to installation, right? So someone who uses them for that would find it confusing if we use them for some other purpose.
- When we run an install, even if it fails at the pre-install hook, doesn't that leave stuff behind which would need a "helm uninstall" to clean up?
The point here is to run an entirely separate test to ensure that the environment is ready prior to attempting an install.
What I'd really like to do is run all the preflight checks (fail or pass) prior to an install attempt, and have a reasonably clear and concise view of what failed so the end user knows what they need to go fix before the install can work.
If I'm barking up the wrong tree that's great.
Hi,
I work with a project ( https://troubleshoot.sh/) that includes a binary which executes tests on a cluster (or a host) prior to installation of an app. These tests generally include items which prove the environment meets the app's requirements for installation and operation, and happen prior to installation so that folks don't have to spend time diagnosing failed installs - the output just tells them what's missing and they can go fix it before trying again.
As our folks use Helm more and more, we're wondering if there's a variation on 'helm test' that might run before installation, rather than afterwards? And if not, is there appetite for a contributor to come along and add a command (e.g. `helm preflight`) to run a different set of tests prior to install? Of course we could use a plugin for that, however this feels like something which would be more about an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a feature like that.
Thanks Xav
-- Xav Paice (they/them) Engineering Manager
|
|
|
On Tue, Mar 28, 2023 at 7:50 PM Chris Berry < bez625@...> wrote: Hi Xav,
Regarding the intent the comment in the docs is "Helm provides a hook mechanism to allow chart developers to intervene at certain points in a release's life cycle" so no, I dont believe it is exclusively for environment setup. It's about the release life cycle and that could be a preflight check. Happy for others to correct on that.
I might have misunderstood the docs - certainly, it's worth having a play with it.
TIL :)
BTW I'm not a maintainer, I'm just trying to suggest options with the current version of helm. I've tried to PR changes in to helm before and it can be quite a lengthy process so if you can use existing features it might be the quickest win for you.
Really appreciate your response - I'm still quite green with Helm!
On Tue, Mar 28, 2023 at 7:33 AM Xav Paice < xav@...> wrote:
On Tue, Mar 28, 2023 at 6:53 PM Chris Berry < bez625@...> wrote: Hi Xav,
Could you not use a pre-install hook to achieve this for your projects?
Hi, thanks for the reply.
Yeah, I've looked at this but have a couple of concerns: - The pre-install hooks are really for setting things up prior to installation, right? So someone who uses them for that would find it confusing if we use them for some other purpose.
- When we run an install, even if it fails at the pre-install hook, doesn't that leave stuff behind which would need a "helm uninstall" to clean up?
The point here is to run an entirely separate test to ensure that the environment is ready prior to attempting an install.
What I'd really like to do is run all the preflight checks (fail or pass) prior to an install attempt, and have a reasonably clear and concise view of what failed so the end user knows what they need to go fix before the install can work.
If I'm barking up the wrong tree that's great.
Hi,
I work with a project ( https://troubleshoot.sh/) that includes a binary which executes tests on a cluster (or a host) prior to installation of an app. These tests generally include items which prove the environment meets the app's requirements for installation and operation, and happen prior to installation so that folks don't have to spend time diagnosing failed installs - the output just tells them what's missing and they can go fix it before trying again.
As our folks use Helm more and more, we're wondering if there's a variation on 'helm test' that might run before installation, rather than afterwards? And if not, is there appetite for a contributor to come along and add a command (e.g. `helm preflight`) to run a different set of tests prior to install? Of course we could use a plugin for that, however this feels like something which would be more about an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a feature like that.
Thanks Xav
-- Xav Paice (they/them) Engineering Manager
|
|
|
I've done a bit of experimenting to solidify what I'm thinking.
I tried making a pre-install hook, one that runs and one that fails. Real simple stuff, just a sleep and an exit. I added a deletion policy " helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation,hook-failed
That approach achieved the goal of "check things are good prior to installing my actual application". However, when I run an install, the stdout from the hook goes to logs, but there's no output provided to inform the user what is happening (if I sleep for 20s, there's just no output), and what failed if it did. I got a message about BackoffLimitExceeded which is fine, but had to go look at the log to see what the failed job was doing, which meant I had to remove the hook-failed deletion policy. I had hoped that --debug would help out, and it kind of did, but it's not what I want to put in front of a novice.
What I want to do is have a separate run that just tests "do I have the environmental conditions needed to be able to run the installation of this chart?". If not, I want output as to what failed (even if it's a dozen things), and if so I'd like to get a positive "yes". The `helm test` command does just what I'd like, but after installation rather than before.
For an illustration, take a look at https://troubleshoot.sh/docs/preflight/cluster-checks/ where we can run a command to check that the k8s version is within boundaries. We're considering if we add a way to read a CRD or a secret from a chart (and values.yaml) and feed it to the preflight command, or if it's better for us to add to Helm to have that built in.
On Tue, Mar 28, 2023 at 7:50 PM Chris Berry < bez625@...> wrote: Hi Xav,
Regarding the intent the comment in the docs is "Helm provides a hook mechanism to allow chart developers to intervene at certain points in a release's life cycle" so no, I dont believe it is exclusively for environment setup. It's about the release life cycle and that could be a preflight check. Happy for others to correct on that.
I might have misunderstood the docs - certainly, it's worth having a play with it.
TIL :)
BTW I'm not a maintainer, I'm just trying to suggest options with the current version of helm. I've tried to PR changes in to helm before and it can be quite a lengthy process so if you can use existing features it might be the quickest win for you.
Really appreciate your response - I'm still quite green with Helm!
On Tue, Mar 28, 2023 at 7:33 AM Xav Paice < xav@...> wrote:
On Tue, Mar 28, 2023 at 6:53 PM Chris Berry < bez625@...> wrote: Hi Xav,
Could you not use a pre-install hook to achieve this for your projects?
Hi, thanks for the reply.
Yeah, I've looked at this but have a couple of concerns: - The pre-install hooks are really for setting things up prior to installation, right? So someone who uses them for that would find it confusing if we use them for some other purpose.
- When we run an install, even if it fails at the pre-install hook, doesn't that leave stuff behind which would need a "helm uninstall" to clean up?
The point here is to run an entirely separate test to ensure that the environment is ready prior to attempting an install.
What I'd really like to do is run all the preflight checks (fail or pass) prior to an install attempt, and have a reasonably clear and concise view of what failed so the end user knows what they need to go fix before the install can work.
If I'm barking up the wrong tree that's great.
Hi,
I work with a project ( https://troubleshoot.sh/) that includes a binary which executes tests on a cluster (or a host) prior to installation of an app. These tests generally include items which prove the environment meets the app's requirements for installation and operation, and happen prior to installation so that folks don't have to spend time diagnosing failed installs - the output just tells them what's missing and they can go fix it before trying again.
As our folks use Helm more and more, we're wondering if there's a variation on 'helm test' that might run before installation, rather than afterwards? And if not, is there appetite for a contributor to come along and add a command (e.g. `helm preflight`) to run a different set of tests prior to install? Of course we could use a plugin for that, however this feels like something which would be more about an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a feature like that.
Thanks Xav
-- Xav Paice (they/them) Engineering Manager
|
|
|
I've been toying with the same idea (and planning to use
troubleshoot for it). My plan was to just use `kubectl apply`
something like:
helm template redpanda/redpanda --set mode=preflight | kubectl apply
-f - --wait
Or something like that. Would be even better if I could think of a
way to just get the logs as they're generated.
On March 27, 2023 7:06:07 PM PDT, "Xav
Paice via lists.cncf.io" <xav=replicated.com@...>
wrote:
Hi,
I work with a project ( https://troubleshoot.sh/) that
includes a binary which executes tests on a cluster (or a
host) prior to installation of an app. These tests
generally include items which prove the environment meets
the app's requirements for installation and operation, and
happen prior to installation so that folks don't have to
spend time diagnosing failed installs - the output just
tells them what's missing and they can go fix it before
trying again.
As our folks use Helm more and more, we're wondering if
there's a variation on 'helm test' that might run before
installation, rather than afterwards? And if not, is there
appetite for a contributor to come along and add a command
(e.g. `helm preflight`) to run a different set of tests
prior to install? Of course we could use a plugin for that,
however this feels like something which would be more about
an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a
feature like that.
Thanks
Xav
--
Sent from my Android device with K-9 Mail. Please excuse my
brevity.
|
|
|
The templating idea would work well - a couple of mods would be required in Troubleshoot:
* making it accept stdin (via a CLI switch probably) so we can use a pipe
* making it filter so that it only selects the yaml docs that are apiVersion: troubleshoot.sh/v1beta2 and kind: Preflight
Those changes are absolutely achievable.
The end result could be:
helm template somerepo/somechart | kubectl preflight --stdin
|
|
|
Looks good. Looks like you should be
able to do that today, without the flag:
kubectl preflight <(helm template somerepo/somechart)
With the flag, it might be nice if the short version of the flag
was '-f' so it would be consistent with both kubectl and helm.
On 4/3/23 8:00 PM, Xav Paice via lists.cncf.io wrote:
toggle quoted message
Show quoted text
The templating idea would work well - a couple of mods would be
required in Troubleshoot:
* making it accept stdin (via a CLI switch probably) so we can use
a pipe
* making it filter so that it only selects the yaml docs that are
apiVersion: troubleshoot.sh/v1beta2 and kind: Preflight
Those changes are absolutely achievable.
The end result could be:
helm template somerepo/somechart | kubectl preflight --stdin
|
|
|
Yeah it's a shame logs from hooks aren't output by the helm client. I did try to PR this in to helm, but it's been dying in the PR queue for over a year now https://github.com/helm/helm/pull/10309
On Wed, Mar 29, 2023 at 7:05 AM Joe Julian < me@...> wrote:
I've been toying with the same idea (and planning to use
troubleshoot for it). My plan was to just use `kubectl apply`
something like:
helm template redpanda/redpanda --set mode=preflight | kubectl apply
-f - --wait
Or something like that. Would be even better if I could think of a
way to just get the logs as they're generated.
On March 27, 2023 7:06:07 PM PDT, "Xav
Paice via lists.cncf.io" <xav=replicated.com@...>
wrote:
Hi,
I work with a project ( https://troubleshoot.sh/) that
includes a binary which executes tests on a cluster (or a
host) prior to installation of an app. These tests
generally include items which prove the environment meets
the app's requirements for installation and operation, and
happen prior to installation so that folks don't have to
spend time diagnosing failed installs - the output just
tells them what's missing and they can go fix it before
trying again.
As our folks use Helm more and more, we're wondering if
there's a variation on 'helm test' that might run before
installation, rather than afterwards? And if not, is there
appetite for a contributor to come along and add a command
(e.g. `helm preflight`) to run a different set of tests
prior to install? Of course we could use a plugin for that,
however this feels like something which would be more about
an execution order than a whole new set of software.
I'm happy to write up the HIP if there's appetite for a
feature like that.
Thanks
Xav
--
Sent from my Android device with K-9 Mail. Please excuse my
brevity.
|
|
|
