1. Cncf-Helm
  2. Topics

Feedback on Helm

bloritsch@...
 

I'm in the process of modernizing our deployment to use containers and kubernetes to deploy a complex microservices application.  The current deployment uses python scripts to place things where they need to go, and while it works, that deployment model doesn't deliver on some promises we made.

We have around 30 different services we need to deploy, most of which are Spring Boot based.  The bottom line is that the deployment for individual services should be reasonably the same.  The availability check would use the Spring Actuator health endpoint, and the only a few things need to be distinct (such as service port, ingress path, resource limits, and things of that nature).

My goal was to have a very small deployment of three microservices (discovery, config, and one of the simpler non-database services).  There's some things that I like:

  • Templating makes it easier to customize deployments
  • The default template is a good 80% solution, which I was able to make some needed customizations
  • It wasn't hard to figure out how to make starters for my project
  • I can collocate the chart definition with my service source code

That said, there are a lot of rough edges:

  • You can't avoid the templating language if you customize your deployment, and if go-lang isn't something you do in your shop, it can be challenging to do otherwise simple things.
  • The templates themselves are part of the chart.  That means if I convert 15 out of 30 services to Helm, and discover something I need to fix in the generated templates, I have to change my starter and 15 services to provide that piece.
  • Helm does not have transitive package resolution, like most package management software I am used to using (Maven, Nuget, etc.)

I'm hoping I am missing something, or that there are already plans to address these rough edges.  The biggest problem is the transitive package resolution.  My helm charts look like this:
  • logging-configmap
  • system-configmap
  • eureka
    • logging-configmap
  • config-server
    • eureka
    • system-configmap
  • simple-service
    • eureka
    • config-server
The remainder of my ~30 services will be like simple-service.  In order for a successful deployment of that to happen I need the two other dependent services.  However each of those declare a copy of their dependencies, and so on.  When I run the command "helm dependency build" I will have the sub-charts loaded.

Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes.

At any rate, I would like to help make helm better.  I do have time constraints, and I'm not very familiar with go-lang.  I think this is a good start in a mostly correct direction.  It just needs to address those rough edges.

--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708

Matt Farina
 

Berin,

Thanks for such an extensive write-up. I can't address all of it now but I wanted to talk about transitive dependencies. I am curious what you are looking for in respect to that as far as handling goes.

When you have a tree of dependencies you can't exactly flatten them. For example, if you have a dependency on WordPress and Drupal and they both have a dependency on MySQL you can't flatten the tree to have one MySQL. You will end up with two as they are operating as services, possible with very different configurations, for each of those. So, the transitive handling is mildly different from something like a programming language package manager.

When `helm package` is run (to create the archive) the dependencies are bundled in the `charts` directory. These are the direct dependencies. Each of the direct dependencies dependencies would be bundled in its own `charts` directory and so on.

This is why I ask about your use case. I wonder what's going on. I'd like to better understand it and the usage pattern.

Cheers,
Matt Farina

On Fri, May 1, 2020, at 3:16 PM, Loritsch, Berin wrote:
I'm in the process of modernizing our deployment to use containers and kubernetes to deploy a complex microservices application.  The current deployment uses python scripts to place things where they need to go, and while it works, that deployment model doesn't deliver on some promises we made.

We have around 30 different services we need to deploy, most of which are Spring Boot based.  The bottom line is that the deployment for individual services should be reasonably the same.  The availability check would use the Spring Actuator health endpoint, and the only a few things need to be distinct (such as service port, ingress path, resource limits, and things of that nature).

My goal was to have a very small deployment of three microservices (discovery, config, and one of the simpler non-database services).  There's some things that I like:

  • Templating makes it easier to customize deployments
  • The default template is a good 80% solution, which I was able to make some needed customizations
  • It wasn't hard to figure out how to make starters for my project
  • I can collocate the chart definition with my service source code

That said, there are a lot of rough edges:

  • You can't avoid the templating language if you customize your deployment, and if go-lang isn't something you do in your shop, it can be challenging to do otherwise simple things.
  • The templates themselves are part of the chart.  That means if I convert 15 out of 30 services to Helm, and discover something I need to fix in the generated templates, I have to change my starter and 15 services to provide that piece.
  • Helm does not have transitive package resolution, like most package management software I am used to using (Maven, Nuget, etc.)

I'm hoping I am missing something, or that there are already plans to address these rough edges.  The biggest problem is the transitive package resolution.  My helm charts look like this:
  • logging-configmap
  • system-configmap
  • eureka
    • logging-configmap
  • config-server
    • eureka
    • system-configmap
  • simple-service
    • eureka
    • config-server
The remainder of my ~30 services will be like simple-service.  In order for a successful deployment of that to happen I need the two other dependent services.  However each of those declare a copy of their dependencies, and so on.  When I run the command "helm dependency build" I will have the sub-charts loaded.

Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes.

At any rate, I would like to help make helm better.  I do have time constraints, and I'm not very familiar with go-lang.  I think this is a good start in a mostly correct direction.  It just needs to address those rough edges.

--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708



Paul Czarkowski <pczarkowski@...>
 

Berin,

A few things to address your template concerns.  

Firstly you can create your own chart starter packs (https://helm.sh/docs/topics/charts/#chart-starter-packs) which you can base subsequent charts off, this is really useful to create a sane set of defaults for similar apps in the same languages. You can also use Library Charts (https://helm.sh/docs/topics/library_charts/) to reduce overlap/copy/paste and generally create DRYer charts.

As for "Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes."   

It's pretty common to include the dependencies in your charts so that the apps can be installed independently and start their own eureka/config-server etc, but when deployed to staging/prod you'd disable the dependencies (see condition - https://helm.sh/docs/topics/charts/#tags-and-condition-fields-in-dependencies ) and instead have the infrastructure deployed as its own thing.  This also gives you the ability to maintain the infrastructure pieces of your application at a different lifecycle as your actual applications. You'd still use the rest of the settings for that subchart to configure your own application to talk to the upstream server.

If you have pretty complex relationships between the apps and need to be installed in a very ordered way, you can also reach towards a higher level tool like Helmfile (a declarative spec for deploying helm charts) to perform that work (Helm tries really hard not to boil the ocean). I use Helmfile extensively for this very purpose.

Also of note, Spring has some very good integrations with Kubernetes, giving you options for using kubernetes itself for service discovery and config server, which may in a homogenous environment allow you to simplify some of your app's infrastructure.  I'm far from a Spring expert, but I work pretty closely with a number of the Spring team and they're working hard to make Kubernetes a first class platform for Spring, especially when it comes to the developer experience itself.

If there's anything specific I can do to help, I'd be happy to get further into the weeds with you.

On Fri, May 1, 2020 at 2:16 PM Loritsch, Berin <bloritsch@...> wrote:
I'm in the process of modernizing our deployment to use containers and kubernetes to deploy a complex microservices application.  The current deployment uses python scripts to place things where they need to go, and while it works, that deployment model doesn't deliver on some promises we made.

We have around 30 different services we need to deploy, most of which are Spring Boot based.  The bottom line is that the deployment for individual services should be reasonably the same.  The availability check would use the Spring Actuator health endpoint, and the only a few things need to be distinct (such as service port, ingress path, resource limits, and things of that nature).

My goal was to have a very small deployment of three microservices (discovery, config, and one of the simpler non-database services).  There's some things that I like:

  • Templating makes it easier to customize deployments
  • The default template is a good 80% solution, which I was able to make some needed customizations
  • It wasn't hard to figure out how to make starters for my project
  • I can collocate the chart definition with my service source code

That said, there are a lot of rough edges:

  • You can't avoid the templating language if you customize your deployment, and if go-lang isn't something you do in your shop, it can be challenging to do otherwise simple things.
  • The templates themselves are part of the chart.  That means if I convert 15 out of 30 services to Helm, and discover something I need to fix in the generated templates, I have to change my starter and 15 services to provide that piece.
  • Helm does not have transitive package resolution, like most package management software I am used to using (Maven, Nuget, etc.)

I'm hoping I am missing something, or that there are already plans to address these rough edges.  The biggest problem is the transitive package resolution.  My helm charts look like this:
  • logging-configmap
  • system-configmap
  • eureka
    • logging-configmap
  • config-server
    • eureka
    • system-configmap
  • simple-service
    • eureka
    • config-server
The remainder of my ~30 services will be like simple-service.  In order for a successful deployment of that to happen I need the two other dependent services.  However each of those declare a copy of their dependencies, and so on.  When I run the command "helm dependency build" I will have the sub-charts loaded.

Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes.

At any rate, I would like to help make helm better.  I do have time constraints, and I'm not very familiar with go-lang.  I think this is a good start in a mostly correct direction.  It just needs to address those rough edges.

--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708

Loritsch, Berin <bloritsch@...>
 

Thanks for your response.  I am creating starter templates, which will help once I've got the basics nailed down.  I'm only on my second component and found things I needed to modify in the templates I customized.  Hopefully by the third one I'll be almost complete.  The library charts are something that slipped my attention.  There just isn't a lot of articles that help with the trade-offs and how to go about a decent sized microservices deployment.  One of the things that attracted me to Helm in the first place was the ability to modularize the deployment packages.

I am going to have to look into the library charts to see if they scratch the itch I have for the microservices deployment.  I'm kind of drinking from a firehose here since learning how to write charts means also learning how Kubernetes works--and most tutorials are too basic for the level of detail I need.

I am aware of the Spring integrations with Kubernetes, but with an established application I do need to take care with this lift into kubernetes infrastructure.  I fully intend to ditch Eureka and potentially Config Server in favor of services and config maps once we know what we are doing and how we want to do it.  It's hard enough getting something that is working to continue to work in a container environment.  As you said, don't boil the ocean.

The biggest questions I have right now deal with namespace management.  Helm 3 removed creating namespaces, which then makes the idea of having a hierarchical namespace harder.  For example, I would want a master namespace for each deployment (dev, int, qa, prod), but break down my microservices so they are part of a subsection.  The namespaces do allow me to set policies independently for each group of microservices.  Within "dev" I might have "dev-product1", "dev-product2", "dev-shared", "dev-frontend", "dev-data-ingest" as sub namespaces.  Do I create a helm chart to render the namespace creation?

On Fri, May 1, 2020 at 4:55 PM Paul Czarkowski <pczarkowski@...> wrote:
Berin,

A few things to address your template concerns.  

Firstly you can create your own chart starter packs (https://helm.sh/docs/topics/charts/#chart-starter-packs) which you can base subsequent charts off, this is really useful to create a sane set of defaults for similar apps in the same languages. You can also use Library Charts (https://helm.sh/docs/topics/library_charts/) to reduce overlap/copy/paste and generally create DRYer charts.

As for "Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes."   

It's pretty common to include the dependencies in your charts so that the apps can be installed independently and start their own eureka/config-server etc, but when deployed to staging/prod you'd disable the dependencies (see condition - https://helm.sh/docs/topics/charts/#tags-and-condition-fields-in-dependencies ) and instead have the infrastructure deployed as its own thing.  This also gives you the ability to maintain the infrastructure pieces of your application at a different lifecycle as your actual applications. You'd still use the rest of the settings for that subchart to configure your own application to talk to the upstream server.

If you have pretty complex relationships between the apps and need to be installed in a very ordered way, you can also reach towards a higher level tool like Helmfile (a declarative spec for deploying helm charts) to perform that work (Helm tries really hard not to boil the ocean). I use Helmfile extensively for this very purpose.

Also of note, Spring has some very good integrations with Kubernetes, giving you options for using kubernetes itself for service discovery and config server, which may in a homogenous environment allow you to simplify some of your app's infrastructure.  I'm far from a Spring expert, but I work pretty closely with a number of the Spring team and they're working hard to make Kubernetes a first class platform for Spring, especially when it comes to the developer experience itself.

If there's anything specific I can do to help, I'd be happy to get further into the weeds with you.

On Fri, May 1, 2020 at 2:16 PM Loritsch, Berin <bloritsch@...> wrote:
I'm in the process of modernizing our deployment to use containers and kubernetes to deploy a complex microservices application.  The current deployment uses python scripts to place things where they need to go, and while it works, that deployment model doesn't deliver on some promises we made.

We have around 30 different services we need to deploy, most of which are Spring Boot based.  The bottom line is that the deployment for individual services should be reasonably the same.  The availability check would use the Spring Actuator health endpoint, and the only a few things need to be distinct (such as service port, ingress path, resource limits, and things of that nature).

My goal was to have a very small deployment of three microservices (discovery, config, and one of the simpler non-database services).  There's some things that I like:

  • Templating makes it easier to customize deployments
  • The default template is a good 80% solution, which I was able to make some needed customizations
  • It wasn't hard to figure out how to make starters for my project
  • I can collocate the chart definition with my service source code

That said, there are a lot of rough edges:

  • You can't avoid the templating language if you customize your deployment, and if go-lang isn't something you do in your shop, it can be challenging to do otherwise simple things.
  • The templates themselves are part of the chart.  That means if I convert 15 out of 30 services to Helm, and discover something I need to fix in the generated templates, I have to change my starter and 15 services to provide that piece.
  • Helm does not have transitive package resolution, like most package management software I am used to using (Maven, Nuget, etc.)

I'm hoping I am missing something, or that there are already plans to address these rough edges.  The biggest problem is the transitive package resolution.  My helm charts look like this:
  • logging-configmap
  • system-configmap
  • eureka
    • logging-configmap
  • config-server
    • eureka
    • system-configmap
  • simple-service
    • eureka
    • config-server
The remainder of my ~30 services will be like simple-service.  In order for a successful deployment of that to happen I need the two other dependent services.  However each of those declare a copy of their dependencies, and so on.  When I run the command "helm dependency build" I will have the sub-charts loaded.

Problem is when the kubernetes template is actually rendered I have a new instance of the logging-configmap for each service, etc.  I need to be able to build and deploy my services in isolation as well as part of the whole for testing purposes.

At any rate, I would like to help make helm better.  I do have time constraints, and I'm not very familiar with go-lang.  I think this is a good start in a mostly correct direction.  It just needs to address those rough edges.

--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708



--
Berin Loritsch

Systems Integration Lead


7921 Jones Branch Drive
McLean, VA 22102
Email bloritsch@...

Office (703) 735-6281

Mobile (571) 215-7708

1 - 4 of 4