New idea proposal vetting - split large Helm encoded release data across multiple K8s secrets


ralfv
 
Edited

Hello,

Idea

Some of our charts create during install/upgrade secrets that are bigger than 1MB. Then Helm install/upgrades fail.
 
We can take some measures to remove content and cleanup charts but we can only do so much.
 
Our idea is to have Helm create multiple secrets to hold a Helm release.
 
The encoded release data would be split across multiple secrets created as needed if the encoded release data is > 1MB.
 
We have a working implementation of this implemented as an additional storage driver called "chunkedsecrets" which can be enabled by setting the HELM_DRIVER=chunkedsecrets environment variable.
 
But we feel it is better to integrate chunking into the standard secret/secrets storage driver and add automatic chunking only if the size of the encoded release data becomes too big.

Proposal

---
hip: 9999
title: "Support Helm release data stored in multiple K8s Secrets"
authors: [ "Ralf Van Dorsselaer" ]
created: "2022-02-10"
type: "feature"
status: "draft"
---
 
## Abstract
 
Helm release data encoded into the release field in a Kubernetes Secret can cause the Secret as a whole to exceed 1 MiB (as defined by MaxSecretSize in kubernetes/pkg/apis/core/types.go).
 
When this happens, the Kubernetes Secret cannot be stored in Etcd and the Helm install or upgrade fails.
 
Splitting the Helm release data across multiple Kubernetes Secrets resolves this problem.
 
## Motivation
 
Helm release data that causes the Kubernetes Secret size to exceed MaxSecretSize will make Helm install or upgrade fail.
 
The existing HELM_DRIVER=secret storage driver does not handle Kubernetes Secrets become too big when storing a large encoded Helm release data.
 
## Rationale
 
The logical place to support this is in the HELM_DRIVER=secret storage driver.
 
## Specification
 
The feature is transparent to users. 
 
When the HELM_DRIVER=secret storage driver notices that the resulting Kubernetes Secret will exceed MaxSecretSize then the storage driver will split the encoded release data across multiple Kubernetes Secrets.
 
Splitting requires additional metadata to be added to the Kubernetes Secret:
 
- number of chunks: integer, the number of Kubernetes Secrets that holds the release data for this Helm release version.
- chunk: integer, the chunk number. Each Secret gets a chunk number. Reconstituting the release data can be done by reading the Secret with chunk=1, then subsequently reading in numerical order any number of additional Secrets up to and including the "number of chunks".
 
Splitting requires the name of the Secret to include a "chunk" suffix. Example: sh.helm.release.v1.test.v1.1 where the number after the last dot is the "number of chunk".
 
If no splitting is required then no metadata and no dot"number of chunk" suffix are included/set.
 
## Backwards compatibility
 
If the chunking is implemented in the secret/secrets HELM_DRIVER then backward compatibility to previously installed Helm releases is guaranteed. If no chunking information is encoded in the release secret then the driver can behave as before chunking support was added.
 
## Security implications
 
Theoretically a MTM Man in The Middle attack is possible where requests to the Kubernetes API server are intercepted. But this type of attack applies to any access to the Kubernetes API server.
 
## How to teach this
 
The feature is transparent to users.
 
The documentation (in advanced section(s)) needs to explain that multiple Kubernetes Secrets can be created for a Helm release to counter the size limitation.
 
## Reference implementation
 
TODO add link to our PR.
 
## Rejected ideas
 
TODO Why certain ideas that were brought while discussing this HIP were not
ultimately pursued.
 
## Open issues
 
TODO Any points that are still being decided/discussed.
 
## References
 
TODO A collection of URLs or materials used as references through the HIP.

Thanks,
Ralf

Join cncf-helm@lists.cncf.io to automatically receive all group messages.