New idea proposal vetting - split large Helm encoded release data across multiple K8s secrets
Some of our charts create during install/upgrade secrets that are bigger than 1MB. Then Helm install/upgrades fail.
We can take some measures to remove content and cleanup charts but we can only do so much.
Our idea is to have Helm create multiple secrets to hold a Helm release.
The encoded release data would be split across multiple secrets created as needed if the encoded release data is > 1MB.
We have a working implementation of this implemented as an additional storage driver called "chunkedsecrets" which can be enabled by setting the HELM_DRIVER=chunkedsecrets environment variable.
But we feel it is better to integrate chunking into the standard secret/secrets storage driver and add automatic chunking only if the size of the encoded release data becomes too big.
title: "Support Helm release data stored in multiple K8s Secrets"
authors: [ "Ralf Van Dorsselaer" ]
Helm release data encoded into the release field in a Kubernetes Secret can cause the Secret as a whole to exceed 1 MiB (as defined by MaxSecretSize in kubernetes/pkg/apis/core/types.go).
When this happens, the Kubernetes Secret cannot be stored in Etcd and the Helm install or upgrade fails.
Splitting the Helm release data across multiple Kubernetes Secrets resolves this problem.
Helm release data that causes the Kubernetes Secret size to exceed MaxSecretSize will make Helm install or upgrade fail.
The existing HELM_DRIVER=secret storage driver does not handle Kubernetes Secrets become too big when storing a large encoded Helm release data.
The logical place to support this is in the HELM_DRIVER=secret storage driver.
The feature is transparent to users.
When the HELM_DRIVER=secret storage driver notices that the resulting Kubernetes Secret will exceed MaxSecretSize then the storage driver will split the encoded release data across multiple Kubernetes Secrets.
Splitting requires additional metadata to be added to the Kubernetes Secret:
- number of chunks: integer, the number of Kubernetes Secrets that holds the release data for this Helm release version.
- chunk: integer, the chunk number. Each Secret gets a chunk number. Reconstituting the release data can be done by reading the Secret with chunk=1, then subsequently reading in numerical order any number of additional Secrets up to and including the "number of chunks".
Splitting requires the name of the Secret to include a "chunk" suffix. Example: sh.helm.release.v1.test.v1.1 where the number after the last dot is the "number of chunk".
If no splitting is required then no metadata and no dot"number of chunk" suffix are included/set.
## Backwards compatibility
If the chunking is implemented in the secret/secrets HELM_DRIVER then backward compatibility to previously installed Helm releases is guaranteed. If no chunking information is encoded in the release secret then the driver can behave as before chunking support was added.
## Security implications
Theoretically a MTM Man in The Middle attack is possible where requests to the Kubernetes API server are intercepted. But this type of attack applies to any access to the Kubernetes API server.
## How to teach this
The feature is transparent to users.
The documentation (in advanced section(s)) needs to explain that multiple Kubernetes Secrets can be created for a Helm release to counter the size limitation.
## Reference implementation
TODO add link to our PR.
## Rejected ideas
TODO Why certain ideas that were brought while discussing this HIP were not
## Open issues
TODO Any points that are still being decided/discussed.
TODO A collection of URLs or materials used as references through the HIP.