|
Hi. It was suggested to me that this topic may be of interest to the GitOps WG. I'd be happy to come discuss it during a meeting, or on the mailing list. We created a WYSIWYG GUI experience layered over GitOps that reduces the friction of interacting with Kubernetes configuration and git. This shows a simple use case to start (not app delivery), but I think shows the potential of the approach. The experience has no YAML editing, no patch authoring, no templates, no coding for configuration consumers, and low coding for platform builders. It leverages a similar transformation-based approach as kustomize that I call Configuration as Data. There's also no manual branching, editing, committing, tagging, pushing, merging, etc., which sounds like not a big deal, but having done it by hand lots of times, it's more friction than one might think, especially for small changes or for non-developers. https://cloud.google.com/blog/products/containers-kubernetes/lets-improve-gitops-usabilityhttps://www.youtube.com/watch?v=L_x7z4CXHDwhttps://twitter.com/bgrant0607/status/1525109177504280576 https://twitter.com/bgrant0607/status/1526231548109869061https://kpt.dev/guides/rationaleTake a look and let me know what you think. It's in the early stages, but I'm excited about it. We are looking to engage with the community to advance the technology forward. Project contact info is here: https://kpt.dev/contact/The code can be found here: https://github.com/orgs/GoogleContainerTools/repositories?q=kpt&type=all&language=&sort=
|
|
|
Josh Gavant
Thanks for sharing Brian. The most significant bit to consider IMO is that kpt enables syncing back to the git repo, whereas the likes of Flux and ArgoCD are one-way only (AFAIK) - the ArgoCD and Flux controllers read config from git but can't push changes back.
The ability to sync back to the git repo in turn enables you to provide a forms-based GUI over kpt and then use kpt to push changes back.
This could lead us to ask: should other GitOps tools also offer an "edit in GUI and sync back to git" option? And/or should such a capability be developed independently of any specific GitOps operator?
|
|
|
Stefan Prodan
Flux controllers read config from git but can't push changes back.
Flux can write back to Git, but only for container image update automation. Flux image-reflector-controller scans container registries to find new versions, then Flux image-automation-controller patches the image tags based on user-defined policies, then commits and pushes the changes to Git. Like kpt v1, Flux uses kyaml setters to patch YAMLs in Git, to preserve comments. Docs here https://fluxcd.io/docs/guides/image-update/
|
|
|
Thank you for bringing this to the group. I legit wish I had this when I was working in the Duke Health system. Would've made our lives much easier.
I can see extensive use of this being made in highly regulated environments right off the bat. This is also useful if you want to create safe, happy paths for folks to release their services/code with the downside of if you're not on the happy path you're going to have to go through other approval process(es) of some sort. This has been a practice for a while just never quite this tight (nor this native to Kubernetes).
w/r/t GUIs: By default, I don't think GUIs are necessary for GitOps tooling. If there is an edit source option in the GUI, a request should be made to commit a changeset through normal approval processes, not a direct commit to the source of truth. This can be managed a number of ways but, we should strive to encourage folks to build out tooling in a production-ready manner with appropriate safeguards either in place or documented where needed.
w/r/t writing back to the repo: To be clear, this needs to be safeguarded somehow; drift detection only works if there's something different to compare against. If you're saving your diffs back to the git repo while breaking your environment you'll wish you hadn't when you realize prod is down and there's 17 revisions to rollback through. I urge some caution here and ask folks to remember the rigor needed to approve changes to environments and/or known good configurations. That's where we as a subproject could write something up or reference something so folks can build out their approval processes in a GitOps way to save some headaches (approval processes only work if they're followed). He/Him/His TZ=America/Detroit
On Wed, May 25, 2022 at 4:59 AM Stefan Prodan <stefan@...> wrote:
Flux controllers read config from git but can't push changes back.
Flux can write back to Git, but only for container image update automation. Flux image-reflector-controller scans container registries to find new versions, then Flux image-automation-controller patches the image tags based on user-defined policies, then commits and pushes the changes to Git. Like kpt v1, Flux uses kyaml setters to patch YAMLs in Git, to preserve comments. Docs here https://fluxcd.io/docs/guides/image-update/
|
|
|
Thank you for bringing this to the group. I legit wish I had this when I was working in the Duke Health system. Would've made our lives much easier.
I can see extensive use of this being made in highly regulated environments right off the bat. This is also useful if you want to create safe, happy paths for folks to release their services/code with the downside of if you're not on the happy path you're going to have to go through other approval process(es) of some sort. This has been a practice for a while just never quite this tight (nor this native to Kubernetes).
The Backstage plugin is in its early stages. The review/approval flow needs to have at least diffs added. Some of that functionality can be seen in our earlier prototype:
There are also validation functions that run, akin to admission control in Kubernetes.
w/r/t GUIs: By default, I don't think GUIs are necessary for GitOps tooling. If there is an edit source option in the GUI, a request should be made to commit a changeset through normal approval processes, not a direct commit to the source of truth. This can be managed a number of ways but, we should strive to encourage folks to build out tooling in a production-ready manner with appropriate safeguards either in place or documented where needed.
Whatever safeguards are desired can be done through this flow.
In the prototype video above, we showed generation of pull requests. Pull requests are somewhat of a pain due to different APIs and models in different providers, and the need to set up branch protections and CODEOWNERS, map identities between providers and possibly impersonate users, and so on. We moved away from pull requests to automated branching and merging. We are not directly committing to the main branch. If the repo is only for configuration packages, then a configuration-change-focused workflow can be used.
Additionally, once fully automating configuration generation and the change control workflow, we found git was not helping us and caused some challenges, so we're in the progress of adding container images as storage, which can be versioned similarly to git, are ubiquitously available in production environments, are more granular, support package-level metadata, and have more standardized APIs than git providers.
w/r/t writing back to the repo: To be clear, this needs to be safeguarded somehow; drift detection only works if there's something different to compare against. If you're saving your diffs back to the git repo while breaking your environment you'll wish you hadn't when you realize prod is down and there's 17 revisions to rollback through. I urge some caution here and ask folks to remember the rigor needed to approve changes to environments and/or known good configurations. That's where we as a subproject could write something up or reference something so folks can build out their approval processes in a GitOps way to save some headaches (approval processes only work if they're followed).
Of course, monitoring the live state is desirable in a closed-loop system, just as if changes were made more manually.
Additionally, pulling from HEAD is tantamount to using the :latest tag on container images. It's simpler to orchestrate (well, except for rollbacks), but has similar risks.
He/Him/His TZ=America/Detroit
On Wed, May 25, 2022 at 4:59 AM Stefan Prodan <stefan@...> wrote:
Flux controllers read config from git but can't push changes back.
Flux can write back to Git, but only for container image update automation. Flux image-reflector-controller scans container registries to find new versions, then Flux image-automation-controller patches the image tags based on user-defined policies, then commits and pushes the changes to Git. Like kpt v1, Flux uses kyaml setters to patch YAMLs in Git, to preserve comments. Docs here https://fluxcd.io/docs/guides/image-update/
|
|
|
I tried to respond to this message yesterday, but the reply didn't go through.
The similar twitter thread is here: Thanks for sharing Brian. The most significant bit to consider IMO is that kpt enables syncing back to the git repo, whereas the likes of Flux and ArgoCD are one-way only (AFAIK) - the ArgoCD and Flux controllers read config from git but can't push changes back.
There are a number of other capabilities also, some mentioned in the twitter thread above.
The ability to sync back to the git repo in turn enables you to provide a forms-based GUI over kpt and then use kpt to push changes back.
This could lead us to ask: should other GitOps tools also offer an "edit in GUI and sync back to git" option? And/or should such a capability be developed independently of any specific GitOps operator?
The package orchestrator can be used with any GitOps operator. The only interaction with the GitOps sync mechanism currently is through git.
There's a small integration in the backstage UI plugin, to create the necessary sync resource and poll status, but it could be extended to support multiple GitOps tools, or could be made pluggable.
|
|
|
Chris Short <cbshort@...>
Will watch video for sure.
To be clear, you don't have to use git, any object store that's versioned will do. If git is in the way, use something that works and is versioned.
Also, I think it was Dan or Scott that said if you're pulling from latest all the time you're gonna have a bad time.
Chris Short
He/Him/His
Sr. Developer Advocate, AWS Kubernetes (GitOps)
TZ=America/Detroit
toggle quoted message
Show quoted text
|
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the
sender and know the content is safe.
|
Thank you for bringing this to the group. I legit wish I had this when I was working in the Duke Health system. Would've made our lives much easier.
I can see extensive use of this being made in highly regulated environments right off the bat. This is also useful if you want to create safe, happy paths for folks to release their services/code with the downside of if you're not on the happy path you're going
to have to go through other approval process(es) of some sort. This has been a practice for a while just never quite this tight (nor this native to Kubernetes).
The Backstage plugin is in its early stages. The review/approval flow needs to have at least diffs added. Some of that functionality can be seen in our earlier prototype:
There are also validation functions that run, akin to admission control in Kubernetes.
w/r/t GUIs: By default, I don't think GUIs are necessary for GitOps tooling. If there is an edit source option in the GUI, a request should be made to commit a changeset through normal approval processes, not a direct commit to the source of truth.
This can be managed a number of ways but, we should strive to encourage folks to build out tooling in a production-ready manner with appropriate safeguards either in place or documented where needed.
Whatever safeguards are desired can be done through this flow.
In the prototype video above, we showed generation of pull requests. Pull requests are somewhat of a pain due to different APIs and models in different providers, and the need to set up branch protections and CODEOWNERS, map identities between
providers and possibly impersonate users, and so on. We moved away from pull requests to automated branching and merging. We are not directly committing to the main branch. If the repo is only for configuration packages, then a configuration-change-focused
workflow can be used.
Additionally, once fully automating configuration generation and the change control workflow, we found git was not helping us and caused some challenges, so we're in the progress of adding container images as storage, which can be versioned similarly
to git, are ubiquitously available in production environments, are more granular, support package-level metadata, and have more standardized APIs than git providers.
w/r/t writing back to the repo: To be clear, this needs to be safeguarded somehow; drift detection only works if there's something different to compare against. If you're saving your diffs back to the git repo while breaking your environment you'll
wish you hadn't when you realize prod is down and there's 17 revisions to rollback through. I urge some caution here and ask folks to remember the rigor needed to approve changes to environments and/or known good configurations. That's where we as a subproject
could write something up or reference something so folks can build out their approval processes in a GitOps way to save some headaches (approval processes only work if they're followed).
Of course, monitoring the live state is desirable in a closed-loop system, just as if changes were made more manually.
Additionally, pulling from HEAD is tantamount to using the :latest tag on container images. It's simpler to orchestrate (well, except for rollbacks), but has similar risks.
He/Him/His
TZ=America/Detroit
On Wed, May 25, 2022 at 4:59 AM Stefan Prodan < stefan@...> wrote:
Flux controllers read config from git but can't push changes back.
Flux can write back to Git, but only for container image update automation. Flux image-reflector-controller scans container registries to find new versions, then Flux image-automation-controller patches the image tags based on user-defined policies, then commits
and pushes the changes to Git. Like kpt v1, Flux uses kyaml setters to patch YAMLs in Git, to preserve comments. Docs here https://fluxcd.io/docs/guides/image-update/
|
|
|
Thank you for bringing this to the group. I legit wish I had this when I was working in the Duke Health system. Would've made our lives much easier.
I can see extensive use of this being made in highly regulated environments right off the bat. This is also useful if you want to create safe, happy paths for folks to release their services/code with the downside of if you're not on the happy path you're going to have to go through other approval process(es) of some sort. This has been a practice for a while just never quite this tight (nor this native to Kubernetes).
w/r/t GUIs: By default, I don't think GUIs are necessary for GitOps tooling.
BTW, on GUIs being necessary:
A GUI isn't strictly necessary, but this makes it Possible. It also makes other types of automation possible.
It eliminates the requirement of manual editing of templates, patches, code written using configuration languages, or code written using general-purpose languages in an IDE or text editor in order to do something even as simple as adding a field to a resource. If there is an edit source option in the GUI, a request should be made to commit a changeset through normal approval processes, not a direct commit to the source of truth. This can be managed a number of ways but, we should strive to encourage folks to build out tooling in a production-ready manner with appropriate safeguards either in place or documented where needed.
w/r/t writing back to the repo: To be clear, this needs to be safeguarded somehow; drift detection only works if there's something different to compare against. If you're saving your diffs back to the git repo while breaking your environment you'll wish you hadn't when you realize prod is down and there's 17 revisions to rollback through. I urge some caution here and ask folks to remember the rigor needed to approve changes to environments and/or known good configurations. That's where we as a subproject could write something up or reference something so folks can build out their approval processes in a GitOps way to save some headaches (approval processes only work if they're followed). He/Him/His TZ=America/Detroit
On Wed, May 25, 2022 at 4:59 AM Stefan Prodan <stefan@...> wrote:
Flux controllers read config from git but can't push changes back.
Flux can write back to Git, but only for container image update automation. Flux image-reflector-controller scans container registries to find new versions, then Flux image-automation-controller patches the image tags based on user-defined policies, then commits and pushes the changes to Git. Like kpt v1, Flux uses kyaml setters to patch YAMLs in Git, to preserve comments. Docs here https://fluxcd.io/docs/guides/image-update/
|
|
|
