Supporting pre-existing Helm Charts and k8s manifests as artifacts is still on the roadmap, and that needs to be achieved first before we can achieve a GitOps philosophy like Weave Cloud.
> and that needs to be achieved first before we can achieve a GitOps philosophy like Weave Cloud.
I think "[automatically] export all generated YAML templates and CLI commands as a git commit / git patch" is something that may be worth supporting even without the import of existing templates.
First, it should be much easier, since you're already doing the hard work of generating those strings. Exporting them as a git patch could be as straightforward as keeping an internal repo folder and running git format-patch.
Second, it would make for a much easier sell to greenfield projects (which don't need import), because if your GUI tool can export back to the usual YAML code, I might consider using it knowing that I can 'eject' at any time, or that if the generated YAML suffers from any ShuttleOps bugs I can at least fix them and apply the fixed YAML manually while I wait for a bugfix.
1. Pre-kub: orchestration is manual through AWS EC2 ui
2. Kub: orchestration is automated and configuration is treated as code (YAML, JSON, proto, whatever) and kept in revision control just like source code
3. Post-kub: orchestration is manual through a ui
In all seriousness I'm sure this is useful in limited cases, but for most the right answer isn't some fancy UI, its middleware or some layer on top of kubernetes that allows you to automate away the toil-y parts of YAML, but keep the flexibility.
What I like about Kubernetes is that it’s a platform you get things like exponential backoff, replication, liveness and health checking and so forth as part of the platform, and plugins for metric collection and monitoring are fairly standard. These are things I don’t even feel most PaaS have done a great job on across the board, so I can certainly see the appeal of having Kubernetes under the hood.
Yeah, there's a whole category of apps that run on heroku with maybe 10 dynos tops that could definitely be deployed on kubernetes with manual / web ui-based configuration.
I actually don’t see why a PaaS couldn’t do all of this better. Especially at larger companies, where containers and monitoring have been the norm internally forever, it is unclear why PaaS offerings still feel primitive... maybe there is some technology limitation here.
Edit: disclaimer, I have not used a PaaS service in a couple years.
Yeah that is understandable, k8s is a platform with a plethora of capabilities. On the other hand, the vast majority of organizations using k8s have multiple users that have multiple work streams and multiple applications (not true by definition, but close). This lends itself to a code as configuration approach, and makes managing orchestration through a UI risky and error prone. We review code, not clicks.
Further, K8s exposes an YAML interface because it is relatively human readable and of course is machine readable. I'm sure the k8s community encourages users to build on top of this. Want to really do code as config? Write a nice DSL in your favorite scripting language to produce these YAMLs. Submit that as source. You've now automated away the vast majority of toil related to k8s YAML config, because your code can be:
I've been experimenting on a personal project with https://www.pulumi.com/ where you can do the configuration and deployment in Python or JavasScript and I think I can recommend it.
> In all seriousness I'm sure this is useful in limited cases, but for most the right answer isn't some fancy UI, its middleware or some layer on top of kubernetes that allows you to automate away the toil-y parts of YAML, but keep the flexibility.
I think Dhall is in the right category of solutions for this problem. YAML needs to be reusable and text templates a la Helm are an amusingly bad solution. Moreover, I think static typing is a very nice feature for things that generate YAML since it’s so easy to make mistakes with these kinds of tools. I’ve used Starlark for these kinds of tasks in the past and the lack of static typing has been a sore point.
Hi! ShuttleOps allows you to create workflows for deploying to Kubernetes without writing YAML manifest files. In that way you could consider it a visual/gui for the terraform kubernetes provider, sure!
Thanks for the question! Currently you can view what has changed on the pipeline executions page. Having a pre-deployment planning phase is on the roadmap as well.
I'm not a k8s expert. Just touched it a bit here and there. And IMO yaml is fine but the problem is you often need yaml templates.
I feel like projects like helm are in the templating html with php days. But instead of html+php you get yaml+go. Something better must be right around the corner because people have been solving similar problems for years.
No offense intended, but a generic marketing elevator pitch for a technical product makes me grumpy. I want to see the technical details, but I'm not signing up for your website just to dig for them. So this creates a missed opportunity where people like me will now not share it with their team.
Comments is one, and while JSON5 seems to support them, it does not look widely adopted or maintained. I do not like YamHell either, but prefer to JSON for readability
I'm using Cuelang these days, Dhall is another option
They are both crappy tools for humans to write anything in and are better as a tools for machines to exchange data, which is accidentally human-readable in case debugging is neccessary.
This quote style in YAML is its best feature, IMO.
It's not massively useful for k8s outside of config maps, but for other uses of YAML, it's great for embedding snippets of other languages that would otherwise need quoting.
Folding and highlighting in the editor make it much easier.
I use yafolding minor mode in Emacs, it folds based on indentation level, and it handles indentation-oriented syntaxes like Python and YAML as well as block-oriented syntaxes like C, Java, JS and Ruby.
yaml-mode in Emacs highlights :| text in the quoted string highlight colour, so it's easy to see you're still in the block, or whether you've unindented your way out to the YAML again.
I generally try and split out my yaml such that the quoted blocks don't need to be enormous, though. So, single SQL statements, rather than a whole SQL script, to take one motivating example.
I also use highlight-indentation-current-column mode, for an extra visual aid.
Developers directly touching k8s are hardly low/no code targets. So this doesn’t seem well placed.
Unless this imagines a world where devs throw over the fence and ops team is bunch of citizen users.
I mean, in an ideal world, it should be a low/no-code target. In the sense that you should be able to manipulate K8s as a dumb user to make it do what you need to do, when you need it, without somebody in your way.
A company called QualiSystems made some software called CloudShell that was basically a web UI to creating architectural diagrams and hooking each component up to code. There's drivers pre-made for each piece. You connect the pieces, click "Deploy app", and it builds everything you need, deploys your app, runs tests, tears it all down after, etc. The devs literally assemble the whole system in a UI and then just "run" the whole thing. Ops is essentially no longer needed except to write drivers and define policy/governance. This is what we need more of.
Instead, we have Ops teams who are in charge of infrastructure, and spend weeks writing custom Terraform modules and CI/CD solutions just to get some basic infrastructure up. Oh, you need a change? Sure, it'll just be another day full of writing and testing code for me to make that one tiny change in infra. It's ridiculous. Devs should be able to do this work themselves as needed, without waiting for someone else to do it for them. The same idea applies to K8s.
Personally I think "deploying to K8s without YAML" is not a challenging problem for most teams as there are lots of different solutions to that. Where they actually need help is in gaining agency and independence, and to do that we need friendly user interfaces to automated guide-rails, so if the dev wants to deploy a new stack, nothing is getting in their way, but the end result is according to a corporate standard.
There’s healthy level of valid pain point you’re citing (custom terraform module, unnecessary long change cycles).
And still, LC/NC isn’t convincing here.
Deployment is one piece. Day 2 ops another.
Deployment: shouldn’t be over complex for devs.
K8s declarative design is actually supportive of this.
Any team which creates undue complexity here is incompetent or has wrong leadership in place.
Day 2 ops is another beast. You get errors, things break, unexpected happens. You can minimize that by architecting a healing, resilient sub-system. But never eliminate it entirely. Expert humans need to step in when it breaks. Not dumb users.
Need of ops team doesn’t go away. And if this team exists, LCNC is not a fit.
If you have to abstract something, do it darn good. If you can’t, don’t step in my way.
56 comments
[ 90.5 ms ] story [ 232 ms ] threadDoes your tool work with GitOps methodology?
Ps: Hofstadter looks cool!
I think "[automatically] export all generated YAML templates and CLI commands as a git commit / git patch" is something that may be worth supporting even without the import of existing templates.
First, it should be much easier, since you're already doing the hard work of generating those strings. Exporting them as a git patch could be as straightforward as keeping an internal repo folder and running git format-patch.
Second, it would make for a much easier sell to greenfield projects (which don't need import), because if your GUI tool can export back to the usual YAML code, I might consider using it knowing that I can 'eject' at any time, or that if the generated YAML suffers from any ShuttleOps bugs I can at least fix them and apply the fixed YAML manually while I wait for a bugfix.
https://hofstadter.io
1. Pre-kub: orchestration is manual through AWS EC2 ui
2. Kub: orchestration is automated and configuration is treated as code (YAML, JSON, proto, whatever) and kept in revision control just like source code
3. Post-kub: orchestration is manual through a ui
In all seriousness I'm sure this is useful in limited cases, but for most the right answer isn't some fancy UI, its middleware or some layer on top of kubernetes that allows you to automate away the toil-y parts of YAML, but keep the flexibility.
EDIT: line breaks.
Edit: disclaimer, I have not used a PaaS service in a couple years.
It takes containers to their logical conclusion by generalizing them to a function while removing quite some operational overhead on the way.
Further, K8s exposes an YAML interface because it is relatively human readable and of course is machine readable. I'm sure the k8s community encourages users to build on top of this. Want to really do code as config? Write a nice DSL in your favorite scripting language to produce these YAMLs. Submit that as source. You've now automated away the vast majority of toil related to k8s YAML config, because your code can be:
1. modular by design
2. tested
3. tracked as code
etc etc
I think Dhall is in the right category of solutions for this problem. YAML needs to be reusable and text templates a la Helm are an amusingly bad solution. Moreover, I think static typing is a very nice feature for things that generate YAML since it’s so easy to make mistakes with these kinds of tools. I’ve used Starlark for these kinds of tasks in the past and the lack of static typing has been a sore point.
EDIT: I don't even know how it is possible, this is an Oauth2 Flow and should never redirect to a non HTTP URL from Github.
Also: the fact that shuttleio doesn't have hsts or at least a 301 to HTTPS from HTTP is not brilliant either.
And contacted ShuttleIO by mail.
This could be ugly, Shuttle asks for broad permissions on Github.
I feel like projects like helm are in the templating html with php days. But instead of html+php you get yaml+go. Something better must be right around the corner because people have been solving similar problems for years.
There is also https://dhall-lang.org of similar philosophy and more mature
I personally hate YAML config files.
I'm using Cuelang these days, Dhall is another option
They are both crappy tools for humans to write anything in and are better as a tools for machines to exchange data, which is accidentally human-readable in case debugging is neccessary.
It's not massively useful for k8s outside of config maps, but for other uses of YAML, it's great for embedding snippets of other languages that would otherwise need quoting.
I use yafolding minor mode in Emacs, it folds based on indentation level, and it handles indentation-oriented syntaxes like Python and YAML as well as block-oriented syntaxes like C, Java, JS and Ruby.
yaml-mode in Emacs highlights :| text in the quoted string highlight colour, so it's easy to see you're still in the block, or whether you've unindented your way out to the YAML again.
I generally try and split out my yaml such that the quoted blocks don't need to be enormous, though. So, single SQL statements, rather than a whole SQL script, to take one motivating example.
I also use highlight-indentation-current-column mode, for an extra visual aid.
Its a no go for me and a small business to use this. 250 is just too much with infra costs at 200 and free doesn't give us any guarantee at all.
A company called QualiSystems made some software called CloudShell that was basically a web UI to creating architectural diagrams and hooking each component up to code. There's drivers pre-made for each piece. You connect the pieces, click "Deploy app", and it builds everything you need, deploys your app, runs tests, tears it all down after, etc. The devs literally assemble the whole system in a UI and then just "run" the whole thing. Ops is essentially no longer needed except to write drivers and define policy/governance. This is what we need more of.
Instead, we have Ops teams who are in charge of infrastructure, and spend weeks writing custom Terraform modules and CI/CD solutions just to get some basic infrastructure up. Oh, you need a change? Sure, it'll just be another day full of writing and testing code for me to make that one tiny change in infra. It's ridiculous. Devs should be able to do this work themselves as needed, without waiting for someone else to do it for them. The same idea applies to K8s.
Personally I think "deploying to K8s without YAML" is not a challenging problem for most teams as there are lots of different solutions to that. Where they actually need help is in gaining agency and independence, and to do that we need friendly user interfaces to automated guide-rails, so if the dev wants to deploy a new stack, nothing is getting in their way, but the end result is according to a corporate standard.
And still, LC/NC isn’t convincing here. Deployment is one piece. Day 2 ops another.
Deployment: shouldn’t be over complex for devs. K8s declarative design is actually supportive of this. Any team which creates undue complexity here is incompetent or has wrong leadership in place.
Day 2 ops is another beast. You get errors, things break, unexpected happens. You can minimize that by architecting a healing, resilient sub-system. But never eliminate it entirely. Expert humans need to step in when it breaks. Not dumb users.
Need of ops team doesn’t go away. And if this team exists, LCNC is not a fit.
If you have to abstract something, do it darn good. If you can’t, don’t step in my way.