The code is really just a pythonized version of the dot format. The real innovation is the fancy-looking icons. When using graphviz directly we have to futz around with images in the filesystem alongside the dot file, in order to avoid simple boxes.
> The code is really just a pythonized version of the dot format.
Not exactly. By utilizing Python as a base language you gain access to control structures as `for` loops, `if`-s and functions. It can make your diagram script much more powerful and/or more complicated.
This is a great idea. I'm not sure about the other major cloud providers, but AWS has the AWS CDK which allows you to use TypeScript, Python, Java, and .NET to create your cloud infrastructure via code. The ability to generate cloud diagrams from AWS CDK code would definitely would be awesome. Hopefully, your work gains transaction, then I can envision other devs writing adapter code for your project.
One problem is that you need meta-data to give weights to different resources so that you highlight a lambda resource and not there associated cloudwatch loggroup resource, and its iam role, and iam policy, and its permission resource, etc
The other problem is the graph itself shows how the cloudformation resources depend on one another, not how your data/execution pipeline flows.
So you need to do it manually or maybe try to add lots of meta-data to the types of resources.
I've used asciidoctor with it's diagram plugin (https://asciidoctor.org/docs/asciidoctor-diagram/) internally on memos/draft documentation, which that gives us an ability to actually just check in the asciidoc with diagram source into git. Then we use merge requests for a real review process. Then, I just usually just export to pdf or word for broader sharing among the masses. (Amazing how this usually gets far more attention than Confluence, etc)
Main benefit is that most of these kinds of CLI diagram tools have much longer lives than any GUI-based diagram software I've used.
But even just having a separate "diagram source" file where you just check in an image is still nice. Then you don't have to bother with wondering who has access to the tools, etc.
The entire contemporary devops process is awfully convoluted. In fact, I've heard DevOps people call themselves "yaml engineers". Often in your docker compose there is some stupid error, there are 3 versions of Docker compose files, there are 7 ways to do something, Docker CLI manual can be printed in 1000 pages and we've created extremely bloated (despite of caching whatever) CI/CD. For what? Dependencies? Let's download 4.5 GB of images because my app needs to be reproduceable.
I tried reading about Kubernetes. Holyshit what a monster we've created. I am sure there is going to be an orchestrator abstraction for multiple Kubernetes clusters. And then we have environment variables, secrets, access keys, a bazillion different parameters that need to trickle down from Gitlab CI settings all the way to the app through many layers.
Good lord. All engineers should be ashamed of this mess. We've forgotten how to build simple, beautiful, lean and parsimonious software. Instead of writing logic, I am configurating things all day.
Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers. But, let's shit on the F-35 program everytime there is a Lockheed article on HN. Because surely, software engineers are the smart bunch ya. I know, I am unreasonably obtuse here, but jeez, no one complains about today's deployment process,... its apparently accepted without a question.
> Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers
I'm with you on the feeling that modern stack is bloated and heavy... However, this is exactly the scenario where the compiler bug is isolated quickly because you're not chasing dependency differences between boxes
Kubernetes and Docker is about moving complexity, not removing complexity. I recently held a presentation for our UK applications team (manager, team lead) about our setup from a provisioning perspective, release process perspective, organizational perspective, QA perspective, operations perspective, troubleshooting perspective, full traceability, ease-of-deployment, time-to-market.
Needless to say they were amazed to see that we can deploy a new isolated, "virtual machine" application running in seconds using dynamic DNS, rather than waiting a year for a new application server from the IT organization not to speak about deploying it, operating it and troubleshooting it.
Finally related topic! I've been meaning to ask, is there any tool (or a suite of tools) that can be used to visualize network along with real-time health status of each nodes? Imagine you have a web server, API server and DB server, lined up left to right, each connected with a line. Each of them has green dot, and when they're down, the dot changes color to red.
The diagram can be manually laid out, but the "health indicator" must be updated automatically.
We have customers doing this with yFiles. But that's a generic graph drawing and layout library, so getting from your network topology to a graph structure and getting health events is not something the library helps with. You'd get automatic layout, however, and colored dots are on the trivial side for a visualization.
I started doing diagrams from code and don't want to go back to dragging things with a mouse again. VSCode plantuml gives live feedback. I started using this for c4 diagrams
https://github.com/RicardoNiepel/C4-PlantUML
30 comments
[ 1.8 ms ] story [ 85.3 ms ] threadNot exactly. By utilizing Python as a base language you gain access to control structures as `for` loops, `if`-s and functions. It can make your diagram script much more powerful and/or more complicated.
Cfn2dot source https://github.com/tobiipro/support-firecloud/blob/master/bi...
Used like this https://github.com/tobiipro/support-firecloud/blob/master/re...
One problem is that you need meta-data to give weights to different resources so that you highlight a lambda resource and not there associated cloudwatch loggroup resource, and its iam role, and iam policy, and its permission resource, etc
The other problem is the graph itself shows how the cloudformation resources depend on one another, not how your data/execution pipeline flows.
So you need to do it manually or maybe try to add lots of meta-data to the types of resources.
Another tool in that space is https://cloudcraft.co/ although only for AWS, and you have to manually do the diagram layout.
Main benefit is that most of these kinds of CLI diagram tools have much longer lives than any GUI-based diagram software I've used.
But even just having a separate "diagram source" file where you just check in an image is still nice. Then you don't have to bother with wondering who has access to the tools, etc.
I tried reading about Kubernetes. Holyshit what a monster we've created. I am sure there is going to be an orchestrator abstraction for multiple Kubernetes clusters. And then we have environment variables, secrets, access keys, a bazillion different parameters that need to trickle down from Gitlab CI settings all the way to the app through many layers.
Good lord. All engineers should be ashamed of this mess. We've forgotten how to build simple, beautiful, lean and parsimonious software. Instead of writing logic, I am configurating things all day.
Let there be a day where there is some deep compiler problem and this entire house of cards comes crashing down because your DevOps team ain't gonna know what happened down in lower abstraction layers. But, let's shit on the F-35 program everytime there is a Lockheed article on HN. Because surely, software engineers are the smart bunch ya. I know, I am unreasonably obtuse here, but jeez, no one complains about today's deployment process,... its apparently accepted without a question.
I'm with you on the feeling that modern stack is bloated and heavy... However, this is exactly the scenario where the compiler bug is isolated quickly because you're not chasing dependency differences between boxes
There are tons of them, people find new ones all the time. You troubleshoot it like every other problem.
Hell, you’ve barely scratched the surface of all the ways a kubernetes install can break — coredns? Etcd? Nginx? Istio?
My god what if there’s a bug in the Linux kernel? We should just give up, computers were a mistake.
Needless to say they were amazed to see that we can deploy a new isolated, "virtual machine" application running in seconds using dynamic DNS, rather than waiting a year for a new application server from the IT organization not to speak about deploying it, operating it and troubleshooting it.
The diagram can be manually laid out, but the "health indicator" must be updated automatically.