I love diagrams and pay for CloudCraft so I’m no stranger to the value of good diagram tools.
But I looked at that cloud formation design and immediately thought “crap that’s probably going to cost way too much to run”. Anyone got a good up to date cloud formation to dollars per month tool and want to estimate it? (I’m on a mobile at the moment)
The only service that appears to cost a flat fee is 'Amazon Neptune', which on the cheapest plan costs ~10 cents an hour or ~$70/month. I imagine all the other costs (e.g. S3, Fargate, API Gateway, CloudFront) are negligible in comparison.
Unfortunately not – this is what building modern cloud based software is like! This software complexity crisis is what we're trying to fix with Encore [1], if you're interested.
Not to mention this is something for which AWS already has all the data. They should just be providing this diagram as a free part of the AWS experience.
draw.io is a good alternative . By default it does not show icon for cloud services but if you click on "More Shapes" option at bottom left, you can add AWS,Azure and GCP and create all sorts for architecture diagrams.
Plus, it is opensource and can run your own self hosted draw.io app from github.
We are long time users of Lucidchart and were using the predecessor of Cloud Insights, AWS Import, for creating cloud diagrams of our existing AWS infrastructure. In February, a LucidChart account manager contacted us stating the AWS Import feature would be sunsetted. He offered to onboard our team onto Cloud Insights but we would have to upgrade to an Enterprise account at $199/license/year AND pay $4,000 a year for the Cloud Insights add-on that covers only up to 50 compute resources. We ended up keeping our account as-is and never heard back from the account manager.
Lucidchart is a great product but we were really turned-off & disappointed they essentially took a feature away from us.
I really would welcome this on my current stack as someone who now primarily works in GCP and is constantly updating architecture diagrams. I love any tool that will help create automated assets to assist with documentation as they really help prevent tribal knowledge and reduce that constant overhead of keeping everything up to date.
This isn't a "named AWS service" in the same vein of typical services like S3, EC2, etc. This is a "AWS Solution Implementation" which is a list of community created "solutions" built using AWS.
My strong suspicion is that this "AWS Perspective" is something that one of AWS's customers insisted on being built customized for their environment, and then the Solution Architects who designed it for the customer also wanted to get some "external credit" (AWS employees have to create a certain number of externally-facing blog posts/GitHub contributions/etc) so they uploaded it here.
Also "Frugality" and "Insist on the Highest Standards" seems to be missing from AWS Perspective (no pun intended, it's literally the name of the service)
I'm from AWS and my team are responsible for development of this project (AWS Perspective).
It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones.
To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can automatically create a diagrammatic representations of that data. Perspective also discovers the relationships between resources and will automatically expose them on the diagram.
The underlying data sits in a graph database (Amazon Neptune). This allows you to explore the relationships in the graph (diagram) on the fly visually to discover how your resources interact with one another.
We also have the early stages of a feature to show the estimated cost of the resources in the diagrams you create.
The project is open source and our roadmap is public. I encourage you to check it out and add any commentary for features you'd like to see (or bugs you find!).
I may be missing some complexity that's not clear from the description, but since all the resources should be available in AWS Config already, why is it using so much extra infrastructure? Why isn't it a "point me at AWS Config dump, I'll give you an explorable static page/SVG" script instead? (The cost part would have to reach out to billing and/or cloudwatch, but still...)
Or expressed a bit differently: what are the infra + 2 data stores doing that's a major improvement over processing raw AWS Config + CUR data which should easily fit in laptop memory. We're talking megabytes of data for a point in time snapshot, right?
Whilst Perspective sources a lot of data from AWS Config, it also supplements and enriches the data from other sources where resources are not supported or where we wanted to show more detail. We'll be adding in other data sources in future, too.
Additionally, the AWS Config query language is quite helpful but doesn't expose a graph of the resources. Our approach is to pull the individual relationship data out and represent those in a graph DB (Amazon Neptune). Backing on to a graph database gives us the ability to let users navigate through the relationships of interest on the fly, rather than dumping what would inevitably be a complicated mess on the screen if we just showed everything.
Using this approach is also one of the ways we've tried to solve the UI/UX challenges presented in this space. Rather than say "show me all the things" that would lead to a complex mess of interconnections (and probably explode browsers) we instead encourage a workflow that starts with a resource or workload and explores outwards from there. It also allows us to implement some interesting features in future -- we intend to keep working on the project.
The ElasticSearch cluster is used by the discovery process to locate resources based on the metadata we collect; it's part of the way we enrich the data and relationships beyond what we get from Config.
As with other comment, I should've been more clear. SVG / generated page doesn't mean no filtering. Just a point in time snapshot which doesn't require API calls once generated.
Speculation on my part, but this seems to be much more geared towards data analysis of your environment rather than just having an SVG for reference.
I'm assuming like most things AWS, this is not targeted at your small-sized startup running 2 VPCs, a handful of subnets, and 10-20 EC2 instances. This is likely built specifically for major enterprises running a hundred thousand instances, with hundreds of VPCs, thousands of subnets, hundreds of thousands of security groups, etc. In such environments, a static SVG isn't feasible and you might want something that allows more analytics instead, which I think is the entire point of using Neptune.
I should've been more clear, but don't confused a static page and SVG with static content. I was thinking closer to a one page interactive app generated from a data snapshot.
Is there really no screenshot of the visualisation, or is it just hard to find / hidden on mobile? I assume the architecture diagram is made in another tool.
Do you have a list of supported services? I've built a very similar solution using AWS Config to feed Neo4j and it has been incredibly useful, but Config's coverage is so anemic that we are left with just a tiny portion of our infrastructure visible.
The decentralized nature of Amazon / AWS rears its head again.
It's much harder, I assume, to get security / teams to allow another team to crawl through its customer data (the Lambda team owns which customers have how many Lambdas running where, for example). Now imagine doing that for every AWS resource to form a graph of relationships. To get around that they deployed something that customers are wholly responsible for.
Forcing customers to cover the bill for something AWS should provide out of the box, in this subpar experience where they also need to manage all the infrastructure, is the opposite of customer obsession.
>To get around that they deployed something that customers are wholly responsible for.
I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs).
This post is an "AWS Solutions Implementation". This isn't a service like S3/EC2/Lambda/etc created to be an integral part of AWS. A "Solutions Implementation" is basically the output of a customer making a very specific request from AWS Solutions Architects to build something customized (eg, customer doesn't want to use AWS Config but they want to build something from scratch with Lambda and RDS that works the same way), and then the solutions architects decide to share it publicly just in case any other customers find the same niche solution useful. It isn't meant for wide appeal or to be used by everyone.
I wouldn't be surprised if the fact that this post is getting decent attention means that AWS will use this solution as a proof of concept for investigating a future managed AWS service where the customer doesn't have to fully manage all of these moving pieces, but this post isn't that.
> I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs).
I doubt a customer asked AWS: hey, can I get a graph of how the resources in my accounts all interconnect while maintaining the clusters for the data stores, handling upgrade, security, etc. myself and paying hundreds of dollars a month for the pleasure?
The alternative is a customer asked for a way to see how resources were connected and how much it was costing them, and AWS, unable to deliver because of N hurdles (I mentioned security, but no doubt others), delivered this instead. Again: this should be a managed service and free.
>I doubt a customer asked AWS: hey, can I get a graph of how the resources in my accounts all interconnect while maintaining the clusters for the data stores, handling upgrade, security, etc. myself and paying hundreds of dollars a month for the pleasure?
I do consulting for a company that works with a lot of AWS customers and I think you'd be surprised. They don't put it in those exact terms, of course, but I have personal experience with a customer who specifically did want (I think due to their own internal regulatory standards) to maintain their own data stores rather than having an AWS managed service. In my example, it was using AWS Config versus having their own in-house software built on top of Lambda and RDS that did the same thing.
Another situation I've seen a lot is that really large companies have a tendency to think they are a super special snowflake with special use cases that no AWS-built service would ever fulfill, and thus they suffer from "not invented here syndrome". I've consulted many companies where I pitched them using an off-the-shelf product or service, but they will insist that it only covers 90% of their use cases and instead they will build their own custom solution using the AWS core services as building blocks, resulting in something similar to what is seen in the OP.
For the record, I think it's dumb as hell in most cases and I'm not in any way advocating that this solution in the OP is preferable in any way... but unfortunately I do understand how/why it was probably created.
Shameless plug: I wrote AWSets[1], which can be used to do some of this (although the graphs aren't as pretty). It was written because at work we regularly interact with hundreds of AWS accounts and wanted something to quickly scan/catalog all of the resources (launch blog post explains more [2]).
It's free to run locally (doesn't use or require Config), the data is normalized & output as JSON for you to do whatever you want with it. It has better coverage than similar tools (>250 resources & growing), and provides relationship mapping between them as well. It's still a work in progress and might have rough edges, so feel free to file issues to help make it better.
For a different angle on cloud infrastructure visualization, check out Abridge (https://abridge.io). I'd love to get any feedback on the idea / site and even a trial, if you're interested!
Abridge collects data across your configured set of AWS accounts & regions. It doesn't draw architecture diagrams, but instead will give you x-vs-y visualizations for a number of different relationships .. IAM users vs. groups, Lambda functions vs. runtimes, EC2 instances vs. keypair, etc.
It'll also provide inventory tables / CSV dumps of the different resource types, and simple free search across all resources.
Security-wise, Abridge collects data via cross-account trusts with the SecurityAudit role, and expires all collected data after 48 hours - more at https://abridge.io/security/.
Wait, this graphing tool literally requires you to deploy Neptune, Elasticsearch, ECS, various VPC's/gateways, publish various reports to S3, run Lambda functions, and then after all that the minimum cost is $500+/month?
Luckily it appears to be a "solution" instead of a core AWS product, otherwise I was about to say AWS has turned into a parody of itself.
This feels like a strange and expensive way to solve this problem.
1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure?
2. 500+ dollars a month for a service that I'll use very infrequently?
3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fine, what about things not in CF, and other metadata sure, but then see (1) and (2).
If this were, say, one dollar a month (or serverless) and the infrastructure was totally abstracted away I'd be a lot more excited.
edit: OK so (2) isn't really their fault, since this isn't an AWS service, but still. And 600 bucks isn't that bad for many companies, I just think there's a bit of a pricetag shock on this one.
The graph database is useful for external analysis of the relationships between your resources. I've built this in the past using just AWS Config and, while incredibly useful, it is extremely limited due to Config's very limited scope.
If this covers a reasonable percentage of AWS services it would be more than cost justified at $1-2k/mo for our environment.
This seems crazy for someone in a smaller account. I just want something that will chart out all my current resources allocated in a VPC/account and send to me once a week or so - so I can spot check if anything got accidentally left on.
I kind of feel like you would need $20k+/month spend for it to be worth $500 a month to do this.
57 comments
[ 2.5 ms ] story [ 46.1 ms ] threadBut I looked at that cloud formation design and immediately thought “crap that’s probably going to cost way too much to run”. Anyone got a good up to date cloud formation to dollars per month tool and want to estimate it? (I’m on a mobile at the moment)
ES network traffic wouldn’t cost if you are only retrieving traffic in the same region.
You can find an estimated cost breakdown here: https://docs.aws.amazon.com/solutions/latest/aws-perspective...
(Disclosure: My team developed this project)
[1] https://encore.dev
[1]: https://play.encore.dev/
Lucidchart is a great product but we were really turned-off & disappointed they essentially took a feature away from us.
My strong suspicion is that this "AWS Perspective" is something that one of AWS's customers insisted on being built customized for their environment, and then the Solution Architects who designed it for the customer also wanted to get some "external credit" (AWS employees have to create a certain number of externally-facing blog posts/GitHub contributions/etc) so they uploaded it here.
I'm from AWS and my team are responsible for development of this project (AWS Perspective).
It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones.
To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can automatically create a diagrammatic representations of that data. Perspective also discovers the relationships between resources and will automatically expose them on the diagram.
The underlying data sits in a graph database (Amazon Neptune). This allows you to explore the relationships in the graph (diagram) on the fly visually to discover how your resources interact with one another.
We also have the early stages of a feature to show the estimated cost of the resources in the diagrams you create.
The project is open source and our roadmap is public. I encourage you to check it out and add any commentary for features you'd like to see (or bugs you find!).
https://github.com/awslabs/aws-perspective
I'm also happy to answer any questions.
Or expressed a bit differently: what are the infra + 2 data stores doing that's a major improvement over processing raw AWS Config + CUR data which should easily fit in laptop memory. We're talking megabytes of data for a point in time snapshot, right?
That's a good question.
Whilst Perspective sources a lot of data from AWS Config, it also supplements and enriches the data from other sources where resources are not supported or where we wanted to show more detail. We'll be adding in other data sources in future, too.
Additionally, the AWS Config query language is quite helpful but doesn't expose a graph of the resources. Our approach is to pull the individual relationship data out and represent those in a graph DB (Amazon Neptune). Backing on to a graph database gives us the ability to let users navigate through the relationships of interest on the fly, rather than dumping what would inevitably be a complicated mess on the screen if we just showed everything.
Using this approach is also one of the ways we've tried to solve the UI/UX challenges presented in this space. Rather than say "show me all the things" that would lead to a complex mess of interconnections (and probably explode browsers) we instead encourage a workflow that starts with a resource or workload and explores outwards from there. It also allows us to implement some interesting features in future -- we intend to keep working on the project.
The ElasticSearch cluster is used by the discovery process to locate resources based on the metadata we collect; it's part of the way we enrich the data and relationships beyond what we get from Config.
I'm assuming like most things AWS, this is not targeted at your small-sized startup running 2 VPCs, a handful of subnets, and 10-20 EC2 instances. This is likely built specifically for major enterprises running a hundred thousand instances, with hundreds of VPCs, thousands of subnets, hundreds of thousands of security groups, etc. In such environments, a static SVG isn't feasible and you might want something that allows more analytics instead, which I think is the entire point of using Neptune.
No, the diagrams of Perspective were not created with Perspective -- we had to stick to existing style guidelines.
If there are more you'd like to see added, please raise an issue: https://github.com/awslabs/aws-perspective
An eventing system that lets you hook node/edge events world be awesome as well.
Will take a look tonight, really interested in this.
It's much harder, I assume, to get security / teams to allow another team to crawl through its customer data (the Lambda team owns which customers have how many Lambdas running where, for example). Now imagine doing that for every AWS resource to form a graph of relationships. To get around that they deployed something that customers are wholly responsible for.
Forcing customers to cover the bill for something AWS should provide out of the box, in this subpar experience where they also need to manage all the infrastructure, is the opposite of customer obsession.
I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs).
This post is an "AWS Solutions Implementation". This isn't a service like S3/EC2/Lambda/etc created to be an integral part of AWS. A "Solutions Implementation" is basically the output of a customer making a very specific request from AWS Solutions Architects to build something customized (eg, customer doesn't want to use AWS Config but they want to build something from scratch with Lambda and RDS that works the same way), and then the solutions architects decide to share it publicly just in case any other customers find the same niche solution useful. It isn't meant for wide appeal or to be used by everyone.
I wouldn't be surprised if the fact that this post is getting decent attention means that AWS will use this solution as a proof of concept for investigating a future managed AWS service where the customer doesn't have to fully manage all of these moving pieces, but this post isn't that.
I doubt a customer asked AWS: hey, can I get a graph of how the resources in my accounts all interconnect while maintaining the clusters for the data stores, handling upgrade, security, etc. myself and paying hundreds of dollars a month for the pleasure?
The alternative is a customer asked for a way to see how resources were connected and how much it was costing them, and AWS, unable to deliver because of N hurdles (I mentioned security, but no doubt others), delivered this instead. Again: this should be a managed service and free.
I do consulting for a company that works with a lot of AWS customers and I think you'd be surprised. They don't put it in those exact terms, of course, but I have personal experience with a customer who specifically did want (I think due to their own internal regulatory standards) to maintain their own data stores rather than having an AWS managed service. In my example, it was using AWS Config versus having their own in-house software built on top of Lambda and RDS that did the same thing.
Another situation I've seen a lot is that really large companies have a tendency to think they are a super special snowflake with special use cases that no AWS-built service would ever fulfill, and thus they suffer from "not invented here syndrome". I've consulted many companies where I pitched them using an off-the-shelf product or service, but they will insist that it only covers 90% of their use cases and instead they will build their own custom solution using the AWS core services as building blocks, resulting in something similar to what is seen in the OP.
For the record, I think it's dumb as hell in most cases and I'm not in any way advocating that this solution in the OP is preferable in any way... but unfortunately I do understand how/why it was probably created.
It's free to run locally (doesn't use or require Config), the data is normalized & output as JSON for you to do whatever you want with it. It has better coverage than similar tools (>250 resources & growing), and provides relationship mapping between them as well. It's still a work in progress and might have rough edges, so feel free to file issues to help make it better.
[1] - https://github.com/trek10inc/awsets [2] - https://www.trek10.com/blog/awsets-aws-resource-listing-made...
Abridge collects data across your configured set of AWS accounts & regions. It doesn't draw architecture diagrams, but instead will give you x-vs-y visualizations for a number of different relationships .. IAM users vs. groups, Lambda functions vs. runtimes, EC2 instances vs. keypair, etc.
It'll also provide inventory tables / CSV dumps of the different resource types, and simple free search across all resources.
Security-wise, Abridge collects data via cross-account trusts with the SecurityAudit role, and expires all collected data after 48 hours - more at https://abridge.io/security/.
Luckily it appears to be a "solution" instead of a core AWS product, otherwise I was about to say AWS has turned into a parody of itself.
1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure?
2. 500+ dollars a month for a service that I'll use very infrequently?
3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fine, what about things not in CF, and other metadata sure, but then see (1) and (2).
If this were, say, one dollar a month (or serverless) and the infrastructure was totally abstracted away I'd be a lot more excited.
edit: OK so (2) isn't really their fault, since this isn't an AWS service, but still. And 600 bucks isn't that bad for many companies, I just think there's a bit of a pricetag shock on this one.
If this covers a reasonable percentage of AWS services it would be more than cost justified at $1-2k/mo for our environment.
you might find this interesting then:
https://github.com/aws-cloudformation/cfn-python-lint/pull/1...
https://github.com/aws-cloudformation/aws-cfn-lint-visual-st...
I kind of feel like you would need $20k+/month spend for it to be worth $500 a month to do this.