This is very interesting. If you're managing hundreds of accounts pulling data even from Config from the master payer level is tedious and time consuming. If that data can be pulled out and queried with a real query language (config ql just doesn't cut it) it would make mass management so much easier.
Damn this looks really good. Data-first SQL integrated interfaces always get a +1 from me, managing resources without unified abstractions is always painful, especially when you have to do this at scale, standard CLIs always end up falling short in some way
Thanks! This is definitely one of the use-cases to have an SQL abstraction of many resources. Also the idea is to be able to define rules in cross-account, cross-apps - For example: making a rule/notification that depends on information in Okta and in GCP/AWS (This is pretty simple with SQL join).
Regarding Custodian - This is definitely a great tool with it's own advantages. Some of the dis-advantages that lead me to start cloudquery is that it has a proprietary query language that you have to learn and as such is more limited than SQL (The example above won't be possible out-of-the-box).
Thanks! This is on the roadmap, I'm not sure it will be via Insert statements but there will be an option of action/change via some level of abstraction that is not via API/writing code.
There's a lot of stuff that needs to happen for that, and error-handling would be quite tricky.
This would require either having a logic to "converge" the infrastructure – meaning implementing a big chunk of Terraform and adding some features that would make this thing look like a Kubernetes cousin – or having the insert statements blocking for minutes at a time.
Yes, a complete support for insert will definitely look like k8s/terraform cousin. As mentioned in the comment below we probably will support some specific pre-defined actions. Anyway, currently we only support query and as we gather more feedback we will add the action functionality which has it's set of nuances some of which you described very well.
1) Fetch data and put into a sql db
2) Query sql db
Wouldn't it be better, performance aside, if there was just one step, i.e. querying the cloud with SQL, essentially an SQL-to-cloud-APIs interface. Otherwise you're always operating on out-of-date data.
The idea/advantage to have the data in regular SQL database - is to be able to work with all the standard tools and analytics dashboard you have and familiar with (AWS quicksight, Google DataStudio, sqlite shell, etc...)
The flipside is that performing joins in a process across many document databases (which is what this approach would resemble) is wildly non-performant compared to doing it in a relational database.
But now every time you run a query without a where clause you're firing API calls to all AWS services. And then maybe recursive API calls on top of that.
This is fine if you're a personal account, but then you wouldn't need this tool in the first place.
I work on a product named http://vantage.sh/ that provides something similar as a managed service specifically for AWS. I respect the effort that went into this!
Very fair. We're about to ship a new marketing site with a features page that has more screenshots and feature explanations as well as a video demo on the home page.
18 comments
[ 2.7 ms ] story [ 48.4 ms ] threadAlso, relevant tool for similar applications (albeit not SQL based): https://github.com/cloud-custodian/cloud-custodian
Regarding Custodian - This is definitely a great tool with it's own advantages. Some of the dis-advantages that lead me to start cloudquery is that it has a proprietary query language that you have to learn and as such is more limited than SQL (The example above won't be possible out-of-the-box).
If I could create resources with INSERT statements, even better.
There's a lot of stuff that needs to happen for that, and error-handling would be quite tricky.
This would require either having a logic to "converge" the infrastructure – meaning implementing a big chunk of Terraform and adding some features that would make this thing look like a Kubernetes cousin – or having the insert statements blocking for minutes at a time.
The idea/advantage to have the data in regular SQL database - is to be able to work with all the standard tools and analytics dashboard you have and familiar with (AWS quicksight, Google DataStudio, sqlite shell, etc...)
This is fine if you're a personal account, but then you wouldn't need this tool in the first place.