AFAICT, this actually isn't running SQL against data in DynamoDB. Rather, it syncs the the data in DynamoDB with their [Rockset's] own proprietary database, and runs SQL against that data.
It's effectively the same but technically quite different.
Link to a tutorial that is as simple with code examples / screenshots as this one?
I ask because I have looked and haven't found one that I could follow without having to dedicate several hours to it.
To the point where I want to move off of dynamodb to aurora because I can't believe how hard it is to get simple insights like how many rows exist out of dynamo (I inherited the dynamo choice would not have used it myself)
If you're running this regularly (e.g. once an hour) to dump tables the cost of Glue can really add up due a minimum runtime length.
We switched to a scheduled Fargate task to dump data from DynamoDB into S3 as parquet files. It's really reliable, costs us ~$4/month and completely configurable.
10 comments
[ 1.9 ms ] story [ 24.7 ms ] threadManaging a database can be pretty expensive and time consuming.
It's effectively the same but technically quite different.
I ask because I have looked and haven't found one that I could follow without having to dedicate several hours to it.
To the point where I want to move off of dynamodb to aurora because I can't believe how hard it is to get simple insights like how many rows exist out of dynamo (I inherited the dynamo choice would not have used it myself)
Should take about an hour with testing to get a Pyspark script together to read in a DynamoDB table and write it out to S3.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programm...
You’ll then need to crawl the S3 data to add it to your Glue catalog and then you can query it with Athena.
We switched to a scheduled Fargate task to dump data from DynamoDB into S3 as parquet files. It's really reliable, costs us ~$4/month and completely configurable.
https://aws.amazon.com/about-aws/whats-new/2018/07/aws-glue-...