Maybe I’m missing something, but looking at the examples in the README, I’m not sure what value this is providing over the boto3 DynamoDB client. Why would I want to use this library over the SDK client?
I hope DynamoDB is relegated to the niche use cases it's appropriate for. The hoops people are jumping through trying to use in place of say Aurora make for a very sad developer experience.
yea, the worst part is that the mistake isn't often realized until after the initial implementation is complete and it's time to iterate. The barrier to entry is very low so people don't bother taking the time to understand how it works until it's too late.
(I especially like how they market any schemaless database capable of strong consistency in the CAP sense as fully ACID compliant RDMS alternative, because if you can’t specify a rule making database states invalid than the consistency in the ACID sense is trivial because every database state is valid.)
Can you expand on that? This is my understanding so far, feel free to correct it:
It looks like a good place for quick key-values lookups. It's a "table" as a service, with 1 (or 2) columns that you can index on. Anything else and you need an RDBMS.
It's useful for simple storage, without the overhead of setting up an RDS.
The problem is that at the low-end where you just want something cheap and simple, it’s too complicated compare to Aurora. At the higher-end where you want something with extreme-performance, it’s got too many weird edge cases.
If you have a team of people that know how to use it well, I’m sure it’s an excellent tool. But it shouldn’t be casually thrown into an architecture, because there is nearly always a better solution.
The issue is that the keys can be used to various extent to handle relational data (this re:Invent video shows this well: https://youtu.be/HaEPXoXVf2k?t=1363). What people miss though is that in order to not shoot themselves in the foot, you have to have stable, known access patterns which meet the constraints of DynamoDB.
This means that it's a good use case for shopping cart of Amazon since there's limited innovation possible there and it fits the bill nicely, but for a new application with relational data where the requirements change all the time - it's a terrible, terrible choice for which you will pay dearly.
It is really way more than a 'table' as a service if you design it right. It is actually your entire database as a service if done right. AWS actually has some great documentation in this regard.
Dynamodb has some really great features and high performance when used correctly but it also has a steep learning curve especially for anyone coming from sql and other relational databases.
AWS tries to push the concept of single table design which allows you to store an entire application's data in a single dynamodb table and using a bunch of complicated indexing and overloading columns with multiple types of data to sort of get relational data without any joins.
This is really easy to get wrong and you need to have all your access patterns figured out ahead of time so that you can create the indices you'll need when you create the table. You can't create new local indices later... only global secondary indices which duplicate the entire table (and double your storage costs).
Dynamodb streams are really cool - great way to setup webhooks for data changes.
Dynamodb pay per request is super cheap and fast for a serverless db. Aurora serverless isn't pay per request and has slow cold starts.
My recommendations if you are using dynamodb because you want something serverless and NOT because nosql is actually the best storage option:
* don't do single table design - just make more requests to different tables and join your data in code
* if joining in code is too slow then use a sql database...
* abstract all your data access calls so you can eventually migrate to a sql database
Do you have a good recommendation for learning about what use cases DynamoDB makes sense for? I've used MongoDB and various SQL databases extensively, and when I tried DynamoDB it felt like a solution in search of a problem compared to my usual DBs of choice.
I built a pseudo-graph database for relational data for my company on DynamoDB that has been really amazing in both performance and utility. The data model is defined entirely by configuration, so adding use cases is just modifying the configuration. I've only had to touch the core system once or twice in 3 years and we've added about 10 use cases to it. The only thing it can't do well are things that a RDBMS has no trouble doing - namely getting all of one entity for analytical purposes, but we solved that problem using redshift and dynamoDB streams.
Hey cool! I'm super curious to hear more about this. I _also_ wrote a pseudo-graph-database on DynamoDB (https://github.com/aplbrain/grand) :) It pretends it's a networkx.Graph, generally, but we also have a Cypher implementation on top of it.
Would love to chat more about this sometime if you were interested!
I use that term because my system only allows 1 true level of depth before you have to query again. You also have to predefine the data model, but that's so that it can automatically choose the right entity type (to format the PK/SK correctly) and so that it supports automatic Global Secondary Index selection as it dynamically generates the query. It also means that each entity can only have up to 5 extra indexable key pairs.
Well, I was wrong. I just looked it up and you can now have 20 GSI. It used to be 5 haha.
Anyway, it would be cool to talk about your project. I'll check it out this week.
It is best in extreme cases. Some types of high scale use and super super cheap serverless/bootstrapping startup smaller scale.
Low-scale - it is easy to create a table and just start writing to it. The pay per request pricing is simple and plays well with IAM, other AWS services and the serverless framework.
High-scale - scales easily up and down, global replication is easy, backups and high availability are easy - it is all self-managed so a lot of this is baked in or very easy to toggle on. Depending on your ops resources, the easy tooling may be worth the extra costs or maybe your data lends it self particularly well to how dynamodb works.
Some of our use cases ....
- user profiles to supplement cognito user attributes
- chat/chat groups/chat group memberships/chat websocket watchers/chat messages (this one was not good. I would do it again with just messages and the websocket watchers)
- tenant data with lots of custom attributes
- forms and form submissions
- event webhooks from 3rd party services - catch/store/process using dynamodb streams
- step function states and logs - it has some nice versioning write patterns
- web scraping data from ebay
- web scraping data from golf tee time sites
When you need search or flexible querying you can put elasticsearch in front of it and use dynamodb streams to push items into the index. If you need analytics you can use another layer like redshift.
yes and no at decent load the only value it provides over Aurora is if you can eclipse performance Aurora scales to or you have extremely spiky workload.
A previous company I worked on had some sort of Serverless cargo-cult that forced us to use Dynamo as general purpose NoSQL DB, instead of something more sensible like Aurora or hell, even MongoDB in a EC2 instance.
It appears this was specifically made for use with IAM key/secrets; you should consider making these optional so that boto3 can pick up the corresponding environment variables itself. And you could also allow for temporary IAM credentials keys+secret+session.
The code as it stands is of course really dumb, but I think you can just pass `None` to each of those and have boto3 use the standard credential precedence.
On top of that they have mutable keyword arguments and generally quite terrible code: no exceptions thrown, just logging.error(), `if len(x)` rather than `if x`, etc etc.
I'll release my DynamoDB database, this one seems point less. Minor improvements at best. My one turns the table into a similar object to flask Session for quick
db.get('Pk.sk') or db.get('Pk.sk', ingore=['one'])
db.get('Pk.sk', model=pydantic_model)
db.update(...
db.put(...
Ect
Leave a comment if people are interested in me releasing this project.
33 comments
[ 3.3 ms ] story [ 74.6 ms ] threadIsn’t that always the case?
(I especially like how they market any schemaless database capable of strong consistency in the CAP sense as fully ACID compliant RDMS alternative, because if you can’t specify a rule making database states invalid than the consistency in the ACID sense is trivial because every database state is valid.)
It looks like a good place for quick key-values lookups. It's a "table" as a service, with 1 (or 2) columns that you can index on. Anything else and you need an RDBMS.
It's useful for simple storage, without the overhead of setting up an RDS.
If you have a team of people that know how to use it well, I’m sure it’s an excellent tool. But it shouldn’t be casually thrown into an architecture, because there is nearly always a better solution.
Even the documentation of DynamoDB says so:
https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
This means that it's a good use case for shopping cart of Amazon since there's limited innovation possible there and it fits the bill nicely, but for a new application with relational data where the requirements change all the time - it's a terrible, terrible choice for which you will pay dearly.
Dynamodb has some really great features and high performance when used correctly but it also has a steep learning curve especially for anyone coming from sql and other relational databases.
AWS tries to push the concept of single table design which allows you to store an entire application's data in a single dynamodb table and using a bunch of complicated indexing and overloading columns with multiple types of data to sort of get relational data without any joins.
This is really easy to get wrong and you need to have all your access patterns figured out ahead of time so that you can create the indices you'll need when you create the table. You can't create new local indices later... only global secondary indices which duplicate the entire table (and double your storage costs).
Dynamodb streams are really cool - great way to setup webhooks for data changes.
Dynamodb pay per request is super cheap and fast for a serverless db. Aurora serverless isn't pay per request and has slow cold starts.
My recommendations if you are using dynamodb because you want something serverless and NOT because nosql is actually the best storage option:
* don't do single table design - just make more requests to different tables and join your data in code
* if joining in code is too slow then use a sql database...
* abstract all your data access calls so you can eventually migrate to a sql database
Would love to chat more about this sometime if you were interested!
I use that term because my system only allows 1 true level of depth before you have to query again. You also have to predefine the data model, but that's so that it can automatically choose the right entity type (to format the PK/SK correctly) and so that it supports automatic Global Secondary Index selection as it dynamically generates the query. It also means that each entity can only have up to 5 extra indexable key pairs.
Well, I was wrong. I just looked it up and you can now have 20 GSI. It used to be 5 haha.
Anyway, it would be cool to talk about your project. I'll check it out this week.
It is best in extreme cases. Some types of high scale use and super super cheap serverless/bootstrapping startup smaller scale.
Low-scale - it is easy to create a table and just start writing to it. The pay per request pricing is simple and plays well with IAM, other AWS services and the serverless framework.
High-scale - scales easily up and down, global replication is easy, backups and high availability are easy - it is all self-managed so a lot of this is baked in or very easy to toggle on. Depending on your ops resources, the easy tooling may be worth the extra costs or maybe your data lends it self particularly well to how dynamodb works.
Some of our use cases ....
- user profiles to supplement cognito user attributes
- chat/chat groups/chat group memberships/chat websocket watchers/chat messages (this one was not good. I would do it again with just messages and the websocket watchers)
- tenant data with lots of custom attributes
- forms and form submissions
- event webhooks from 3rd party services - catch/store/process using dynamodb streams
- step function states and logs - it has some nice versioning write patterns
- web scraping data from ebay
- web scraping data from golf tee time sites
When you need search or flexible querying you can put elasticsearch in front of it and use dynamodb streams to push items into the index. If you need analytics you can use another layer like redshift.
This book is quite good. Used it to get a deeper understanding of Dynamodb.
Thanks for the laugh. Dynamodb is stupid expensive under any decent workload.
The free tier isn't limited to one year and low-usage is inexpensive.
https://github.com/dineshsonachalam/Lucid-Dynamodb/blob/mast...
On top of that they have mutable keyword arguments and generally quite terrible code: no exceptions thrown, just logging.error(), `if len(x)` rather than `if x`, etc etc.
https://github.com/pynamodb/PynamoDB
db.get('Pk.sk') or db.get('Pk.sk', ingore=['one'])
db.get('Pk.sk', model=pydantic_model)
db.update(... db.put(...
Ect
Leave a comment if people are interested in me releasing this project.