Scale To Millions Of Users With AWS DynamoDB
ฝัง
- เผยแพร่เมื่อ 27 ก.ค. 2024
- We cover the unique features that DynamoDB offers, what sorts of use cases it is a good fit for, and what use cases it is not a good fit for. We also cover how to store and retrieve data with DynamoDB, as well as DynamoDB schema design.
00:00 DynamoDB Features
01:07 Creating Items
03:01 Retrieving Items
05:27 Local Secondary Indexes
06:29 Global Secondary Indexes
09:23 Schema Design
11:30 Table Creation & Scaling
---
Stuff I use to make these videos - I absolutely love all of these products. Using these links is an easy way to support the channel, thank you so much if you do so!!!
Camera: Canon EOS R5 amzn.to/3CCrxzl
Monitor: Dell U4914DW 49in amzn.to/3MJV1jx
Lens: Sigma 24mm f/1.4 DG HSM Art for Canon EF amzn.to/3hZ10mz
SSD for Video Editing: VectoTech Rapid 8TB amzn.to/3hXz9TM
Microphone: Rode NT1-A amzn.to/3vWM4gL
Microphone Interface: Focusrite Clarett+ 2Pre amzn.to/3J5dy7S
Tripod: JOBY GorillaPod 5K amzn.to/3JaPxMA
Keyboard: Redragon Mechanical Gaming Keyboard amzn.to/3I1A7ZD
Mouse: Razer DeathAdder amzn.to/3J9fYCf
Computer: 2021 Macbook Pro amzn.to/3J7FXtW
Caffeine: High Brew Cold Brew Coffee amzn.to/3hXyx0q
More Caffeine: Monster Energy Juice, Pipeline Punch amzn.to/3Czmfox - วิทยาศาสตร์และเทคโนโลยี
Great tutorial. Efficiently detailed and right to the point. thank you!
Very good video! Thanks.
Never used DynamoDB before, I should give it a try.
Indeed at 12:00 it makes me scream ;)
Thanks, glad you liked it!
Re: 12:00, I know what you mean. I remember having a visceral reaction when I was told this. I didn't go into too much detail about it in this video, if you're interested in more this is a fantastic video th-cam.com/video/HaEPXoXVf2k/w-d-xo.html
@@codetothemoon thanks I’ll check this right now.
I’d like to try a Lambda with Rust and DynamoDB. Would be fun I suppose.
what i found out after running any scale with databases is mostly in case of app's success you still need to run database with close to same restrictions as dynamodb. Query anything what's not on index perfectly or using something like COUNT or any aggregation really is only possible on lower data volume. Yet i use sql for 1st iteration of saas because of flexibility it provides to adapt quickly, which is more essential at early stage. Also on DynamoDB time series analytics is very possible, actually building that atm. Good part about time series is that data is often not changing. All meta which does change is usually only needed for the visual and that can be fetched separately and not involved in data aggregation pipe.
Re: close to the same restrictions as ddb - totally agree with this, I think ddb imposes some constraints that initially seem harsh because other databases tend to abstract away so much that you're not aware of many inefficiencies, while ddb's constraints make those inefficiencies painfully obvious.
Re: time series data, yeah definitely possible, would love to hear more about your use case and access patterns - ie are you needing to view the data from several different angles? Are you doing aggregations in your application code?
Is there any ways to query a map type data and to search for items by querying through the subtext of the map datatype.
currently trying to get away from dynamo. Used it for all my databases.
Main drawbacks:
- vendor lock
- lack of features (mongodb has more features and I feel like no drawbacks compared to dynamo)
- super bad javascript api
You get used to the gsi flow and really adjust your thinking about data structure. However creating a gsi for a small new feature instead of just living with a bit slower query (like you would in mongo, where you can always create new indices to speed up querys) is super bad.
your experience with DDB is a great data point to have!
re: vendor lock - yeah this is definitely an issue.
re: features - what missing features would be valuable to you?
re: api, I agree. It seems overly verbose.
re: mongo indices - been awhile since i've used mongo, what advantages do its indexes have over GSIs? Was under the impression that they work similarly. Maybe with the difference that it doesn't break the primary key into partition/sort keys.
@@codetothemoon the difference is that I do not directly need to create an index in mongo. I can just query some edge cases and have a generally more flexible approach. Maybe some queries happen rarely and not having an index is fine. In dynamo I solved them by creating a new index and introducing another key (date for searching) that acts as a sort key on the index so not all the data is replicated again because all unused items do not have that key (e.g comments on blog posts could have a commentdate). Still results in a lot of redundant data and infrastructure for something that should be trivial. (Also I ended up with lots of indexes for some tables, which probably means I should not have used dynamo in the first place)
With dynamo you are forced into this very unflexible design pattern, while mongo allows the same pattern with the same speed advantages but doesnt enforce it.
Does mongo have something like Dynamodb streams?
Super Nice :)
thanks, glad you liked it!
@@codetothemoon I am starting learn towards something like planelscale, seems the most flexible solution. If speed is necessary stick redis instance up or using something like upstash.
@@codetothemoon loved it, great summary.
When would you use MongoDB instead of DynamoDB?
That's a fantastic question, one that I don't have a great answer to yet. One use case that does stand out is if you need to do aggregations inside the DB to reduce the amount of data returned to the consumer - I believe MongoDB supports this but DDB definitely does not.
@@codetothemoon issue on aggregation is if there is more volume of data, aggregation becomes super expensive on resources. on dynamo you usually have stream to lambda or something which pre-calculates aggregated form of your data to quickly query it. This is scalable to any level. You may export data to s3 if needed and aggregate it on some ec2 or any compute unit, not lambda. There are honestly enough ways to resolve aggregation "issue". from lambda triggering on live updates to have close-to-live aggregation, to unlimited amount of instances running and processing whatever amount of data.
How do I contact you man!?
ken@cttm.io 😎
@@codetothemoon thanks
I would question your point around video analytics ... Disney+ gave a good talk at re:invent 2020 where they covered their DDB usage: th-cam.com/video/TCnmtSY2dFM/w-d-xo.html
My favourite thing around using DDB is the thought patterns you're required to follow in order to design a table - it really forces you to understand what problems you're trying to solve and focus your work
Thanks for the link, I'll give it a watch! I'm curious how they address situations where they need to perform aggregations or query on fields that aren't a partition key.
I agree with what you say about DDB forcing you to design your tables a certain way - I feel like some databases allow you to follow antipatterns (like querying on a field that isn't indexed) and not even realize that you're doing something extremely suboptimal.
Dynamodb is disgusting to work with, very bad to scale and not affordable if your table will grow. Overall total waste of time. Same goes for provider api's, didn't really like it.
I think this could be a great case study. Scaling is supposed to be one of DDB's huge advantages - what issues did you run into there? Was your table unaffordable due to data size or request volume? Finally, what did you wind up using instead and did it resolve those issues?