How we use DynamoDB streams to replicate data into Elasticsearch
ฝัง
- เผยแพร่เมื่อ 5 ก.ย. 2024
- Sharing a high level design for how dynaomdb streams work and how we used it to duplicate data over to elasticsearch
------------
🤑 Patreon / webdevjunkie
🔔 Newsletter eepurl.com/hnderP
💬 Discord / discord
📁. GitHub github.com/cod...
![[Re:all TREASURE] EP.6 in 방콕ㅣ🚩 쩡캠의 방콕 투어 빵 마이 와이 👍](http://i.ytimg.com/vi/uyFw-nItA_s/mqdefault.jpg)
Dude i feel ya! 😅 DynamoDB is nice but its hell to serve data to other applications. Especially if you have a lot of data which needs to be paginated on complex queries its simply not possible.
We currently query all data then do further processing in lambda and put the results in memory. But your video gave me an idea how to restructure. Thx!
excellent. brief and clear.
Crisp and Clean Explanation
It would be expensive process.
I need to know how I to improve dynamodb modeling and mapping process, it took me so much time to come up with few access patterns.
Why not use a SQL database? I'm not judging or anything I'm just trying to learn so I can make some decisions in my job.
i wish we did. I'm not a fan of dynamo unless you have "super large data" needs.
Nice design, but perhaps we could buffer the stream using an SQS (with a DLQ) so the Lambda doesn't choke and so we could recover from errors in the lambda without losing data. WDYT?
I think the dynamodb streams (kinesis) is built to hold a LOT of messages, so I don't think it would ever choke. Also the lambda processing the streams from the dynamodb stream already have a fixed limit on how many message they can read per second, so I don't think they will ever choke per say.
@@WebDevCody Lambda could fail cause of other unexpected reason. Don't you think adding a SQS as buffer (with a DLQ) with ensure data processing of each event streamed from dynamoDB. Just curious can we ensure 100% processing of streams without involving Buffer SQS but including DLQ
I think a SQS (with DLQ) should be added for example just in case the communication between Lambda and ElasticSearch faces some unexpected issue, and the data record to be inserted is lost. Having the SQS, the message involved in the issue will go back to the SQS and event will be launched again. Am I ok?
I dont know much about elastic serach but i wonder why do you need dynamodb, cant you just use elasticsearch as your database?
bonsai.io/blog/why-elasticsearch-should-not-be-your-primary-data-store
Good job babe!!!
Great, thanks!
Is there a service that can provide similar functionality as dynamodb streams for planetscale mysql?
What if I update, or delete a record in dynamodb? will it also reflect in Elasticsearch?
Yes if you handle the update or delete event in the stream, you can also remove it from elastic search.
Nice🤓
Where we get that dynagrams images
Which one? You’d need to take a screenshot
@@WebDevCody I find it, no worries thanks