Speeding up Queries by Pre-computation & Transformation

Many applications are more read-intensive than write-intensive. Meaning your application performs more reads than writes. However, we often persist data that is optimized for writes making querying and composing data intensive at runtime. What's a solution? I will review different ways of creating a materialized views and some trade-offs.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via TH-cam
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
1:59 Materialized Views
4:23 CQRS
8:05 Event Streams
Eventual Consistency is a UX Nightmare
• Eventual Consistency i...
#softwarearchitecture #softwaredesign #cqrs

[@allinvanguard]

It's kind of funny that pre-computing data can feel like a somewhat obscure / unknown technique in software development, while our buddies over in business intelligence and big data have basically been doing this for several decades

[@CodeOpinion]

Yup! Pretty much, only difference is tooling and ways of distributing data has changed about from being ETL to more real-time.

[@nove1398]

This is a topic I have been thinking on for the past week or so for optimization strategies

[@tharun8164]

Just a simple insight:
Cache: Use it if there are recurring requests so as to improve read latency. We store a subset of data that's required to cater recurring requests.
Index: Even this is used to improve read latency but indexing will happen on the entire dataset unlike the subset.
It's not mutually exclusive and even the in-memory db like redis can be used for both use cases.

[@jwbonnett]

You used my conversion! 😃

[@ilijanl]

The flow you give in 5:50 can be very tricky since it requires to process events in order. A better alternative is probably to use the broker to notify the projection logic that it has some new data and let the projection catch up (reading new events from the write db). This operation can be throttled/denounced to have less load on the write db

[@CodeOpinion]

It depends what's in the events. This is where people start getting into event-carried state transfer and fat events that primarily are about state/data rather than events than the result of some behavior of the system that executed. Not a fan in most situations, but it's usually a progression people end up going down.

[@aaaarvil]

I do this with my high-traffic event-sourced system that streams to a reporting database. The biggest issue I had was ensuring idempotency (because of at least once delivery) and handling events that arrive that may potentially be out of order. The good thing, though, is that reporting databases are usually SQL anyway, so I can just add the event id on an event-processed table (per table, or per row, depending on the use case) and put that in a transaction along with my update. It wasn't easy when we started, but the load on our read database has been really manageable thanks to this approach.

[@CodeOpinion]

Nice. Thanks for the comment/insights.

[@nighma]

Hey! Thanks for the video! You talked about a link in the description about eventual consistency and how to deal with it at 9:46 but I can't find the link. Ty

[@CodeOpinion]

Here you go: th-cam.com/video/wEUTMuRSZT0/w-d-xo.html

[@swarupmahapatra1]

Event sourcing is a great technique where events are source of truth and projections can be rebuilt by replaying events.
However very few frameworks support this.
Axon in Java ecosystem.
Is event driven architecture a more logical approach if event replay isn't available?
In JavaScript world there is no framework that treats events as first class citizen and supports replay.

[@marcom.]

I had to use Axon in a project at my previous employer. I hated it. From my point of view it was a wrong decision to rely on this framework. It would go too far here to list all the reasons why, but it was one of the reasons why I changed jobs. I would recommend everyone to look into using Apache Kafka instead.

[@pdgiddie]

I think frameworks can be overrated. Event sourcing, at its core, is a relatively simple design pattern. Frameworks will usually offer a very specific approach, and it may actually introduce a lot of unnecessary complexity.

[@tinytownsoftware3837]

What's the difference between materialized views and indexed views?

[@CodeOpinion]

There can be differences depending on the database and how they term it. SQL server as an example calls them indexed views. Oracle has materialized views. MySQL has no such concept natively, you could build it out with triggers etc.

[@kneeh]

Does the price of dealing with concurrency worth it ?

[@CodeOpinion]

In what sense? Concurrency in the sense of have multiple consumers (competing consumers) process events concurrently to update a materialized view? If so that's just one situation if you're using hte competing consumers pattern.

[@kneeh]

@@CodeOpinion In context of the example in the video, say that the same user can alter his Cart from multiple devices simultaneously, how can I ensure valid up to date data on the materialized view (Total cart cost column for example) ?
Or am I understanding materialized view wrong and its only for data that isn't going to change anymore ?

[@clashclan4739]

Lets say i have 10 different api with different query composition from one main table. Do i need to matiralize view for all 10 scenarios on write time for read optimization. Then if new requirements come changes in existing. Its going to lot complex

[@CodeOpinion]

"It Depends". You absolutely could create a more generic set of data that's still more read optimized rather than creating a use-case specific materialized view.