Another great video Derek. There are a lot of devs out there not building new things with shiny latest tech, but really dealing with improvements to legacy systems 😊 So videos like this are really helpful.
Great advice! To get management buy-in, focus on other benefits as well. While important to reduce failure rate, it's also great to get better opportunities to e.g. consolidate logic, improve performance, add a cache, add monitoring, modernize tech stack, move from vertical to horizontal scaling (containers?) or whatever you might be eyeing at the same time..
I probably can't post a link in comments, but your video titled "Change Data Capture + Event Driven Architecture" at ~2 : 52 is a good one to pair with this! As always Derek, thank you for the great approachable content. Hope one day I can meet you at a conference or similar.
Through the years I've become convinced that it is not an app that owns the database, it is the opposite. The database owns the app (or multiple apps). You take care of the database and evolve the apps accordingly
As i here your wise advice i have failed promise of OOP before my eyes, especially missed shoot at encapsulation level - betting on objects more than modules
Another great video, thank you. I work in a warehouse execution system and there is no ownership of data. Inventory, putaway orders, and pick orders for example are modified in multiple services with no concern of ownership or concurrency issues. One service could be cancelling an order while another is picking it and there is no type of concurrency safety in place, I contribute that to what you mention in this video since the writing and reading of data is spread across every service. This makes everything from implementation to testing difficult because you have no idea who owns or changes what data.
Maybe implement a status flag? When its waiting for picking, its "Ready" status (allow cancel) when someone "pick" an order, set it to "Picking" (no one can pick again and cancel will need extra internal workflow), when picked order is out for delivery, its "In Transit" (not allowed to cancel).... etc.
Timely advice. I currently have.a system where data is duplicated in mongo, maria db, sybase and SAP where the same tables are updated via multiple apps. I am trying to convince management the idea of ownership. Then there is the reporting where they hit a mariadb directly via a third party reporting system (tables with 200+ columns) and I keep telling them to create an API because maria db is failling them and we need to migrate since scaling is an issue. I think they are sick of me in meetings because I keep asking which app should own the data . I think they understand but quite frankly there are so many apps spread out and what I am suggesting will take time just with existing functionality it is virtually impossible to get buy in.
I'm currently working on a big spaghetti codebase where I'm trying to migrate databases. It does a lot of things including generate reports for that data. And the way I've approached it with this mindset is I'm moving out reporting to a new service. It has basically forced us to implement a way to know when data changes to be able to update the data in the new reporting database. This also means that when I start moving out other parts of the system, I will be able to still have a way to hook into these events and have the data I need to perform whatever it is I need to do. For years this project has struggled because it is too big and being supposedly impossible to refactor without a big bang rewrite but I finally see a light at the end of the tunnel thanks to this pattern. Great advice.
Reporting/Query purposes is often times what is the biggest hurdle. I wanted to mention it more but yes, creating a specialized DB or even creating a schema that resembles the current schema but purely for read purposes is also a strategy, and the segregating the write schemas behind their proper boundaries. At least as a stop gap to keep existing queries working while you transform everything.
Stealing "6:50 Free for all in the database" :) Finishing up a 200 table ORM jungle with tons of transactions cutting across multi billion row tables with millions of users. Usually I see the database schema is in star or snowflake 3NF when all the user data changes heavily in time together and should be event sourced. I've found looking for a common "fact" table that has a sequential primary key, joining off of that to get to all the "dimension tables", then persisting + reacting to that works great. As a bonus you get exactly once delivery instead of using a persistent message broker.
I work on a 20+ years old shitty laravel codebase without any test and with gigantic, 2k+ lines, files 😅 I doubt we can fix it, even more so with only 3-4 devs on the team 🤣🤣🤣
Ya it's a challenge. You may never get it to where you want, but small incremental changes to isolate functionality and the data behind it can be beneficial.
Another great video Derek. There are a lot of devs out there not building new things with shiny latest tech, but really dealing with improvements to legacy systems 😊 So videos like this are really helpful.
Great advice! To get management buy-in, focus on other benefits as well. While important to reduce failure rate, it's also great to get better opportunities to e.g. consolidate logic, improve performance, add a cache, add monitoring, modernize tech stack, move from vertical to horizontal scaling (containers?) or whatever you might be eyeing at the same time..
That's a good comment and topic about management buy-in.
I probably can't post a link in comments, but your video titled "Change Data Capture + Event Driven Architecture" at ~2 : 52 is a good one to pair with this!
As always Derek, thank you for the great approachable content. Hope one day I can meet you at a conference or similar.
Through the years I've become convinced that it is not an app that owns the database, it is the opposite. The database owns the app (or multiple apps). You take care of the database and evolve the apps accordingly
As i here your wise advice i have failed promise of OOP before my eyes, especially missed shoot at encapsulation level - betting on objects more than modules
Another great video, thank you. I work in a warehouse execution system and there is no ownership of data. Inventory, putaway orders, and pick orders for example are modified in multiple services with no concern of ownership or concurrency issues. One service could be cancelling an order while another is picking it and there is no type of concurrency safety in place, I contribute that to what you mention in this video since the writing and reading of data is spread across every service. This makes everything from implementation to testing difficult because you have no idea who owns or changes what data.
Maybe implement a status flag?
When its waiting for picking, its "Ready" status (allow cancel)
when someone "pick" an order, set it to "Picking" (no one can pick again and cancel will need extra internal workflow),
when picked order is out for delivery, its "In Transit" (not allowed to cancel).... etc.
Thanks for this very actionable advice!
Everything comes down to managing coupling and encapsulation.
Bingo
Timely advice. I currently have.a system where data is duplicated in mongo, maria db, sybase and SAP where the same tables are updated via multiple apps. I am trying to convince management the idea of ownership. Then there is the reporting where they hit a mariadb directly via a third party reporting system (tables with 200+ columns) and I keep telling them to create an API because maria db is failling them and we need to migrate since scaling is an issue. I think they are sick of me in meetings because I keep asking which app should own the data . I think they understand but quite frankly there are so many apps spread out and what I am suggesting will take time just with existing functionality it is virtually impossible to get buy in.
Thanks for the comment. As another commentor mentioned as well, approval/buy-in really is a roadblock beyond technical.
I'm currently working on a big spaghetti codebase where I'm trying to migrate databases. It does a lot of things including generate reports for that data. And the way I've approached it with this mindset is I'm moving out reporting to a new service. It has basically forced us to implement a way to know when data changes to be able to update the data in the new reporting database. This also means that when I start moving out other parts of the system, I will be able to still have a way to hook into these events and have the data I need to perform whatever it is I need to do. For years this project has struggled because it is too big and being supposedly impossible to refactor without a big bang rewrite but I finally see a light at the end of the tunnel thanks to this pattern. Great advice.
Reporting/Query purposes is often times what is the biggest hurdle. I wanted to mention it more but yes, creating a specialized DB or even creating a schema that resembles the current schema but purely for read purposes is also a strategy, and the segregating the write schemas behind their proper boundaries. At least as a stop gap to keep existing queries working while you transform everything.
Stealing "6:50 Free for all in the database" :) Finishing up a 200 table ORM jungle with tons of transactions cutting across multi billion row tables with millions of users.
Usually I see the database schema is in star or snowflake 3NF when all the user data changes heavily in time together and should be event sourced.
I've found looking for a common "fact" table that has a sequential primary key, joining off of that to get to all the "dimension tables", then persisting + reacting to that works great. As a bonus you get exactly once delivery instead of using a persistent message broker.
"200 table ORM Jungle" 😂... I don't think that is that uncommon, hence the video.
Shoutout Detroit
👍👍
I work on a 20+ years old shitty laravel codebase without any test and with gigantic, 2k+ lines, files 😅
I doubt we can fix it, even more so with only 3-4 devs on the team 🤣🤣🤣
Ya it's a challenge. You may never get it to where you want, but small incremental changes to isolate functionality and the data behind it can be beneficial.
Similar here (not laravel) and change requests keep coming weekly XD.
JavaScript
Javascript?