Not quite. It's a shared distributed log, but with the processing (the database engine) embedded in each service. So you could think of it as a database per service, with all these databases sharing the same shared file system. It's a little different to this in practice as the services you build end up being more about reacting to events, but it's a closer analogy. So if I build a service that sends emails to customers after they make a payment, I would subscribe to the stream of payments, I would then create a table of Customer info inside my service using the KStreams API (both projections of the streams in Kafka), then I would join them together, pull out the fields I need, again all inside my service and send the email. So the coupling between services is very low (just the event distribution) but it's easy for any service to pull out the data it needs and do useful work on it.
How to address agreement for a common data structure? How about addressing bugs that caused incorrect data to persist for months in the log. How's the performance?
Do you actually build real world computer systems? Your overall points don’t seem very innovative but generally ok but your asides including the functional garbage are less than useful.
I though it was kind of neat seeing a speaker act natural for once. It seems like in every presentation every speaker tries to follow the same guidelines of how they are "supposed to" present without much personality or feeling of their own and in my opinion its both boring and unnecessary.
Yes I guess it is. I actually spent five years working on this problem in a large company without Kafka. We essentially rolled our own, but life would have been a lot easier had we had it.
Is this basically a data mesh?
Nice talk. From problem to current ways of solving it to a new different approach.
Thanks
Isn’t this sharing a database?
Not quite. It's a shared distributed log, but with the processing (the database engine) embedded in each service. So you could think of it as a database per service, with all these databases sharing the same shared file system. It's a little different to this in practice as the services you build end up being more about reacting to events, but it's a closer analogy. So if I build a service that sends emails to customers after they make a payment, I would subscribe to the stream of payments, I would then create a table of Customer info inside my service using the KStreams API (both projections of the streams in Kafka), then I would join them together, pull out the fields I need, again all inside my service and send the email. So the coupling between services is very low (just the event distribution) but it's easy for any service to pull out the data it needs and do useful work on it.
He addresses this concern at the 11:00 mark.
Yes, it actually is.
How to address agreement for a common data structure? How about addressing bugs that caused incorrect data to persist for months in the log. How's the performance?
Do you actually build real world computer systems? Your overall points don’t seem very innovative but generally ok but your asides including the functional garbage are less than useful.
He has built plenty. Check his LinkedIn.
@@mcscubin8935 Not sure you want to follow this recipe, though.
One piece of advice: stop moving around so much!
Yes - noted!
I though it was kind of neat seeing a speaker act natural for once. It seems like in every presentation every speaker tries to follow the same guidelines of how they are "supposed to" present without much personality or feeling of their own and in my opinion its both boring and unnecessary.
@@stopfob It was a good presentation. Move how you like.
Really interesting - thanks. Ben is there a sample application that showcase the principles you talked about?
Yes! But give me a couple of weeks :)
I am very interested in it. Talks are good but code and integration examples helps a lot. Thanks.
In the end, the conclusion, the whole system is heavily resembling a Datomic like data store
I think in fairness this is maybe half true. To me Datomic is more of a database. A streaming platform puts more emphasis on data in flight.
Sounds like a promotion for Kafka
Yes I guess it is. I actually spent five years working on this problem in a large company without Kafka. We essentially rolled our own, but life would have been a lot easier had we had it.
Ben Stopford why didn’t you use Kafka?
It didn't exist back then :)