@@Matlus do you think there could be an issue in case the caller is relying on each streamed entity having a different reference? (which of course could be avoided with implementing IEquatable on the entity). I guess it would be up to the implementer to make sure to properly implement IEquatable on the returning entity?
@@hansrudolf5849 The optimization works only if the caller also is streaming. If any caller needs to accumulate the sequence (i.e. they need a collection/list/array rather than a sequence), then this optimization won't work as I demonstrated in the video.
Very important feature. I do agree with you that this feature is release since a long time but still developers are not aware of it. You made it very simple to understand. I wish more and more C# developer watch your video and get benefited. I get disappointed when I see people run behind jargon like cloud, micro-services etc. and do not focus on such basic stuff. Your efforts are commendable.
Thank you Vipul! Thank you also for taking the time to write a comment. Yes, sadly, we're infected with buzz words and as you mentioned =, most don't know what these techniques/technologies truly mean.
Superb example of using iterator pattern with real world example, as you said even though it's exist from 3.5. I have seen people using this way, even I my self never thought to use the way you showed. It would be really nice if you make this kind of videos too, some other feature videos or may be c# 9.0 videos. Thank you every video of your channel is pure nugget.
So if you don't have direct access to the database and need to stream over an API, how do you get around the http object accumulating all of the records before returning them as one giant list to the client in .Net 6? I've tried using IEnumerable & IAsyncEnumerable along with await foreach, and it always waits for all records at some point in the pipeline before returning. The database may return results one at a time, and the client may render them one at a time, but a result isn't being rendered in the UI as it's being returned from the database. The http object is feeding them to the client one at a time from it's own collection it accumulated from the database.
Thank you Robert! The latest video (just posted) on Real-World gRPC, I show a better example of streaming. Source code for the project is also available on my GitHub.
People giving thumbs down should have to explain why before it gets counted. There are only positive vibes and great knowledge that transpires from this video. 14 thumbs up.
Some fraction of views always generate dislikes due to youtube mobile UI design, some fraction of the people will hit that dislike when they start scrolling down for the comments. So if the ratio is overwhelmingly positive then chances are that the few dislikes are simply misclicks.
Does yield return hit database every time when streaming data one by one like this ? Or is it taking it out from memory every time? Sorry for the dumb question.
I'm afraid, there is no code available to download for this video. The intention was to just have folks understand the concepts rather than get caught up in code.
@@Matlus I think that is the right approach and you did GREAT with the concept. I actually only wanted to check out your data access code because I was taught to do it in dapper but your way looks like it uses vanilla code and looks like it works better than dapper. Do you have a video about data access to SQL databases using vanilla code?
@@rossthemusicandguitarteacher Aah, I see I can point you to another project that has a lot of data access using ADO.NET directly. No streaming as shown here, though. But the correct way to access data from a database using ADO.NET github.com/matlus/MovieServiceTH-cam
@@Matlus Perfect! Thanks so much. Also, do you do any 1 on 1 consulting or mentoring? I have been teaching myself software engineering for about a year and half now, and now I am in charge of systems for the company I work for. However, I have nobody to give me feedback on my work, so I would be curious if you provide that as a service. :)
@@rossthemusicandguitarteacher I appreciate you asking about consultation work. However since I do have a full time job and I have my hands full with other commitments as well I can't commit to anything else at this time. If you do have questions that I can translate into topics for videos I'd be happy to do that but I understand that, that may not work for you since you need solutions right away to any questions you may have. I am very curious to know what kinds of questions you do have though. So if you don't mind, please jot them down.
Thank you so much Sir for all these "under the hood" videos that you make. Pretty unique ones! One question regarding the yield return of the Movie instance example. When does exactly the calling of DB take place the second time ? Each and every time, it opens a connection, gets the data and then close the connection ? And then it keeps doing it 2000 times ? Isn't this too much overhead to go to the DB 2000 time instead of batch some of these calls ? Or is it a trade off and a decision we need to make between memory consumption for the objects and DB calling overhead ?
Thank you Yiannis! Regarding your question related to the Database. No, the connection is opened only once. The data is streamed from the database to your app if you use the DbDataReader. The TCP/IP connection is open and there is only one call being made to the database. But the data takes time to return to the calling application. The data comes is packets (think if it as chunks) , so why the process the data as it arrives, rather than wait and accumulate the data and only then start to process it? That's essentially what streaming it all about. I hope that helps. You certainly need to be aware that you're "streaming" from end to end of the application (as it's not going to happen by fluke). But there is no trade off really. The calling application will see the data as soon as you receive it rather than a dead time when you're accumulating the data rather than streaming it.
While this feature works beautifully it’s only used in scenarios where you are going to simply retrieve the data and return it to the caller. If you have to write any logic around some or all the returning records then it’s not really helpful. Hope that make sense or is there a way?
Could you not pass that additional step into a separate thread/async function from the original calling function, say inside the foreach in the main method in this example. Not sure if that would interfere with the iterator but worth a thought.
@@maximeumbra7235 How about returning a IEnumerable response wrapped in an object which provides additional properties such as pagination, sorting, status, messages etc etc. How will that work with async stream? My understanding for async stream to work you must be returning a pure collection data and not a wrapped response object.
@@evilroxxx Apologies didn't notice this when you replied, i've been on vacation and haven't played around with these much so i can't really say, still learning how to use these proficiently from memory so i'd rather not make any fast claims :)
@@maximeumbra7235 I understand. No apologies necessary. Everyone deserves vacation and to disconnect from technology and the world once in a while. I’d love to see the OP post a follow up video to show how to do this when a response wrapper is returned as a response to the api post action call for example.
this is really cool, i've never seen this pattern before...is this going to make 2000 database calls instead of 1 though? Edit: nevermind, I continued watching and it sounds like since the database is streaming db records back to the app as they're ready, and the app is streaming through the results in real-time, it's still only a single database call. this is awesome!
Single instance streaming: I think you should derive a type from the Movie class with some sort of pre/postfix that explains what to expect, because this is very much not expected (don't surprise). Not sure what phrase to use but something like `SingleInstance` or `Shared` or something... Thanks for the tip.
I agree. Surprises are bad. I've only done this sort of thing where I needed to squeeze out more performance than I was getting. We got almost 40% improvement for this method and an overall performance improvement of 23-25%.
What a beautiful example to explain the concept of yield return. Thank you sir for taking your time to make this video and serve the community.
Thank you Afzal! Glad you liked it!
Keep up the good work Shiv - that's some high quality knowledge which you are providing!
Thank you Hans! Appreciate the comment.
@@Matlus do you think there could be an issue in case the caller is relying on each streamed entity having a different reference? (which of course could be avoided with implementing IEquatable on the entity). I guess it would be up to the implementer to make sure to properly implement IEquatable on the returning entity?
@@hansrudolf5849 The optimization works only if the caller also is streaming. If any caller needs to accumulate the sequence (i.e. they need a collection/list/array rather than a sequence), then this optimization won't work as I demonstrated in the video.
Very important feature. I do agree with you that this feature is release since a long time but still developers are not aware of it. You made it very simple to understand. I wish more and more C# developer watch your video and get benefited. I get disappointed when I see people run behind jargon like cloud, micro-services etc. and do not focus on such basic stuff. Your efforts are commendable.
Thank you Vipul! Thank you also for taking the time to write a comment. Yes, sadly, we're infected with buzz words and as you mentioned =, most don't know what these techniques/technologies truly mean.
Superb example of using iterator pattern with real world example, as you said even though it's exist from 3.5. I have seen people using this way, even I my self never thought to use the way you showed. It would be really nice if you make this kind of videos too, some other feature videos or may be c# 9.0 videos. Thank you every video of your channel is pure nugget.
Thank you Saurabh! I'm glad you're enjoying these videos and learning something from them.
i am very thankfull for this video 😄am learning from you a lot actually so thank you for your efforts.
Excellent example of streaming and its advantages. Really useful for processing large collections. Thanks
Thank you M1ke22! Appreciate you taking the time to post a comment here.
I just discovered your channel. Greatly appreciate your insight.
So if you don't have direct access to the database and need to stream over an API, how do you get around the http object accumulating all of the records before returning them as one giant list to the client in .Net 6? I've tried using IEnumerable & IAsyncEnumerable along with await foreach, and it always waits for all records at some point in the pipeline before returning. The database may return results one at a time, and the client may render them one at a time, but a result isn't being rendered in the UI as it's being returned from the database. The http object is feeding them to the client one at a time from it's own collection it accumulated from the database.
The channel I was looking for 🔥
Thanx for the content you provide you helped me a lot 👌👌
Very candid presentation Shivkumarji
Thank you Manas!
Sir Perfect unique deep diving Explanation 👌
Awesome, made perfect sense all way through
Thank you Robert! The latest video (just posted) on Real-World gRPC, I show a better example of streaming. Source code for the project is also available on my GitHub.
People giving thumbs down should have to explain why before it gets counted. There are only positive vibes and great knowledge that transpires from this video. 14 thumbs up.
Thank you Jean! Yes, I'd love to know what/why when folks give a thumbs down so I know what to improve etc.
Some fraction of views always generate dislikes due to youtube mobile UI design, some fraction of the people will hit that dislike when they start scrolling down for the comments. So if the ratio is overwhelmingly positive then chances are that the few dislikes are simply misclicks.
I just wonder what camera you used to capture. Video was so high quality.
Good job with this. Thanks.
Awesome video. Keep it up, thanks for bringing this knowledge to us.
My pleasure Abhishek!
Awesome explanation. Thanks a lot.
Glad you liked it Nandkishor
Does yield return hit database every time when streaming data one by one like this ? Or is it taking it out from memory every time? Sorry for the dumb question.
Thanks for this!
Do you have the github link, kind sir? :) Thanks so much for this video.
I'm afraid, there is no code available to download for this video. The intention was to just have folks understand the concepts rather than get caught up in code.
@@Matlus I think that is the right approach and you did GREAT with the concept. I actually only wanted to check out your data access code because I was taught to do it in dapper but your way looks like it uses vanilla code and looks like it works better than dapper. Do you have a video about data access to SQL databases using vanilla code?
@@rossthemusicandguitarteacher Aah, I see I can point you to another project that has a lot of data access using ADO.NET directly. No streaming as shown here, though. But the correct way to access data from a database using ADO.NET
github.com/matlus/MovieServiceTH-cam
@@Matlus Perfect! Thanks so much. Also, do you do any 1 on 1 consulting or mentoring? I have been teaching myself software engineering for about a year and half now, and now I am in charge of systems for the company I work for. However, I have nobody to give me feedback on my work, so I would be curious if you provide that as a service. :)
@@rossthemusicandguitarteacher I appreciate you asking about consultation work. However since I do have a full time job and I have my hands full with other commitments as well I can't commit to anything else at this time. If you do have questions that I can translate into topics for videos I'd be happy to do that but I understand that, that may not work for you since you need solutions right away to any questions you may have.
I am very curious to know what kinds of questions you do have though. So if you don't mind, please jot them down.
Wow great example 👏🏿👏🏿
Thank you so much Sir for all these "under the hood" videos that you make. Pretty unique ones!
One question regarding the yield return of the Movie instance example. When does exactly the calling of DB take place the second time ?
Each and every time, it opens a connection, gets the data and then close the connection ? And then it keeps doing it 2000 times ?
Isn't this too much overhead to go to the DB 2000 time instead of batch some of these calls ?
Or is it a trade off and a decision we need to make between memory consumption for the objects and DB calling overhead ?
Thank you Yiannis! Regarding your question related to the Database. No, the connection is opened only once. The data is streamed from the database to your app if you use the DbDataReader. The TCP/IP connection is open and there is only one call being made to the database. But the data takes time to return to the calling application. The data comes is packets (think if it as chunks) , so why the process the data as it arrives, rather than wait and accumulate the data and only then start to process it?
That's essentially what streaming it all about. I hope that helps.
You certainly need to be aware that you're "streaming" from end to end of the application (as it's not going to happen by fluke). But there is no trade off really. The calling application will see the data as soon as you receive it rather than a dead time when you're accumulating the data rather than streaming it.
@@Matlus Thank you Sir for the prompt reply and detailed explanation. It makes totally sense now!
While this feature works beautifully it’s only used in scenarios where you are going to simply retrieve the data and return it to the caller. If you have to write any logic around some or all the returning records then it’s not really helpful. Hope that make sense or is there a way?
Could you not pass that additional step into a separate thread/async function from the original calling function, say inside the foreach in the main method in this example. Not sure if that would interfere with the iterator but worth a thought.
@@maximeumbra7235 How about returning a IEnumerable response wrapped in an object which provides additional properties such as pagination, sorting, status, messages etc etc. How will that work with async stream? My understanding for async stream to work you must be returning a pure collection data and not a wrapped response object.
@@evilroxxx Apologies didn't notice this when you replied, i've been on vacation and haven't played around with these much so i can't really say, still learning how to use these proficiently from memory so i'd rather not make any fast claims :)
@@maximeumbra7235 I understand. No apologies necessary. Everyone deserves vacation and to disconnect from technology and the world once in a while.
I’d love to see the OP post a follow up video to show how to do this when a response wrapper is returned as a response to the api post action call for example.
this is really cool, i've never seen this pattern before...is this going to make 2000 database calls instead of 1 though?
Edit: nevermind, I continued watching and it sounds like since the database is streaming db records back to the app as they're ready, and the app is streaming through the results in real-time, it's still only a single database call. this is awesome!
Single instance streaming: I think you should derive a type from the Movie class with some sort of pre/postfix that explains what to expect, because this is very much not expected (don't surprise). Not sure what phrase to use but something like `SingleInstance` or `Shared` or something... Thanks for the tip.
I agree. Surprises are bad. I've only done this sort of thing where I needed to squeeze out more performance than I was getting. We got almost 40% improvement for this method and an overall performance improvement of 23-25%.
WOW!
Thank you Ross!
looks i found briliant in TH-cam Trash . Thanks
Thank you for your kind words Mike