How IEnumerable can kill your performance in C#
ฝัง
- เผยแพร่เมื่อ 29 ก.ย. 2024
- The first 100 of you can use code SCHOOL2022 for 20% off courses and bundles at dometrain.com
Become a Patreon and get source code access: / nickchapsas
Hello everybody I'm Nick and in this video I will show you how IEnumerable can harm your application's performance. I will explain why it happens, what you can do about it and how to deal with it in future scenarios.
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: bit.ly/ChapsasG...
Follow me on Twitter: bit.ly/ChapsasT...
Connect on LinkedIn: bit.ly/ChapsasL...
Keep coding merch: keepcoding.shop
#csharp #dotnet
This enumeration style is called co-routine for those who didn't know. You basically have a function on hold that can give you the next element right when you need it 😄
Actually this is a crazy efficient way to represent e.g. endless streams like indices from 1 to n, e.g. for n=int.MaxValue this is 2^31-1 * 4 byte. Your PC would simply explode if you'd call ToList() on it because it's 8GB of data. But a co-routine like Enumerable.Range() could do that with just 2 int variables and 8 byte.
It really makes a huge difference as you can keep this little chunk of 8 byte in faster cache levels of your CPU and crank on it like crazy. A ToList() too less or too much can make your program run 2 hours instead of 1ms 😅😅😅
I don't think that is called a "co-routine", I think it is called an "iterator". So it is like a class that implements an interface that has a next() method, so the foreach loop calls the next method every time it loops.
@@fred.flintstone4099 Historically speaking, when coroutines were invented, all machines were single-core. Real hardware multiprocessing wasn't even a thing until mid 2000s.
So, the resulting programs back then were fetching from something really similar to what we call an iterator nowadays. It's all about the illusion of concurrency by decoupling consumer from producer.
And honestly, most time in compute is still spent waiting. Waiting for operations to write results back, waiting for registers to be loaded from cache, waiting for jumps because of control flow, waiting for cache synchronization between processor cores, etc. Our code in C# is really just an illusion of what's actually happening.
But feel free to tell me where I'm wrong. Maybe I can learn something profound.
@@marcotroster8247 No, I think you're right. I think an iterator that waits might also be called a "generator". In C# you can use the "yield" keyword for iterators. There is also IAsyncEnumerable.
Wow! 👏
This man has some serious knowledge.
Great illustration of how/why this happens. Something I can send to my peers that get confused as to why their code is hitting an API twice when running around with IEnumerable or IQueryable.
I feel like you can explain this in 1 minute if they're already experienced programmers, but if they're new this would be helpful
I swear this is one of those things resharper has taught me with it's warnings, I rarely see it now because I know better. Great explanation of multiple enumerations.
Agreed - Before using Rider I had no idea lol
Me too
Visual studio also gives you the same warning.
@@TonoNamnum there must be a setting for that. We recently just upgraded to Rider from VS and have caught a few times we missed it. I have never seen the warning until we upgraded.
haha same, when I installed resharper I got aware of this situation.
You have no idea how much this helped me today! I was looking at a problem where counting an IEnumerable with zero elements in it resulted in a significant delay and I thought I was going crazy! I had no idea that IEnumerable would be lazily evaluated. Thanks for the help! :)
You could also adress how yield return's are dangerous in that the source list can be changed between enumerations, e.g. items removed between the .Count() and the output.
I do actually know of this, and typically do take this into account; I'd be lying if I said I catch myself [or others] every single time :). It's one of those things you miss if you're working fast
Great and very important point Nick
Strangely there's no OOB class or ext method to do this better. ToList would create a list that reallocates as it grows. What would be cool is a linked list of exponentially growing blocks - takes O(n) memory and time to build, but more efficient than just a linked list. And a common rule is: "iterate multiple times - use IReadOnlyCollection"
If you are creating a list internally, why not just return a list instead of IEnumerable?
What are the advantages of returning IEnumerable?
thanks man u r the best!
I don’t have resharper and don’t know about this until now…damn
I know about this feature and always take it into account. But I know too many developers who don't know about it. It's easier to list those who know
Most often I see the multiple enumeration warning on projects where people use Visual Studio. I think it's partly Microsoft's fault that they still don't warn people about the multiple enumeration possibility, so people don't care
Great video, but I think it is also important to mention that calling ToList() should be done only if underlying implementation is not enumerated. For example, in your example when your IEnumerable was returning List (instead of yield), a call to ToList() would copy the same list, which increases memory usage. And for new developers, they could start thinking after the video that ToList() should always be done if they are using a method that returns IEnumerable.
There are checks in ToList to prevent the extra allocation so you won’t increase the memory
@@nickchapsas but why then rider shows additional allocation of System.Int32[] and a new array in memory? And that additional +1 only happens AFTER ToList(). See the image here: snipboard.io/XkNj4A.jpg
And it even does the same if I use List as return type for GetNumbers. After ToList - a new array is allocated in memory.
And just for fun, I added four ToList calls one after another. dotMemory still sees the allocation snipboard.io/e2NrTl.jpg
@@nickchapsas incorrect. .ToList() will create a new list.
I really like your videos. This one was good too but the title was not really correct. Your presentation makes it look like the problem is IEnumerable. Its the correct way to return something when you don't want to limit the implementation of the method to a specific collection type. In fact, the problem is the Select() and knowing when to use Select().ToList() . In fact, you didn't even use ToList() instead you used a clunky list.Add() method. Also Count() is to be avoided in some cases. Length and Count are preferred (but hopefully Count() uses Length and Count behind the scenes, but it still requires a method call and an if statement). Another thing to realize is that by returning a LinQ QueryCollection type instead of IEnumerable you can avoid the double iteration? In that case, your Title would have been more appropriate. Whatever, I enjoyed the video anyway. Your presentations are always top notch in quality !
Can anybody telll me what Nick is using to show stuff like "x: "Nick Chapsas, 29"" and "splitline: string[2]" when he debugs? It seems to show the count as he debugs as well.
It’s just part of the Rider debugger
I can't say I have ever seen a yield return of an IEnumerable in 15 years of C#. Return of an IEnumerator in coroutines, yes, but not this. Must be a business software thing.
This is another example of Linq hiding bad coding. If you don't use Linq here, it's a lot more obvious that you have to iterate the enumerable to get a count, since you don't have a Count property/method.
Hello Nick, from everybody
Example too simple, can not apply to production code. I don't understand.
IEnumerable - The fast-food restaurant of programming
Materialization is not good option too. What if you file is 200GB size? Or what if there is pseudo infinite enumerable like network data or reading database cursor? So you can't always cast to list because of memory. So yes, watch you code and do what you understand. yield return is not bad if you know what you are doing.
I agree! Understanding your code is the mantra - at one point in time IEnumerable was THE way to do it in my company. For large payloads IEnumerable is great but applying it everywhere is an antipattern
Then treat it as a stream. On the scenario he gives, if the file is 200GB ReadAllLines will create a 200GB memory array of strings, so ReadAllLines is not the appropriate method to read such a large file, you have to stream it in.
@@battarro it would actually be well to 400GB, the file is likely ascii/utf-8, whereas the in memory representation is UCS-2, which is 16-bit (and similar to utf-16).
@@battarro ReadAllLines won't do streaming but _ReadLines_ will
I been programming with C# for about 15 years and there are parts about it that still mystify me. Your example of obtaining a count via an IEnumerable reminded me of how I learned on my own a similar situation with your example. In my case I was loading over 100k records. EF was new to me and I couldn't understand why my app was taking a performance hit until I discovered the difference between IEnumerable and IQueryable. From then on it forced me to take into consideration the overall purpose of the program and how to use IEnumerable properly. You are very well versed in the programming language, more than me after working with C# for so long.
On a side note, back when I was learning programming in 1991 I asked a senior developer of our mainframe why people are sloppy with their code. He told me that it will only get worse because as computers get faster it will compensate for bad coding practices and the end result will be lazy programmers. I came from learning to program on a mainframe environment where every byte counted. We ran accounts payable and payroll for 300 employees. All of it was done on a 72 megabyte hard drive.
I started with assembly code. U and I are in the same boat. 😅
When I started to think about it more deeply, this system is actually very very good:
If we have some enumerable thing A given for a consumer B, how could B assume that it has enough memory to hold all elements of A?
Ans: It can not, and thus it protects itself with this solution of multiple enumerations: If file read in this video was gigantic (let's assume milions of lines) then multiple enumeration IS desired!
The solution is just to use IReadOnlyList, which has enough space saved prior to the enumeration.
A common recurring problem among programmers is not knowing how the code they're using works. At the very least, they should understand what the API commits to doing. Deferred enumeration of IEnumerable is a great feature in C#, but if you're using any API that exposes an IEnumerable object, you should always assume that you need to enumerate at some point, unless your objective is to merely chain subsequent operations to perform on the object.
In fact, if you never actually enumerate the sequence, it will never actually execute, and this is also an easy trap to fall into. So my best advice would be to write your API according to what client code should expect. If you're returning a finite object, rather than returning IEnumerable, you should return IReadOnlyCollection or IReadOnlyList (or any read-only interface). That way, client code knows that enumeration has already been performed. If you return IEnumerable, client code should assume that enumeration will be required, and even the implementation should probably avoid enumerating to a terminal operation.
JetBrains.Annotations also has the [NoEnumeration] attribute that you can assign to an IEnumerable method parameter to indicate that your method isn't performing a terminal operation over the parameter.
What do you mean by 'you should always assume that you need to enumerate at some point, unless your objective is to merely chain subsequent operations to perform on the object' ? I don't understand the part ' you need to enumerate at some point'. Could you explain it for me?
@@youcantsee3867 it means that if you're dealing with an API that provides an IEnumerable object to you, you should assume that no actual query operation has yet to happen behind the scenes. It is only when you enumerate that the query is actually executed.
For example, if the API operation is fronting a database call, no query is run against the database until you first enumerate over the results.
This is different from working with a list or an array, which has already been materialized with contents. The enumerable object has no contents until you enumerate.
@@marcusmajarra Thanks for your reply. I have one more question, so the word 'enumerate' is means calling some method like 'count()' 'toList()' or calling for each. Am I right?
@@youcantsee3867 essentially. If you're not digging into the results, you're not enumerating.
@@youcantsee3867 A good example would be for instance in C# you call a function that returns an IEnumerable - IEnumerable list = GetList(); - you would think that doing a foreach (var item in list) twice would use the same results from GetList(); but the IEnumerable interface doesn't actually grab any data until it is 'enumerated' in a foreach. That means each foreach of list will execute all the actions GetList() did to build the IEnumerable result. You could something like
IEnumerable query = customers.Where(c => c.Active);
List result1 = query.ToList();
List result2 = query.ToList();
It's possible that result1 differs from result2. if customers data is backed by a database, each call of ToList (or any foreach statements) on the IEnumerable will build the query, execute it, and return a new result.
My personal choice is to return an I…Collection, so that the consumer knows that the “inner” code isn’t deferred. Of course, there are situations where an IEnumerable is better, for instance when implementing repositories. But such repositories are mostly consumed from other application specific services.
It's fine if result is supposed to be mutable because ICollection exposes things like Add and Remove. Otherwise you'd want to return something read-only: IReadOnlyCollection or ImmutableList or similar.
I seem to remember the LINQ documentation explicitly stating that Enumerables are lazy-evaluated. It is a feature, one that all developers should be cognizant of so that they can force one-time evaluation when appropriate.
A bigger problem with methods that return IEnumerable is when they take parameters like a Stream or any IDisposable.
Why does no one have this problem with `Iterator`/`IntoIterator` in Rust? 🤔
Because nobody uses Rust so there aren't any people who would complain to begin with
The beauty of IEnumerables is lazy/deferred execution.
A trap (per this video's message) if you don't have a grasp of what it is.
Lazy/deferred execution I believe was borrowed from the Functional paradigm.
The idea is that you have a set of logic/algorithm which wont be executed/evaluated
unless with explicit intention.
In C# LINQ, you express the 'intention' by calling operators like
.First()
.ToList()
.Count()
.Any() etc.
Examples of lazy LINQ operators,
.Where()
.Select()
.OrderBy() etc.
These return an IEnumerable of .
Lazy/deferred execution shines when composing/chaining functions and
when you intend to use your functions in between a "pipeline". Hence the above 3 are often used in a query chain/pipe.
Pertaining to collections, lazy evaluation passes only 1 item to each node/operator in the chain/pipe at a time.
But for eager evaluation, the whole collection is evaluated and passed down.
If there were conditions of 'early breaks', the latter won't benefit as the collection has been prematurely evaluated.
E.g. a lazy pipe/chain
products
.Where(p=> p.InStock()) // each product 'in stock', will flow down..
.Where(p=> p.Price < 3.14) // but only 1 at a time and not the full list because 'where' is lazy.
.Select(p=> p.ToShippable()) // Concatenated lazy chains act and behave as one (select is also lazy).
// I often combine multiple individual lazy operators to solve complex problems with very little concern for performance penalty.
// Shifting the order of the operators around is also quite easy as they are somewhat stand alone..
BTW, there's a CSV competition article that covers all the CSV parsers for .NET and since the last time I looked the Sylvan.Data.Csv library was the winner and shockingly fast even when using types.
Oh, that's neat.
I'm still using TextFieldParser from the VisualBasic namespace, since it's the best one (and the only one) built into .NET itself.
The biggest problem I have with Linq in general rather then IEnumerables is the heap allocation that takes place when evaluating queries with ToList() and the like in memory-sensitive hot paths. In almost every other scenario it's absolutely fine, but it makes my life hell when I have to do rate calculations on 100-500 messages/s.
Would be good to see you cover MemoryPool and ArrayPool at some point, those types have truly saved my bacon!
I have a video about object pooling coming, probably around October or November. It's a really interesting topic
@@nickchapsas Perfect, I look forward to it! =)
@@nickchapsas Sweet!
so basically yield returning an IEnumerable is great when you work with transient data that doesn't get reused? just to know if i understood right
it really is quite clean, there is even a warning (at least in visual studio) CA1851: Possible multiple enumerations of IEnumerable collection
it just requires the developer to read the warning and handle it. (or the senior developers elevate this from warning to a compilation error)
Whoa, for the past one year I was getting sometimes warnings "Possible multiple enumerations" and never knew what does it mean :V Thank you!
I learned this the hard way playing with Entity Framework when it first came out years and years ago. It was not a fun or quick learning experience! Unlike this video, which was both fun and quick 👍
The most surprising fact is an amount of people who doesn't know that. If you ever used "yield" keyword it kinda obvious that IEnumerable MyMethod() returns T one by one and don't store the whole thing in memory hence the second call to this method will calculate Ts one by one again. I can see a confusion though if you always get IEnumberable from a third party and never used "yield" keyword yourself then yes, it's not obvious.
Hey nick, shouldn't we just use List as return type for collections every time? and IEnumerable only in cases where we are sure that we need it ? or there are disadvantages of using List everywhere? would love to hear your thoughts. Thanks :)
Only problem would be if you don't want the user to change the items in the list (use IReadOnlyList or IReadOnlyCollection then) or for interfaces or abstract implementations, though in that case I think you can still return the List, even if the interface says IEnumerable.
Basically, the only place I would ever have an IEnumerable return or out parameter is in an interface that might need it to be that way.
Of course, if you *are* enumerating something and it doesn't make sense to return a list, do return an IEnumerable. E.g. an "infinite" list.
Example: infinite fibonacci sequence
IEnumerable Fibonacci()
{
long prev = 0;
long curr = 1;
while(true)
{
yield return curr;
var p = prev;
prev = curr,
curr = p + curr;
}
}
Of course it's not "infinite". Since it grows with a factor of 1.618, with long it'll take less than 100 steps.
Consider BigInteger for a more painful experience 😁. Or enumerating every integer.
@@phizc Thank you for sharing your opinion. So, I think I'm saying right that use IEnumerable only in particular cases while Lists actually have more use cases when considering real application case scenarios. I asked this question cuz I've seen a lot of the people always returning IEnumerable and then doing .ToList(); to use it. maybe they do it to maintain some level of abstraction ..?!!
I think it's up to the person using the code. Like Nick mentioned you might want to add in a Where clause or other instructions before enumerating. Deferred execution is a good feature of C# if you know how to use it. You can also avoid iterating the whole collection in case you use methods like First, Any or similar methods. In any case where you only need one iteration a IEnumerable works fine, if you need multiple iterations you can use ToList or ToArray first so for me an IEnumerable is best as it's flexible
@@jackoberto01 Yeah i think it depends on the case...but again i don't think you'll be able to use IEnumerable more than List as List cases are more in my experience.
@@Moosa_Says I deal with files that are large enough that they just don't fit in memory often enough that it's not worth writing different code just for that case. But it doesn't matter if the file fits in memory, if I don't need to access the data by index and can process it sequentially, I have no reason to load it all at once.
When you switch to .Select(), the file is only read once while each line is selected twice; and then you could just append .ToList() to the .Select() to return a list of Customer's without splitting each line twice.
Oh boy, this one hit hard on me
Some time ago I've had to import a 15 billion line csv into database and IEnumerable gave me the impression that it would help me but it actually didn't 'cause of the multiple interations.
The only solution that occurred to me in that time was to slowly add some values to a list and them import, otherwise it would throw a memory exception.
Without this approach, the entire proccess would take almost 3 hours to complete. After some modifications and making it more "listy" it ended up being just some minutes.
IReadOnlyCollection
I heavily use co-routines on my applications, specially for large operations, I can read 28k lines from the database, make them as objects, turn them into JSON and send them over to clients without loading 28k things in memory, then making 28k objects, then making a 28k items json array, saves a lot of RAM and avoids high-gen GC.
Where is Chap Nicksas? 😞
Had to fire him but he re-applied for we might see him again
@@nickchapsas glad u hired Saps Nickchap👌
me putting .ToList() everywhere to prevent enumerables from behaving unpredictably: there. now I should be safe.
Garbage collector: **whining in agony**
I love that JetBrains catches possible multiple enumerations
BUT OH MY GOD If you don't enumerate the same IReadOnlyList multiple times it will NAG You endlessly to change to parameter to IEnumerable
Which is annoying because your parameter already conforms to IReadOnlyList then it will again nag you to change it back when you enumerate one more time
yay - I was able to find the issue before Nick pointed it out. =) Though, I only looked for it because Nick pointed out there was a problem though...probably would not have cought it in real life.
IEnumerable can kill your performance in C#, but if you want to be 100% sure, combine IEnumerables with databases and Lazy loading features. Devastating.
Ive know this for awhile and I always thought it was a problem with .Count(). I have creatively coded around .Count() in many projects.
That IEnumerable kinda looks like a JavaScript Generator Function. ;-)
I've switched to using IReadOnlyCollection or IReadOnlyList in most cases. The only time I use IEnumerable is when I don't need/want all items to be in memory at the same time, or if there could be a reason to only enumerate some of the items. If I had a CSV with 1,000,000 customer names and I wanted to know how many Nick's there are, I could read the file line by line, check if the name is Nick, increment a count, and move to the next line without storing all the names. Or if I wanted to get the address of the first 5 Nick's in the file, I could enumerate till I find the 5 Nick's, and then stope enumerating.
8:45: "Know that the warning might be there but there might not be multiple enumeration in every single one of those occasions."
Enumeration is the process of _going over the elements of the enumerable,_ not of creating it. When you have a List, LINQ's Count() can call the Count property directly and not have to enumerate the list to count it. But if you had multiple foreach loops or LINQ queries, that would indeed be multiple enumeration even if it's of a List or an array.
As the video says, the warning is confusing. The problem isn't multiple enumeration; it's multiple _generation,_ which can be a problem for more reasons than just the performance hit. If the generation of the enumerable is non-deterministic for whatever reason, (maybe you have a call to a random number generator in there, or you're querying a database twice and someone else INSERTs something into it in between your two calls,) you can end up enumerating two different sequences of values when you intuitively thought you'd be enumerating the same values twice, which can cause bugs in your code.
I will not stress about loading the file everytime. I will not stress about loading the file everytime. I am stressing about loading the file everytime. Oh god it hurts.
"Hello Worldish" topic, and every jumior knows it (at least every juior I've worked with)
Thank you for tip, quite interesting.
One question.. Does your courses come with certificate at conclusion?
To be fair, this is why I hate using Var. It can hide these issues. Where specifying type makes sure you know from the outset that it has still yet to be resolved.
I questioned the system when I used a Where clause on a list and then when I modified a value on that list it suddenly got added/removed based on the where clause. When I never updated the IEnumerable collection to reflect the changes. After trial and error I debugged it and write a before and after count and my value change somehow magically reflected in the collection, I got an inkling of how this enumeration worked after watching this video I've now discovered that Count and iterating through it, reexecutes the Where clause everytime.
Thanks for this video. I don't use IEnumerable. After that video i'll still so :) but i learn why i'm not.
Won't VS and Rider give you a warning if you're enumerating multiple trips times?
Oh LINQ and IEnumerables, such a blessing and a curse. Very easy to abuse and misuse.
does ObservableCollection have the same behavior since it implements IEnumerable in its base source ?
looks like a problem for the I don't know what I am doing codingstyle.
Its called Lazy initialization ? Can I keep this feature, when I create my own class, which is inherited from IEnumerable?
If you do it in the GetEnumerator method, yes.
The IEnumerable interface only has that method IIRC, and it gets called by foreach, and the LINQ methods (eventually).
@@phizc what exactly i must to do in GetEnumerator method for capture this feature?
one interviewer asked me to compare IQueryable and IEnumerable :)
I usually IReadonlyCollection or IReadOnlyList, unless really need lazy evaluation
Where is Sas Chapnick customer? He was always around before((
Taken out by the CIA for learning Obama's surname
It is very bad practice to save IEnumerable at variable. If you need to save IEnumerable need to save it as array. By my opinion array is most basically type instead of list.
Yes, I knew about this problem because Rider told me :)
The yield in the the example does not have the same behavior as returnn list.Select()
If you have 1M rows in your file, is better the first approach.
Awesome video as always!
It’s worth noting however that the implementation of GetCustomers using Select behaves subtly differently to yield return. The call to GetCustomers itself is deferred until enumeration when using yield return whereas it’s called only once when using the Select, when it is assigned to the customers variable. Still absolutely a performance issue as the iteration over the lines still happens twice but the file is only loaded into memory once in the Select example.
it never come to me that method enumerable behaves same way, but indeed it does. Nice stuff ty!
As an old programmer I must say I don't like those modern ideas... gross
I havent watch vid yet Nick "Epic" Chapsas is a Legend
7 years c# programmer, just knew about it
After watching your videos, it feels like all the programming I have been doing is just wrong..
This is why I hate programming. Everything you do is bad. If you want to make anything, you have to spend 10 years coding overhead because it's more optimal than being productive.
It doesn't necessarily take longer to write good code. It takes knowledge. And yes it can take a long time to learn that knowledge. (I'm not sure what you mean by "overhead".) There are other jobs that require a lot less learning.
Interesting. In the languages I work with, doing something like this simply fails because the iterable can only be iterated over once, so you'll find out quickly that you need to cache the results to use them twice.
@Adam M java also
This happens in C# for `IEnumerator`. However `IEnumerable` is really just a factory of `IEnumerator`s
A key phrase, which is maybe more of a LINQ term, that I was also hoping to hear was "deferred execution". Great topic to be privy to, and great video, Nick!
I might LINQ my ting from Barking! 🤣🤣
I really want a Nick Chapsas for Rust!
Great explanation of multiple enumerations
"var" is hidding valuable type info from dev. IMHO it is the worst feature added to c#.
Not really, most of the time you know based on the text to the right - e.g. var x = new Customer() -- x is a Customer type. So it's easier to read when you put var. However when you don't know what might be coming back, or it's not obvious, then it's recommended to not use var.
Hi Nick, love your channel, could you explain why the following is 20x slower on my pc? I have seen it mentioned here or there but never seen a good explanation
using System.Diagnostics;
var list = Enumerable.Range(0, 100_000_000).ToArray();
IEnumerable enumerable = list;
Stopwatch sw = Stopwatch.StartNew();
foreach (var i in enumerable)
{
var d = i + 1;
}
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
sw.Restart();
foreach (var i in list)
{
var d = i + 1;
}
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
The IEnumerable is much slower than enumerating an array, if I change the order around so the array is enumerated first and the IEnumerable second, the result is the same (array is much faster still)
The foreach ienumerable is implemented using the iterator interface (movenext, current methodes)
The foreach array is implemented like a for loop, no method calls involved.
@@ryan-heath Thanks, that makes sense
@@ryan-heath except the compiler, or at least the JITc would know that it is a list, just presented as an IEnumerable.
For the JIT to optimize it that way it might have to upgrade to Tier 1 compilation though, and it'll only do that if it's explicitly told to, or encountered the method 30+ times.
why is it that he can write a method without any enclosing class?
I rolled my own "CachedEnumerable" which lazy-caches the results of an enumeration - it's a wrapper over IEnumerable which tests the underlying enumerable (e.g. is it IList) and skips the cache for enumerables that are already cached/array-based. Using it gives me the best of both worlds - lazy enumeration and automatic caching.
By introducing caching layer you lose actual source data, so if between your enumerations file/db is changed, you have to implement your own cache updating to give the latest set of items to consumers
@@nanvlad Yep. So I don't use it there.
Basically it's very simple: LINQ is a pipeline (and chained yield returning function calls are as well). It's a series of enumerators chained together like a single expression, and will not be running until you run a loop on it, to perform actual work.
A function like Count() is a terminating operation, because it does not return an IEnumerable by itself but a computed, numeric result, meaning it has to run a loop on the the preceding expression.
And a self written foreach loop is basically another terminating operation.
ToList and ToArray are as well, they create a new collection in memory and run a loop to fill it with data.
This means that ToList and ToArray come with the disadvantage of extra memory allocations.
While not using them and repeating loops on the expression come with a time and CPU usage penalty, like basically shown in the video.
Count() is actually smart, if the underlying type is an ICollection, it will return ICollection.Count instead of evaluating the IEnumerable.
Great video! I always feel good about myself when I know exactly what the problem is and what your solution is going to be at the start of the video... It doesn't happen often, but when it does, I give myself a gold star :-) Thanks for posting such good content!
I was just giving my developers a lesson on this exact topic last week. I wish I could have just pointed them at this video! Thanks so much!
TLDR: Most C# programmers use APIs and functionality they don't understand.
TLDR: Most -C#- programmers use APIs and functionality they don't understand. 🤣
How do you gain all this knowledge regarding perfomance?
I read a lot of the .NET codebase and try things out
why would you ever need to return ienumerable anyway
Resharper taught me, now i understand it
today lesson. thanks :)
Hi Nick,
Just finish your minimal api course. What the most optimize way to connect and work with CosmoDb? Is Dapper going to work with CosmoDB? A lot of sample codes are using DbContext and Microsoft document just show how to work directly call the database and container. Is that the best way to work with it? Maybe you've done a video on it, I just can't find it. Appreciate you sharing with us.
I would simply use the SDK of cosmos db directly. It’s pretty good
@@nickchapsas thank you, brother. I will give that a go. Appreciate you help.
Great stuff Nick. Your impact on my programming had been tremendous!
In my opinion, all this is obvious.
This video made me subscribe 😁
I like your mustache
my first work day as developer / programmer today
🎉
Congratulations! You gonna smash it 🎉🎉
I've worked on IEnumerable/IEnumerator classes the last couple of days for getting the path to hard links (NTFS) of a specific file. It works, but dang is it annoying. I tried to only make an IEnumerator, but that's not good enough for "foreach". It really wants to call that shiny GetEnumerator method on IEnumerable.
After watching the video, I've decided to just get all the links at once and return an array. The NT kernel methods are set up as enumerating (GetFirst/GetNext), but there's only ever going to be less than 1024 links (hard coded in Windows), and realistically, less than 10.
Also, for my purposes, and probably everyone else's, it doesn't make sense to just get the first few, or not turn it to a List/array anyway.
There's even a winapi method to get the count, before trying to enumerate them.
good to know, did not know before