I thought about the mnist database but just opening the training set seems like a pain in the arse. Not complicated necessarily if you know how to read files in C but you need to be in a good mood.
No.... don't do that. You would suffer forever. Learn the basics of C separately, and then watch his videos. But if you don't give a F to C, go ahead. I meant, if you aren't planning on using C as your primary language of choice
@@Anonymous-fr2op I mean I already knew the C basics fairly well with function pointers, typedefs, enums, etc, I just didn't know any design principles for how to actually lay out a big C application. Also ended up learning Rust and it's pretty neat!
@@barni_7762yeah,tutorials like these, and some other really help us understand how to write code that is easy to maintain, in that case, you are right
One of the things that I implemented in my own library were some utility functions for allocating multidimensional arrays. What I ended up doing was allocating for two levels of pointers so it could be passed as a pointer to a pointer et cetera to the type, but I allocated everything in one step and then ran it through a loop to patch up the second, third, etc. levels of pointers. All the memory was contiguous, so m[x][y] would work or you could cast the first element's address to a singular pointer to the type. Copying was basically a linearly optimizable operation and you could reference singular elements with array indexing directly. When done it was a single call to free and as long as rows were the same length it could be reallocated with minimal effort as well. As for structured types, I use this method from time to time where I do: typedef struct { ...; } TFoo, *PFoo, Foo[1]; and then I can just have all the functions that operate on it take PFoo's and I declare in my functions Foo f; For the various object types in my data structures library I wound up implementing a basic virtual table for objects so I could just describe a given type in function pointers and a tiny bit of data, then generically handle them internally based on that type, _Generic for macros has made some really cool stuff possible with the built-in types too. I've never watched anything on Twitch, so I've not seen the previous streams, just what has been posted here, but I hope the next video has you saving the model so you can train it once and just get the results on subsequent iterations. I'm curious how fast it would be if you took a trained model and used it for basic math.
I'm not sure my comment will be read by the streamer, but here is my hint: you can add extra "1" to all layers input and your bias can be included right into the weights matrix. It will help to remove addition in favor of multiplication. Added at 1:57:20 : It seems your viewer tried to tell you the same idea.
I am a CS student and I am genuinely impressed by your coding skills. Can you please tell me how to become so good like you? Can you share how & where can we find resources , some good tips etc.
@@xhivo97 I have experience in building applications. I have also done some free lancing. Now I am contributing into open source projects like Hyperledger. I have this fear of DSA or should i say I don't like doing DSA. I am familiar with data structures but I get bored when I do questions on it but as you know most of the companies require DSA. How can I deal with this?
@@deveshmeena2573 Code more, all day every day. At some point you will start seeing super high level patterns and it will all feel the same, no matter what domain, language, framework, paradigm, etc.
@@deveshmeena2573 Try to find project for fun but also dedicate a few hours per week to study computer science and maybe a tiny little bit of math. Thing of DSA as a skill, if you master it you can more easily build fun stuf. Also sounds to me like you're on the right track, so just keep at it!
Just like the first part this ML course is really awesome. Definately the best beginners explanation out there (for programmers with no ML experience). Although he is clearly a highly intelligent and capable in the fields of math, programming and ML and teaching, his stream naming skills really suck, in my opinition, the videos really should at least have a part 1-3 suffix. This will be great reference material many years in the future 😄 And he should name his channel "The Ooga Booga Developer/ML Bro/Math Nerd" 😛
Saw this video once of a guy who stored matrices in both row major and column major at the same time, turns out it's faster to store double the data and have elements beside each other than not.
Wow, that seems new and maybe worth looking into it on different compilers. So the reason it's faster is because of L-level cache? Perhaps you have a link somewhere about this guy or his implementation?
I'm following this videos to reinforce my understanding about machine learning and C, and coding togheter this framework to fix this concepts and also learn and get more familiarized to create stuff like that, thanks for the educating recreational programming sections
Loving this tutorial! Particularly how you say uga buga... I have a comment. You did great work making mat_alloc size independent. However all of your other apis use float as input params. Could you make these api's size independent as well?
Damn, it would probably be easier to just flat out write a pytorch clone then... (which isn't actually that hard if you aren't trying to utilize gpu and optimize everything)
Enjoyed the video very much. Last hour you move very quickly through the code and unfortunately it's not always clear on screen? Any chance you would have time to upload to GitHub? Thanks.
This reminded me of Joseph Redmon and his DL framework Darknet which is base on C and Cuda. I was curious how he did all his amazing works from this basic level. I am also trying to learn to use C or C++ in future. Great content as always!
Yes, in the early days, we were a lot of engineers / researchers / hackers developping our own DL frameworks, usually in C or C++. It was very artisanal and very niche but that was doing the stuff. Definitely the most existing times of my programmer's journey. And then, Deep Learning started to attract the big corp and they streamlined the process with their massive manpower leading to the current state: python ecosystem hiding extremely complex codebases with ridiculous amount of abstractions. Anyway, it is as it is.
Your videos are very very educational. I can't thank you enough for it. I have an idea for your next project. the KNN algorithm framwork in c And maybe in wasm too. And make a visualizer that select the nearest neighbors on mouse click. The visualiser have to be scalable to 100 000+ points and do it fast.
@@0xCAFEF00D No!. If you relay on poeple on the internet to do your homework you are set to fail. I'm just experimenting with the knn for hobby stuff. And would love to see his approch on the subject. In case you still think it's homework please make it fast I'm waiting. And it's due next monday.🤣
He said this for serveral times in the video: it's for educational purposes. You have to implement the things by yourself to understand them. Lot of the things he is doing are pretty naive and inefficient. Especially the finite difference function. But that's ok if you just want to demonstrate the basic principles. Modern NN frameworks usually use the autodiff algorithm to calcalute the gradients for the back propagation.
well not relying on the OS to do it for you is a good practice anyway, doesn't your program leave heap alloced memory at exit? i mean, if you use stdlib's exit() that might not be the case, but otherwise are you really that sure that this heap alloced memory is freed anytime before the restart of your machine? if my concerns are not valid in any way, please elaborate, i'd love to hear what you think about it. (i could check with leaks/valgrind ofc but i'm too lazy to do that yet not that lazy to write this somehow)
> are you really that sure that this heap alloced memory is freed anytime before the restart of your machine? Yes. All modern Operating Systems in 2023 deallocate all the memory your program allocated at exit. On Linux your C program always calls the exit syscall even if you just return from the main function. That call is generated for you by the compiler. There is literally no other way for the program to stop executing its instructions in the memory except asking the kernel to step in (which is what the exit syscall is for). Without the exit syscall your program continues executing random stuff out of bounds and eventually runs into non-executable pages and gets killed by the kernel resulting in segfault. Which also results in all of the allocated memory being freed. Also, just buy more RAM.
@@TsodingDaily so then the main concern with memory leaks is that the program uses up more and more memory the longer the program keeps running... Am I understanding that correctly? So freeing it is still a best practice but not as important for this particular program because it doesn't run for super long. Is that right?
23:17 "If you don't really wanna have a fully connected one" ... and then twitch disconnects. AI sure has come a long a way, huh?
Loving this
Thank you
1:27:19 sometimes it takes femboy to become a good C programmer
lmaoo
i just made one myselsf, took me like 10 hours, idk how you manged to do it in 3 hours, you should try and do the mnist database now
it's totally easy it took me 40 minutes
20 to write the mathematics
20 to code it
@@SolathPrime man people must really like you
@@SolathPrime i dont have any prior expirince, and 40 hours was a big exsteration
@@SolathPrime you're so good, but who asked?
I thought about the mnist database but just opening the training set seems like a pain in the arse. Not complicated necessarily if you know how to read files in C but you need to be in a good mood.
The amount of your preparation for streams is just crazy. The narration is very smooth, thanks!
Can't wait for part 3 to be up on YT to watch it again to actually understand the second half after reading more about it.
As an applied mathematic major student who dream on pursing AI after, i would say this is probably the best thing in TH-cam.
Machine learning in C is very interesting!
@Auracle damn thank you brother 🔥
You know, in France, we call Twitter "la poubelle d'internet", literally "the garbage bin of internet"... So yeah, it's definitely Twitter
Am I the only one here who is trying to learn C by watching him code a practical project about a topic I already know?
then there's me, who has no interest in C and has coded neural networks before but is just watching this because it's entertaining
@@raphaelmorgan2307 Yeah thats also part of the reason I'm here xD
No.... don't do that. You would suffer forever. Learn the basics of C separately, and then watch his videos. But if you don't give a F to C, go ahead. I meant, if you aren't planning on using C as your primary language of choice
@@Anonymous-fr2op I mean I already knew the C basics fairly well with function pointers, typedefs, enums, etc, I just didn't know any design principles for how to actually lay out a big C application. Also ended up learning Rust and it's pretty neat!
@@barni_7762yeah,tutorials like these, and some other really help us understand how to write code that is easy to maintain, in that case, you are right
One of the things that I implemented in my own library were some utility functions for allocating multidimensional arrays. What I ended up doing was allocating for two levels of pointers so it could be passed as a pointer to a pointer et cetera to the type, but I allocated everything in one step and then ran it through a loop to patch up the second, third, etc. levels of pointers. All the memory was contiguous, so m[x][y] would work or you could cast the first element's address to a singular pointer to the type. Copying was basically a linearly optimizable operation and you could reference singular elements with array indexing directly. When done it was a single call to free and as long as rows were the same length it could be reallocated with minimal effort as well.
As for structured types, I use this method from time to time where I do: typedef struct { ...; } TFoo, *PFoo, Foo[1]; and then I can just have all the functions that operate on it take PFoo's and I declare in my functions Foo f; For the various object types in my data structures library I wound up implementing a basic virtual table for objects so I could just describe a given type in function pointers and a tiny bit of data, then generically handle them internally based on that type, _Generic for macros has made some really cool stuff possible with the built-in types too.
I've never watched anything on Twitch, so I've not seen the previous streams, just what has been posted here, but I hope the next video has you saving the model so you can train it once and just get the results on subsequent iterations. I'm curious how fast it would be if you took a trained model and used it for basic math.
Currently programming my masters degree project in Pytorch, and I can tell you keeping track of sizes continues to be an absolute pain.
Just wow educational and funny at the same time.
Thank you very much for this series
If you have udemy courses I would buy them all
I'm not sure my comment will be read by the streamer, but here is my hint: you can add extra "1" to all layers input and your bias can be included right into the weights matrix. It will help to remove addition in favor of multiplication. Added at 1:57:20 : It seems your viewer tried to tell you the same idea.
I am a CS student and I am genuinely impressed by your coding skills. Can you please tell me how to become so good like you? Can you share how & where can we find resources , some good tips etc.
The answer is always the same, do more projects. The more you code the better you get it just takes a LOT of time.
He codes.
@@xhivo97 I have experience in building applications. I have also done some free lancing. Now I am contributing into open source projects like Hyperledger. I have this fear of DSA or should i say I don't like doing DSA. I am familiar with data structures but I get bored when I do questions on it but as you know most of the companies require DSA. How can I deal with this?
@@deveshmeena2573 Code more, all day every day. At some point you will start seeing super high level patterns and it will all feel the same, no matter what domain, language, framework, paradigm, etc.
@@deveshmeena2573 Try to find project for fun but also dedicate a few hours per week to study computer science and maybe a tiny little bit of math. Thing of DSA as a skill, if you master it you can more easily build fun stuf. Also sounds to me like you're on the right track, so just keep at it!
ooga booga software developer
Just like the first part this ML course is really awesome. Definately the best beginners explanation out there (for programmers with no ML experience).
Although he is clearly a highly intelligent and capable in the fields of math, programming and ML and teaching, his stream naming skills really suck, in my opinition, the videos really should at least have a part 1-3 suffix. This will be great reference material many years in the future 😄 And he should name his channel "The Ooga Booga Developer/ML Bro/Math Nerd" 😛
Saw this video once of a guy who stored matrices in both row major and column major at the same time, turns out it's faster to store double the data and have elements beside each other than not.
Wow, that seems new and maybe worth looking into it on different compilers.
So the reason it's faster is because of L-level cache?
Perhaps you have a link somewhere about this guy or his implementation?
it is worth to follow and subscribe you.
Paint crashes when starting to describe matrices...I love this series.
I'm following this videos to reinforce my understanding about machine learning and C, and coding togheter this framework to fix this concepts and also learn and get more familiarized to create stuff like that, thanks for the educating recreational programming sections
insane coding this in C language!. will you destroy python as the programming language for Programming?
55:55 python itself is written in C 😂😂
Loving this tutorial! Particularly how you say uga buga... I have a comment. You did great work making mat_alloc size independent. However all of your other apis use float as input params. Could you make these api's size independent as well?
Why not use ssh -J for jumping directly to the virtual machine
That's a weird/unorthodox but correct matrix multiplication.
Just make the x metric a column vector. It is much easier
Thanks for the content, this channel is a goldmine
1:48:00 isn't the amount of cols just 1 always? I mean for each input the output is just a number.
46:48 - The same reason why I dislike Emscripten or complex code. Also, simple software directly means less space for bugs.
When he says "Essentially, essentially..." you know he bout to cook
XOR as example from 13:00 to 21:40:
sigmoid( [x1 x2] * [w1 w2] + [b1 b2] ) = [a1 a2] ⟶ sigmoid( [a1 a2] * [ w5 ] + [ b3 ] ) = [ a3 ]
( [w3 w4] ) ( [ w6 ] )
Where w5, w6 and b3 are the weights and bias for final AND neuron/gate.
The final calculation for it would be:
(NN_INPUT) a0 ⟶ sig[(x1 * w1) + (x2 * w3) + b1] ⟶ a1 ⤵
(NN_INPUT) a0 ⟶ sig[(x1 * w2) + (x2 * w4) + b2] ⟶ a2 ⟶ sig [(a1 * w5) + (a2 * w6) + b3] -> a3 (NN_OUTPUT)
th-cam.com/video/L1TbWe8bVOc/w-d-xo.html
2:18:39
what is the key combination you use to edit those 3 lines simultaneously ? thanks
you mad bro. 😂
You should make an open source decentralized streaming service in rust.
Love this series. Can you also cover Reinforcement Learning and Transformer neural networks?
Damn, it would probably be easier to just flat out write a pytorch clone then... (which isn't actually that hard if you aren't trying to utilize gpu and optimize everything)
18:50 I'm pretty sure you have switched 21 and 12 there ^^
I am finally able to memorize matrix multiplication properly thank you sm
Absolutely superb! Thank you very much!
I was once made to feel like an idiot in a coding interview for having a lightweight struct like that and not passing by pointer.
i couldn't keep my eyes away from tht bottom bar that folder that is 7.0 GiB that is too much man!
99% of ML is done in Fortran actually.
Makes sense.
is there a reason why you don't do your mat_alloc by pointer and return a reference instead of a value?
shouldn't be better to use tensorial notation when you're doing the linear algebra?
I'm here to add to the list of people that are here to inform you that it is indeed a twitter issue.
Enjoyed the video very much. Last hour you move very quickly through the code and unfortunately it's not always clear on screen? Any chance you would have time to upload to GitHub? Thanks.
a
Could you have put the matrix stuff in it's own file?
that differential approximating without symbolic diff was top notch.
This reminded me of Joseph Redmon and his DL framework Darknet which is base on C and Cuda. I was curious how he did all his amazing works from this basic level. I am also trying to learn to use C or C++ in future. Great content as always!
Yes, in the early days, we were a lot of engineers / researchers / hackers developping our own DL frameworks, usually in C or C++.
It was very artisanal and very niche but that was doing the stuff. Definitely the most existing times of my programmer's journey.
And then, Deep Learning started to attract the big corp and they streamlined the process with their massive manpower leading to the current state: python ecosystem hiding extremely complex codebases with ridiculous amount of abstractions. Anyway, it is as it is.
ayooo what folder does bro have in the bottom of the screen🔥
Can you do some cyber security videos thx dad
deadborn
header only & my machine only library
what linux distro is he using?
God series orz
right
TsodinFlow?
brilliant
very interesting
nice coode awesome program
you are amazing
Thank you very much, Tsoding! Next stop: "back propagation", and then transformers with self-attention, embedding, encoding, decoding and so son :)
Your videos are very very educational. I can't thank you enough for it.
I have an idea for your next project.
the KNN algorithm framwork in c
And maybe in wasm too.
And make a visualizer that select the nearest neighbors on mouse click.
The visualiser have to be scalable to 100 000+ points and do it fast.
And probably make hardware that can solve the nearest neighbor very fast using lazers or something like that. Just kiding😄
is this homework?
@@0xCAFEF00D No!. If you relay on poeple on the internet to do your homework you are set to fail.
I'm just experimenting with the knn for hobby stuff. And would love to see his approch on the subject.
In case you still think it's homework please make it fast I'm waiting. And it's due next monday.🤣
26:26 did you just say uwu?
I knew it
but is it viable to implement deep learning models in C ?
RIIR
Can you just tell me why? Are you just doing this for a fun project or is there any purpose for reinventing the wheel in C ?
This whole channel is coding for fun, and when coding is fun its easier to learn.
Recreational programming.
He said this for serveral times in the video: it's for educational purposes. You have to implement the things by yourself to understand them. Lot of the things he is doing are pretty naive and inefficient. Especially the finite difference function. But that's ok if you just want to demonstrate the basic principles. Modern NN frameworks usually use the autodiff algorithm to calcalute the gradients for the back propagation.
@@Pulsar2000 Thank you for the in depth response.
what compiler is this? my clang cant even any of these shet
I’m gonna try and do this in Haskell.
bro mat_free() please............. it hurts (i'm halfway through so i hope you fix the leaks eventually)
anyway thank you for the 10/10 content keep it up!
well not relying on the OS to do it for you is a good practice anyway, doesn't your program leave heap alloced memory at exit? i mean, if you use stdlib's exit() that might not be the case, but otherwise are you really that sure that this heap alloced memory is freed anytime before the restart of your machine? if my concerns are not valid in any way, please elaborate, i'd love to hear what you think about it. (i could check with leaks/valgrind ofc but i'm too lazy to do that yet not that lazy to write this somehow)
> are you really that sure that this heap alloced memory is freed anytime before the restart of your machine?
Yes.
All modern Operating Systems in 2023 deallocate all the memory your program allocated at exit.
On Linux your C program always calls the exit syscall even if you just return from the main function. That call is generated for you by the compiler. There is literally no other way for the program to stop executing its instructions in the memory except asking the kernel to step in (which is what the exit syscall is for). Without the exit syscall your program continues executing random stuff out of bounds and eventually runs into non-executable pages and gets killed by the kernel resulting in segfault. Which also results in all of the allocated memory being freed.
Also, just buy more RAM.
@@TsodingDaily so then the main concern with memory leaks is that the program uses up more and more memory the longer the program keeps running... Am I understanding that correctly? So freeing it is still a best practice but not as important for this particular program because it doesn't run for super long. Is that right?