Your tutorials are a goldmine. I took an Operating Systems class that literally had 0 lectures to explain the concept of multithread --- instead just a word vomit of text --- and this video was worth 10x of those classes. Keep up the great work!
wow. jesus. I've sat through hours of class about this subject and never understood a single sentence, but you just explained 5-6 lectures in 25 minutes like fucking Picasso, I've never felt so confident about understanding Producer Consumer multithreading. Thank you!
These are the best Unix C - Threads/Processes video tutorials ever! As a teacher, I recommend and share your videos with all my students. I recommend you to make video series about sockets and network programming with C. Greetings from Bulgaria!
Thank you so much! I'm glad other teachers appreciate my work. And thanks for the recommendation, it is the most requested so far, I'll look into it as soon as possible
in addition to the excellent explanations of multithreading concepts and examples of how they can be useful in application, i really appreciate how you demonstrate the process of problem solving in every episode, starting from programming the most basic model of the program and debugging it as you add more functionality - it really helps to have that good practice demonstrated for beginners like me. also appreciate how often you look into the camera, it really does make the lesson feel more engaging. thank you so much to you and your team, I'll be referencing your content and supporting your channel for years!
These videos have saved me a lot. My university lectures are so convoluted and over the top it takes me hours to understand what they are trying to explain, yet these videos explain the exact same material in a very clear and concise way. Great job and thank you for these videos on C.
Dude you are a champion, such complex concepts you explained so easily. Your videos are so good, keep posting and continue throwing light in to so many peoples careers
this video is absolutely fantastic. I really like your detailed style. explaining everything. I hope that you're doing well, and that you plan to continue in the future.
I understood both the mutexes and semaphores and I now know how to do my homework. Thank you very much. Very clear explanation. Now I am eager to learn more.
Thank you so much! I have to do a kind of program about producer and consumer for the university, and this video helped me a lot to understand the problem!
This video really makes it clear why they are called semaphors. It was a fun video to watch, the 25 minutes of the video flied by so fast. Your videos and Jacob Sorber's videos explain semaphors nicely.
If you have a producer-consumer-problem and use mutexes, you'd use condition variables along with them. That's much more efficient under load conditions where the producers and consumers are constantly processing.
sem_wait is using a spin lock (busy waiting) on Linux. But the scheduler might change it to a lock if the spin-lock is not unlocked in a reasonable time-frame
Felicitari din nou pentru continut. Sunt developer de 4 ani si am simtit nevoia sa imi reamintesc anumite concepte. Pot spune ca in cateva ore sunt la fel de "fresh" ca in facultate.
i hate seeing someone like you, who is such an amazing explainer, getting views like that , for this videos therre is only 1_000 views !!! also, i hope you doing some machine learning and ANN stuff on youtube in the futur too good luck
Hi CodeVault, thank you for your great explanation. I want to check if my understanding is correct. First, sem_wait(), sem_post() have to be atomic (not be interrupted). Because of that, their check need to be very trivial. I.e, wait() and post() should only check if( empty/full >0 ). This concept is called guarded commands. Now, because these checks have to be simple, we need two semaphore variables for producer and consumer. I.e, for producer, check if empty > 0 and for consumer, check if full > 0. That is the reason why we need two semaphore variables instead of just "int count"; Am I right?
Thanks a lot! On the source code from the description, the if statement inside the loop is different from the one on the video. if (i > 0) instead of if (i % 2 == 0 )
@@CodeVault oh ok that makes sense. But if we want to split the threads between producer and consumer evenly we should keep the mod right? Thanks for all the content by the way, great stuff!
Thank you for clear explanation. As I understand you use both mutex and semaphores because the usage of two different semaphores does not prevent producer and consumer from executing critical section simultaneously. So you need mutex as well. Is my conclusion correct?
Yes. Without a mutex, you could produce multiple items at the same time, have the critical section executed by 2 consumers at the same time which creates a race condition on the array it's reading from
So can we say mutex's main purpose for producer/consumer environment is to lock the critical section and semaphore's main function is to provide synchronisation between the threads?
Yes. Basically semaphores are just there so that you can notify threads to wake up when something happened and mutexes for what you mentioned: locking the critical section
Thanks for your perfect and beautiful code But why for us didn't print done when the buffer was consumed? In my case i produce n items and when all of them was consumed it shows done So i bounded while for n items Is it true?
So can we remove the mutex lock once we use the semaphore or do still we need it? As there might be issues when both producer and consumer are working on buff simultaneously
You still need them as you might have multiple consumers taking from the array at the same time. The idea is that semaphores are synchronization objects for the whole "consumer" and "producer" process while the mutexes are only there to prevent race conditions on the array itself.
No. Because you cannot make a thread wait on an arbitrary value of a semaphore (you need to wait for both producer and consumer threads in different situations)
Why is it a problem? Just use a ring buffer with a read pointer and a write pointer. If the read ptr is the same as the write ptr, the buffer is empty. The write pointer advances the write ptr after the data is stored at that memory location. You will probably need a fence there, to ensure that the pointer is incremented after the data store. The read pointer never increments past the write ptr.
It's a common problem taught by many professors across the globe. You can always argue about its usefulness in the real world but that's not my goal here with those videos. Students are taught poorly about this topic and I'm trying to provide better material for them to learn from I might make a video implementing the improvements you listed there
@@CodeVault oh, cool beans. I had a microcontroller with a cortex-m4 paired up with a m0+ coprocessor. I think it was an NXP LPC 43xx or something. They had a piece of hardware that managed a communication channel between the two cores that worked like I describe. It's kinda like a plan9 channel. Sadly... I don't know if it could be extended beyond two communicating threads...
Great video , is there way thr consumer can add to the buffer with minimum waiting while the slow consumer is busy consuming. Thanks. Thst Is When there still slots on the buffer. Thanks.
I'm not sure what you mean by "minimum waiting" but simply executing what the producer is doing inside the consumer's function should do the trick. Just be careful to not create an infinite loop since then your consumer would also be a producer
@@CodeVault Thanks for the insight. My producer can very rapidly insert new packets to a fixed size circular buffer/array[4], but my consumer is very slow. Producer should only wait if already full of new packets, and consumer can wait if no packets. But producer shouldn't be blocked while trying to insert a new packet when not full. I was worried the slow consumer taking the mutex could prevent the producer to take the mutex to insert a packet to non-full circular buffer. The slowness of the consumer shouldn't affect the producer from adding new packets while not-full. Appreciate any suggestions, thanks, and merry Christmas.
The mutexes on both sides are important as to not cause race conditions. I think you have to think a bit differently about the consumer/producer problem. In the real world, for the vast majority of cases the actual "producing" and "consuming" part of the algorithm is what takes most of the time. The time waiting by locking/unlocking a mutex is insignificant in the grand scheme of things. Now, to improve the performance of inserting/removing from your buffer you could implement it using a linked list with 2 mutexes. Then you would only need to lock certain parts of the linked list to insert and remove
Is this what you mean?: Why when we add sleep(1) to both the consumer and producer threads it doesn't wait 2 seconds? It's because both threads run in parallel so both start waiting 1 second and stop waiting for 1 second at the same time
Great video! Really enjoyed it. I have a question though. If I have multiple producercers , will this code cover that case as well. If not, could you suggest what other modifications are needed. Thank you.
@@CodeVault It only consumes the last produced element. Shouldn't it produce all N elements, then they get consumed one by one and finally produce N elements again?
This is an amazing video! Thank you so much! I just have one question, say we wanted a certain produce a certain amount of integers, say 100. How would that be implemented?
Because you could have multiple threads entering the same critical section and getting unexpected behaviour. The semaphores are there to signal when a thread should consume/produce not necessarily modify the array (usually the consuming and producing process takes more than actually modifying the array).
Great content! 1 follow up question - Let us assume we have 8 threads, 7 consumers, 1 producer. All 7 consumers will wait on sem_wait(semFull). When the producer produces a number, the waits will be released all at once (This is an assumption I am making on how sem_post works) and then the 7 will try to enter critical section and 1 by 1 try to read but after the first consumer has read the data, it will not find any more data to read. How is this handled?
If sem_wait knows to release all the waits 1 at a time for each sem_post, this problem will probably be solved, right? And given that your code does work, I guess that is how sem_wait might work.
Both sem_wait and sem_post are atomic operations. Semaphores have that integer that it keeps track of for threads to know when they should continue execution and when they should not. Well, that process of checking that integer and decrementing it, is atomic. This means that if one sem_wait checked and decreased the value another sem_wait will ALWAYS see the decreased value thus maybe not continuing execution (suppose we have started with a value of 1) I suggest you even make the experiment yourself
Hi, let's say I want to make a data logger that is getting data and logging it to a database. What sort of architecture should I be looking at. I think this is something to start with.
If it's on the same project that you're logging, then yes, 1 consumer for saving to the database and the producers are whatever threads you have on your project. Otherwise, if it's a standalone project just for logging, you might want to use sockets and send the data through them. There you don't really need thread synchronization as the database should be taking care of that for you.
It's fairly simple, just take the first element and move everything to the left by one index: if (count > 0) { y = buffer[0]; for (int j = 0; j < count - 1; j++) { buffer[j] = buffer[j + 1]; } }
One question I tried running the code and it seems like it never ends. How can I give it a limit to work on say like when there is nothing on the buffer to stop or any type of signal to terminate. Video is amazing as usual.
One simple way is to just change the while(1) to some other proper condition. Another way you could do it is have a timer variable and decrement it each second on a separate thread and reset it back to the initial value whenever you have consumed an item. If the timer gets to 0 then the thread simply terminates the whole program (by calling exit() or something)
If we are using mutex in consumers and let's say there are multiple consumers who wants to work parallelly wouldn't that affect the parallelism of the program?
@@ShivamGupta-gn8cn Well, usually you will need to synchronize threads for certain operations. For adding and removing from the tasks list we really have to have a mutex around it. Although, you should consider that the operation of adding or removing from the tasks list is much MUCH less performance intensive than executing the tasks themselves.
Thank you for this great video! I just have a question about how you'd organize the ownership of these mutexes and semaphores in a larger project: Suppose you wanted to use encapsulation as much as possible and avoid creating things in the global scope, as you have done in the video. Maybe you have a producer struct that stores all the properties of one or more producers, and the same thing for the consumer(s). Who should own the mutexes and semaphores? Do you make a separate struct, maybe in it's own translation unit, that contains the mutexes and semaphores that are going to be used for each part of your project? And then the producer struct just stores a reference for each of the mutexes and semaphores it needs? I'm basically just asking: how do you organize it? Thanks!
Hmm... that's a good question. If I had to do it the OOP way I would simply have a main struct that contains information about the consumers, producers and all the synchronization variables (semaphores, mutexes etc.)
@@CodeVault okay, thanks! If you get the chance, I just want to suggest that I'd be interested in seeing a video that talks about how you'd commonly see (medium to large) c projects structured in the wild, and maybe some common design patterns that are typically used. If you've already done that, sorry! I'm still making my way through your videos.
This is incredible👌👏 Just one quick question . Instead of using two semaphores, can we use one condition variable to wait and signal producer and consumer with some conditions?
just a doubt! do that mutexes really matter hereafter having semaphores? if yes, count++ needs the mutex clearly, whats the case that leads to the race condition?, I left my machine for half an hour running without mutexes, and the grep still didn't find a segfault error!
Race conditions are very subtle and hard to debug. It doesn't necessarily cause a seg fault, it could just be ignoring a write or giving wrong values on reading. If the critical section executes very fast, it would also be difficult to find a race condition there, especially on faster hardware. There are many factors that come into play here. What I do to test race conditions on my own machine is first, make the critical section take longer (if you have a read and then a write on a certain variable I would add a sleep of 0.5 seconds in-between). Another thing you can try is create and start a REALLY large amount of threads (100 for example). Also, you have to make sure you print out the proper results and the right time.
Not really. Since the init value of the semaphore is 10, you could have 10 threads updating the buffer at the same time. That could cause a race condition
Oh yeah you’re right. I was thinking of the first version of the code with just one producer and one consumer. In that case, the semaphore alone would be sufficient?
Despite 1 year late, I would like to correct things here: 1. Use rotating buffer instead of linear one, it's a lot faster for big array 2. Rotating buffer uses 2 integers, cursor and current length 3. Don't overcomplicate the code as example using "for" loop to initialize only 2 thread. This hurts beginners
basically you should not be a beginner if you're watching this video. You're supposed to have a good knowledge of programming concepts, C language, Data Structures and Alghoritms. So a for loop should now scary you. It makes no sense to start learning multi-threading programming if you don't have the basics.
9:56, Frankly I would not share control of the buffer variables, instead I would make the consumer thread treat them as read only, something like: #define PRODUCE 10 bool producing = true; uint consuming = false; uint count = 0; int buffer[PRODUCE] = {0}; void* producer( void *ud ) { while ( producing ) { for ( count = consuming; count < PRODUCE; ++count ) buffer[count] = rand() % 100; while ( consuming ) pthread_yield(); consuming = count; } return NULL; } void* consumer( void *ud ) { while ( producing ) { consume: while ( consuming ) printf( "Got %d ", buffer[--consuming] ); } if ( consuming ) goto consume; return NULL; } **Edit:** Thought of a different method I liked better so I modified the above to reflect
Updated: I got a 95 for my final project thanks to you! (-5 was because of my bad documentation) I had to write my code for it, but without this video, I would not know where to start. I even explained how it works to my teammate. Thank you so much!
Your tutorials are a goldmine. I took an Operating Systems class that literally had 0 lectures to explain the concept of multithread --- instead just a word vomit of text --- and this video was worth 10x of those classes. Keep up the great work!
is your prof kamen?
man ... your channel is like a treasure to me ... I'm really happy that I found it
wow. jesus. I've sat through hours of class about this subject and never understood a single sentence, but you just explained 5-6 lectures in 25 minutes like fucking Picasso, I've never felt so confident about understanding Producer Consumer multithreading. Thank you!
This is a fantastic explanation of a concept that I could not wrap my head around. I really appreciate your work!
It is impressive how you can explain abstract concepts so simply. Great video! Your channel has been so helpful for my Operating Systems course.
These are the best Unix C - Threads/Processes video tutorials ever! As a teacher, I recommend and share your videos with all my students. I recommend you to make video series about sockets and network programming with C. Greetings from Bulgaria!
Thank you so much! I'm glad other teachers appreciate my work. And thanks for the recommendation, it is the most requested so far, I'll look into it as soon as possible
in addition to the excellent explanations of multithreading concepts and examples of how they can be useful in application, i really appreciate how you demonstrate the process of problem solving in every episode, starting from programming the most basic model of the program and debugging it as you add more functionality - it really helps to have that good practice demonstrated for beginners like me. also appreciate how often you look into the camera, it really does make the lesson feel more engaging. thank you so much to you and your team, I'll be referencing your content and supporting your channel for years!
These videos have saved me a lot. My university lectures are so convoluted and over the top it takes me hours to understand what they are trying to explain, yet these videos explain the exact same material in a very clear and concise way. Great job and thank you for these videos on C.
Dude you are a champion, such complex concepts you explained so easily. Your videos are so good, keep posting and continue throwing light in to so many peoples careers
Can't accuse this guy of abusing "Hello, World!" for a viewer count.
Threading practical lab ++
Good stuff, as usual.
Happy holidays, btw.
Happy Holidays and hope you have a Happy New Year!
this video is absolutely fantastic.
I really like your detailed style. explaining everything.
I hope that you're doing well, and that you plan to continue in the future.
I understood both the mutexes and semaphores and I now know how to do my homework. Thank you very much. Very clear explanation. Now I am eager to learn more.
Thank you so much! I have to do a kind of program about producer and consumer for the university, and this video helped me a lot to understand the problem!
Subscribed.. I felt you are the best teacher after mycodeschool teacher. I real understood concepts very well. Thank you
Thank you very much for your lecture. I had never understood producer/consumer problem until I see your video.
Another excellent tutorial! Your YT videos are much appreciated. Thank you.
Thank you! I definitely needed this for school. Sometimes having a topic explained makes it click. Subbed!
this channel is the only reason i am passing my course. you are awesome and keep up the C videos!
Awesome ❤. nice demonstration of pthreads and shared memory and synchronization.
This problem quiet confuse me for a long. Thank you for taking such a good example to help me understanding!
This video really makes it clear why they are called semaphors. It was a fun video to watch, the 25 minutes of the video flied by so fast. Your videos and Jacob Sorber's videos explain semaphors nicely.
Very well explained! In understanding how all these things work together, you helped me see the beauty of it and I really appreciate that, thank you!
omg the odds of getting 69 on the first run... awesome when u think abt it
You are the best man! Your videos are actually helping me so much at college .Thank youuu
very nicely explained and covered the problems with mutex solution
That was a good explanation @CodeVault, thank you very much.
echoing the hundreds of comments you've already received, but this is truly the best OS programming in C channel on TH-cam!
If you have a producer-consumer-problem and use mutexes, you'd use condition variables along with them. That's much more efficient under load conditions where the producers and consumers are constantly processing.
Perfect explanation ..... You are a gem brother.
Excelent Job, Great explanation my friend. Thanks for that.
Simply awesome and echo the same many netizens mentioned in the below comment; Thank You!, can you please show an example for the use of Channel API
Excellent. Thanks From India
haha he almost laughed when he got "69" as the random number , gcc is a kinky , love the videos !
Thank you so much for this video, you make these things very fun and easy to understand!
I level up from dummy to beginner, and the explaination are so clear... Thanks. it's a shame I cannot hit le like button 69 times.
I love u so much man, you’re the greatest
great explanation, thank you for uploading!
Thank you, this was extremely helpful!
Thank you so much! You helped me a LOT!
Fantastic content. Thank you very much!
Thank you. Happy new year 2021
Happy new year!
Stellar explanation!
8:51, nice
I have a question: how is the sem_wait implemented internally?
With interrupt mechanism or busy cycled?
sem_wait is using a spin lock (busy waiting) on Linux. But the scheduler might change it to a lock if the spin-lock is not unlocked in a reasonable time-frame
Nice and helpful video. tank you for that.
Felicitari din nou pentru continut. Sunt developer de 4 ani si am simtit nevoia sa imi reamintesc anumite concepte. Pot spune ca in cateva ore sunt la fel de "fresh" ca in facultate.
Yesss more videos
i hate seeing someone like you, who is such an amazing explainer, getting views like that , for this videos therre is only 1_000 views !!!
also, i hope you doing some machine learning and ANN stuff on youtube in the futur too
good luck
this video is a life saver
Amazing video, thank you!
Thank you so much sir. You are an angel :)
On MacOS have to use GCD (dispatch_semaphore_t, etc) because sem_wait and that family are "deprecated".
Interesting to know
Hi CodeVault, thank you for your great explanation. I want to check if my understanding is correct.
First, sem_wait(), sem_post() have to be atomic (not be interrupted). Because of that, their check need to be very trivial. I.e, wait() and post() should only check if( empty/full >0 ). This concept is called guarded commands.
Now, because these checks have to be simple, we need two semaphore variables for producer and consumer. I.e, for producer, check if empty > 0 and for consumer, check if full > 0. That is the reason why we need two semaphore variables instead of just "int count";
Am I right?
Yes that is correct
Perfect explained!
Thanks a lot! On the source code from the description, the if statement inside the loop is different from the one on the video.
if (i > 0) instead of if (i % 2 == 0 )
Thanks for pointing it out! I changed it in the video near the end to only create one consumer. You can use either one of those to experiment yourself
@@CodeVault oh ok that makes sense. But if we want to split the threads between producer and consumer evenly we should keep the mod right?
Thanks for all the content by the way, great stuff!
@@ericbaptista536 Yes, exactly!
Thank you for clear explanation. As I understand you use both mutex and semaphores because the usage of two different semaphores does not prevent producer and consumer from executing critical section simultaneously. So you need mutex as well. Is my conclusion correct?
Yes. Without a mutex, you could produce multiple items at the same time, have the critical section executed by 2 consumers at the same time which creates a race condition on the array it's reading from
So can we say mutex's main purpose for producer/consumer environment is to lock the critical section and semaphore's main function is to provide synchronisation between the threads?
Yes. Basically semaphores are just there so that you can notify threads to wake up when something happened and mutexes for what you mentioned: locking the critical section
Yeah dude amazing lecture
Thanks for your perfect and beautiful code
But why for us didn't print done when the buffer was consumed?
In my case i produce n items and when all of them was consumed it shows done
So i bounded while for n items
Is it true?
I'm not sure what you mean. Can you share the code and explain your question a bit more in detail?
Thank you brother..great video
I loved the video. Can you please tell the solution only using mutex conditions ???
Hmm, I might look into it
So can we remove the mutex lock once we use the semaphore or do still we need it? As there might be issues when both producer and consumer are working on buff simultaneously
You still need them as you might have multiple consumers taking from the array at the same time. The idea is that semaphores are synchronization objects for the whole "consumer" and "producer" process while the mutexes are only there to prevent race conditions on the array itself.
Great video. However, I was wondering if we could just use 1 semaphore will this problem be solved?
No. Because you cannot make a thread wait on an arbitrary value of a semaphore (you need to wait for both producer and consumer threads in different situations)
8:51 this man hella lucky
14:30
Why is it a problem? Just use a ring buffer with a read pointer and a write pointer. If the read ptr is the same as the write ptr, the buffer is empty. The write pointer advances the write ptr after the data is stored at that memory location. You will probably need a fence there, to ensure that the pointer is incremented after the data store. The read pointer never increments past the write ptr.
It's a common problem taught by many professors across the globe. You can always argue about its usefulness in the real world but that's not my goal here with those videos. Students are taught poorly about this topic and I'm trying to provide better material for them to learn from
I might make a video implementing the improvements you listed there
@@CodeVault oh, cool beans. I had a microcontroller with a cortex-m4 paired up with a m0+ coprocessor. I think it was an NXP LPC 43xx or something. They had a piece of hardware that managed a communication channel between the two cores that worked like I describe. It's kinda like a plan9 channel. Sadly... I don't know if it could be extended beyond two communicating threads...
Great video , is there way thr consumer can add to the buffer with minimum waiting while the slow consumer is busy consuming. Thanks. Thst Is When there still slots on the buffer. Thanks.
I'm not sure what you mean by "minimum waiting" but simply executing what the producer is doing inside the consumer's function should do the trick. Just be careful to not create an infinite loop since then your consumer would also be a producer
@@CodeVault Thanks for the insight. My producer can very rapidly insert new packets to a fixed size circular buffer/array[4], but my consumer is very slow. Producer should only wait if already full of new packets, and consumer can wait if no packets. But producer shouldn't be blocked while trying to insert a new packet when not full. I was worried the slow consumer taking the mutex could prevent the producer to take the mutex to insert a packet to non-full circular buffer. The slowness of the consumer shouldn't affect the producer from adding new packets while not-full. Appreciate any suggestions, thanks, and merry Christmas.
The mutexes on both sides are important as to not cause race conditions. I think you have to think a bit differently about the consumer/producer problem. In the real world, for the vast majority of cases the actual "producing" and "consuming" part of the algorithm is what takes most of the time. The time waiting by locking/unlocking a mutex is insignificant in the grand scheme of things.
Now, to improve the performance of inserting/removing from your buffer you could implement it using a linked list with 2 mutexes. Then you would only need to lock certain parts of the linked list to insert and remove
This was so good 👍
you are a genius
Can you please make a content on thread schedule policy
I'm still not understanding why when u put 2 sleep of one second the thread doesn't wait for 2 second before answering. did i miss something ?
Is this what you mean?: Why when we add sleep(1) to both the consumer and producer threads it doesn't wait 2 seconds?
It's because both threads run in parallel so both start waiting 1 second and stop waiting for 1 second at the same time
Great video! Really enjoyed it.
I have a question though. If I have multiple producercers , will this code cover that case as well. If not, could you suggest what other modifications are needed.
Thank you.
Try it for yourself first and see if that works
@@CodeVault It only consumes the last produced element. Shouldn't it produce all N elements, then they get consumed one by one and finally produce N elements again?
This is an amazing video! Thank you so much! I just have one question, say we wanted a certain produce a certain amount of integers, say 100. How would that be implemented?
You could pass the amount to produce as an argument to the producer. Here's how to do that: code-vault.net/lesson/18ec1942c2da46840693efe9b51f9f86
why do we still need to lock with a mutex once we've implemented the semaphore?
Because you could have multiple threads entering the same critical section and getting unexpected behaviour. The semaphores are there to signal when a thread should consume/produce not necessarily modify the array (usually the consuming and producing process takes more than actually modifying the array).
Great content! 1 follow up question - Let us assume we have 8 threads, 7 consumers, 1 producer. All 7 consumers will wait on sem_wait(semFull). When the producer produces a number, the waits will be released all at once (This is an assumption I am making on how sem_post works) and then the 7 will try to enter critical section and 1 by 1 try to read but after the first consumer has read the data, it will not find any more data to read. How is this handled?
If sem_wait knows to release all the waits 1 at a time for each sem_post, this problem will probably be solved, right? And given that your code does work, I guess that is how sem_wait might work.
Both sem_wait and sem_post are atomic operations. Semaphores have that integer that it keeps track of for threads to know when they should continue execution and when they should not. Well, that process of checking that integer and decrementing it, is atomic. This means that if one sem_wait checked and decreased the value another sem_wait will ALWAYS see the decreased value thus maybe not continuing execution (suppose we have started with a value of 1)
I suggest you even make the experiment yourself
Thank you!
Hi, let's say I want to make a data logger that is getting data and logging it to a database. What sort of architecture should I be looking at. I think this is something to start with.
If it's on the same project that you're logging, then yes, 1 consumer for saving to the database and the producers are whatever threads you have on your project. Otherwise, if it's a standalone project just for logging, you might want to use sockets and send the data through them. There you don't really need thread synchronization as the database should be taking care of that for you.
How could the consumer be created to make it a first in first out buffer instead of a first in last out buffer?
It's fairly simple, just take the first element and move everything to the left by one index:
if (count > 0) {
y = buffer[0];
for (int j = 0; j < count - 1; j++) {
buffer[j] = buffer[j + 1];
}
}
@@CodeVault Thank you! This was basically one of the last pieces to completing my class project!
Very nice video💪🏼
Nobody can convince me otherwise that he wasn't just sitting there re-recording his section until he got 69 as a random number
I got it the first time so there was no need to re-record😎
One question I tried running the code and it seems like it never ends. How can I give it a limit to work on say like when there is nothing on the buffer to stop or any type of signal to terminate. Video is amazing as usual.
One simple way is to just change the while(1) to some other proper condition. Another way you could do it is have a timer variable and decrement it each second on a separate thread and reset it back to the initial value whenever you have consumed an item. If the timer gets to 0 then the thread simply terminates the whole program (by calling exit() or something)
Thank you.
If we are using mutex in consumers and let's say there are multiple consumers who wants to work parallelly wouldn't that affect the parallelism of the program?
Well, yes, that's exactly why, when you have a program using say 2 cores instead of 1 you don't see a 2x performance gain
@@CodeVault so what should we want to achieve that particular scenario?
@@ShivamGupta-gn8cn Well, usually you will need to synchronize threads for certain operations. For adding and removing from the tasks list we really have to have a mutex around it. Although, you should consider that the operation of adding or removing from the tasks list is much MUCH less performance intensive than executing the tasks themselves.
can you explain how to check buffer is full and empty
You just check if count >= 10, then you know the buffer is full. If count is 0 then the buffer is empty
Thank you for this great video! I just have a question about how you'd organize the ownership of these mutexes and semaphores in a larger project:
Suppose you wanted to use encapsulation as much as possible and avoid creating things in the global scope, as you have done in the video. Maybe you have a producer struct that stores all the properties of one or more producers, and the same thing for the consumer(s). Who should own the mutexes and semaphores? Do you make a separate struct, maybe in it's own translation unit, that contains the mutexes and semaphores that are going to be used for each part of your project? And then the producer struct just stores a reference for each of the mutexes and semaphores it needs? I'm basically just asking: how do you organize it?
Thanks!
Hmm... that's a good question. If I had to do it the OOP way I would simply have a main struct that contains information about the consumers, producers and all the synchronization variables (semaphores, mutexes etc.)
@@CodeVault okay, thanks! If you get the chance, I just want to suggest that I'd be interested in seeing a video that talks about how you'd commonly see (medium to large) c projects structured in the wild, and maybe some common design patterns that are typically used. If you've already done that, sorry! I'm still making my way through your videos.
No, I haven't. I noted it down and may look into it at some point, thanks!
Thanks best regards...
This is incredible👌👏 Just one quick question . Instead of using two semaphores, can we use one condition variable to wait and signal producer and consumer with some conditions?
Yes, you can do the same thing with condition variables but you'd need two of them. One for producing and one for consuming
Thaks for the video!
just a doubt! do that mutexes really matter hereafter having semaphores? if yes, count++ needs the mutex clearly, whats the case that leads to the race condition?, I left my machine for half an hour running without mutexes, and the grep still didn't find a segfault error!
Race conditions are very subtle and hard to debug. It doesn't necessarily cause a seg fault, it could just be ignoring a write or giving wrong values on reading. If the critical section executes very fast, it would also be difficult to find a race condition there, especially on faster hardware. There are many factors that come into play here.
What I do to test race conditions on my own machine is first, make the critical section take longer (if you have a read and then a write on a certain variable I would add a sleep of 0.5 seconds in-between). Another thing you can try is create and start a REALLY large amount of threads (100 for example). Also, you have to make sure you print out the proper results and the right time.
@@CodeVault thanks, man! Now I got the point, trying different print statements helped me, rectifying my mistake
Now the real question is, can we use semfull value to point the array index (in lifo fashion) instead of count variable. Great content, subbed btw.
Hmm, I'm not sure I understand the question. Can you elaborate more please?
Do you really need the mutex after adding the semaphores?
Yes, because you could have the producer produce 2 or more items at once and that would lead to a race condition when editing shared data.
what about solving the producer consumer problem using monitors?
Haven't made a video on monitors, I'll leave you to figure it out till then ;)
why do you need both a semaphore and a mutex? can't you just use one?
Not really. Since the init value of the semaphore is 10, you could have 10 threads updating the buffer at the same time. That could cause a race condition
Can we remove the mutex? Isn’t the semaphore already protecting the critical section?
It isn't. You could have multiple threads pass the sem_wait at the same time, thus creating a race condition.
Oh yeah you’re right. I was thinking of the first version of the code with just one producer and one consumer. In that case, the semaphore alone would be sufficient?
Yeah, with one consumer and one producer, it should be sufficient
congrats!
Despite 1 year late, I would like to correct things here:
1. Use rotating buffer instead of linear one, it's a lot faster for big array
2. Rotating buffer uses 2 integers, cursor and current length
3. Don't overcomplicate the code as example using "for" loop to initialize only 2 thread. This hurts beginners
basically you should not be a beginner if you're watching this video. You're supposed to have a good knowledge of programming concepts, C language, Data Structures and Alghoritms. So a for loop should now scary you.
It makes no sense to start learning multi-threading programming if you don't have the basics.
thank you for this amazing video its pretty useful
just one question
how can we do the same thing with a binary Semaphore instead of mutex
(for others that might have the same question) I explain it in this lesson here: code-vault.net/lesson/dlppk1iiz7:1609433599467
9:56, Frankly I would not share control of the buffer variables, instead I would make the consumer thread treat them as read only, something like:
#define PRODUCE 10
bool producing = true;
uint consuming = false;
uint count = 0;
int buffer[PRODUCE] = {0};
void* producer( void *ud )
{
while ( producing )
{
for ( count = consuming; count < PRODUCE; ++count )
buffer[count] = rand() % 100;
while ( consuming ) pthread_yield();
consuming = count;
}
return NULL;
}
void* consumer( void *ud )
{
while ( producing )
{
consume:
while ( consuming )
printf( "Got %d
", buffer[--consuming] );
}
if ( consuming )
goto consume;
return NULL;
}
**Edit:** Thought of a different method I liked better so I modified the above to reflect
Thank you so much!
Updated: I got a 95 for my final project thanks to you! (-5 was because of my bad documentation) I had to write my code for it, but without this video, I would not know where to start. I even explained how it works to my teammate. Thank you so much!
awesome content
cheers mate!