AsyncIO: Implementing a Producer-Consumer Model
ฝัง
- เผยแพร่เมื่อ 1 ส.ค. 2024
- A complete example showing how to implement a producer-consumer model using Python asyncio.
#mathbyteacademy #python
Code for this Video
================
Available in GitHub blog repo: github.com/fbaptiste/python-blog
Direct link: tinyurl.com/yvtrrmz3
My Python Courses
=================
- Python 3 Fundamentals (introduction to Python)
www.udemy.com/course/python3-...
- Python 3 Deep Dive (Part 1 - Functional)
www.udemy.com/course/python-3...
- Python 3 Deep Dive (Part 2 - Iteration, Generators)
www.udemy.com/course/python-3...
- Python 3 Deep Dive (Part 3 - Hash Maps)
www.udemy.com/course/python-3...
- Python 3 Deep Dive (Part 4 - OOP)
www.udemy.com/course/python-3...
![เจ้าผู้เดียว - ดิด คิตตี้ [OFFICIAL MV]](http://i.ytimg.com/vi/q0NLCYKBOTo/mqdefault.jpg)
Hey Fred, it's great news you started this channel. Your deep-dives helped me tremendously and I constantly recommend them to Python learners as one of the highest-quality resources out there. As much as I would be most excited about you releasing part 5, I would also gladly welcome your YT videos on such topics as:
- implementing design patterns
- automated testing
- socket programming
- advanced Numpy & techniques of speeding up Python code through vectorization
- cython, numba, pypy (beyond hello world)
- more on using descriptors as a way to improve code's cleanliness
- advanced type hints, generics, static type checkers
- celery/rabbitmq/this kinda stuff
- virtual environments, best practices of dependency management for complex projects (conda vs. pip)
Thanks so much!
Thanks Marceli - those are all great ideas!
Second on that!!!
Yeah.. design pattern is something which is very less talked about in most of the python tutorials and books.. If fred start a series to explain al the solid principles and 20 or so design pattern his course would be the hit.. we pythonist really need good understanding of these concepts to make python enterprise level language..
Hands down your Udemy class is the best I have seen online for a Python course. I bought all of them. I frequently search my email or Google for announcement from you to see if you have a new Python course out. I clicked subscribed here without reading your email :). I will buy any course you put out there. You have an audience out there that will reward you for your effort, Fred.
Completely agree, Fred taught me python, but Fred also expanded my general programmer knoweldge and CS knowledge as well. Such a wonderful lecturer. All time favorite.
His courses are THE best....
I hope Fred keeps doing the awesome work he is doing.
Thank you!
Hi, Fred!
I've never had a task to do in Python but I've started to learn Python for the sake of your courses - best of the best. I'm a database guy (Oracle) and I'd be more than happy if you could share some tips/patterns about how could we asynchronously link Python and Oracle database.
Thank you for your wonderful work!
Hi Cezar,
Unfortunately I have not had the opportunity to use Oracle professionally. I did a few web searches, and I see that Oracle has a Python driver (oracle.github.io/python-cx_Oracle/). As fas as async goes, it looks like there may be ongoing work to make that happen (github.com/oracle/python-oracledb/issues/6). In the meantime, this project exists: github.com/GoodManWEN/cx_Oracle_async
I watched it in one breath. It's what I needed
Thank you, Fred. I own all your Deep Dives on Udemy and Beginner Python on Math Byte. I love when expert Python programmers build from scratch real world examples. These examples help bridge the gap between learning to code and what one actually does with the code to build/create.
Hello Fred, Thanks for your courses and now the blog, they are amazing.
If possible have a video discussing FUnctional/OOP programming, pros, and cons.
Hello Fred, Thanks for your awesome YT videos. I've passed your deep dive course and I believe it is a comprehensive reference to level up in Python3. cheers
Thanks!
Fred, you’re the best python teacher ever. Requesting you to make a video on asynchronous and reactive programming (RxPy probably) and some use cases or design patterns with those.
Async is a course I'm working on - maybe done by end of this year.
great video!
Man, this is beautiful, amazing. Like and subscription
Hi Fred! Your courses are just fabulous. I enrolled in all your four courses available on Udemy.
I would like you to make a detailed video/course on Rest API Automation with Asynio, Aiohttp, etc. Thank You!
Super enjoyed your Udemy courses! So in depth and thorough! Looking forward to your videos on design patterns!!
Thank you!
@@mathbyteacademy is there a course about design pattern planned on youtube?
Hi Fred! when is your Deep Dive part 5 coming? I learnt a lot from your udemy courses. thank you !
Working on it! But not getting enough time to make much progress, hence these videos - this way I can publish videos one by one, instead of the 6-9 months it usually takes me to create a course.
41:03 thanks for making
you're welcome!
to be honest it was very complicated, I have to see again and again!
I think we could create one consumer function and use it as a consumer for work queue and as a consumer for result queue.
So handle_task_result is also a consumer, but it extracts data from another queue (result queue).
And consumer's signature would be def consumer(srcQ, optional destQ, callback)
I'm wrong. In consumer we have async callback, but in result handler it's just a regular function.. hmmmmmm
Thanks for the video, Fred. Could you please consider making some videos on asyncio events, queues and other asyncio nuances?
You're welcome! async topics is on my todo list.
Hi Fred, I am your student on your Udemy courses. I would be really appreciated if you can make videos on asyncio topics.
Hi Fred,
A year ago, I discovered your Deep Dive course, and I have been fascinated by it ever since. Words cannot adequately express my gratitude to you.
I have a question regarding this model: I need to use an Excel workbook variable in the "main" function to read task data, and the same variable in the "task_completed_callback_handler" to write the results. Currently, I am using this variable as a global variable, but I am aware that this is not considered a good practice. What is the correct way to achieve the same goal?
Nothing necessarily wrong with global variables, you just have to be extra careful with shared state. Can't speak to your specific use case.
Hi, can you make a lambda tutorial?, I'm taking your OOP course on udemy.
We could make task_data to be a generator to not hold all the data in memory
intro video on asyncio, please)
I would recommend looking at these videos - very well made and gives a deep understanding of async:
th-cam.com/play/PLhNSoGM2ik6SIkVGXWBwerucXjgP1rHmB.html
Hello Sir,
Thank you for such detailed explanantion. Could you please do deep dive on asyncio in one of your future videos or may next next deep dive 5 udemy course?
You're welcome! I am working on a async course, no ETA yet.
@@mathbyteacademy Waiting impatiently!!
Thank you for this tutorial. Is there a way the consumers can start the work immediately when there is a task in the queue? With this implementation, it looks like the producer must be done creating the tasks before the consumers can start do the work.
Sure, you just have to set things up differently - the tricky bit will be to determine when things are "done". You can always track the completed tasks, and then shut everything down once you know everything has been processed. If it's a never ending process (i.e. continuous production/consumption) then that makes it easier.
by the way, since you are using type annotations, are you planning to make a video about using them?
I am indeed - working on one right now actually
Awesome. I was doing this using redis. Now python seems more useful. I am wondering if we can come up with a python script which will help us to determine optimal figure for constants used in script for producer consumer workflow. May be by testing it for 1 hours. Fred is it possible?
I usually tune it by running test batches. In my case the final callbacks don't do much (they just save to a database), so my main focus is tweaking number of "workers". I run tests with the same batch of work, and look at the overall time as well as queue size (% fill, since queue is bounded). It's rough, but I don't need to refine it that much - keep in mind that all this is still just running on a single core - as long as I get a total time reduction that's acceptable, I go with that. If I need to go beyond a single core, then having distributed consumers/workers is better - in that case something like redis or rabbitmq, or some other queue mechanism, with distributed workers would be more appropriate.
@@mathbyteacademy Thanks for the reply. I guess our decision for distributed consumer/workers should be more inclined towards the kind of task assigning to a worker. If task is network bound then we have some privilege of assigning 2 or sometimes more tasks to a worker. Likewise if task is CPU bound it is ideal to pick a single task to a worker for achieving better concurrency as long as CPU bound tasks and network bound task are handled in different queue and separate workers.
Fred,
Do you have plans to create a course which covers asyncio, threading & parallel processing etc.?
These are complicated topics with no one course covering them all in sufficient detail, at least on Udemy.
Not at the moment, no - just don't have enough time for a full course - but eventually :-)
@@mathbyteacademy Cool. As long as it is on the future roadmap..
By the way, great work. I am buying any course you publish on Udemy. 🤠
Hi Dr. Baptiste, I completed all your deep dive courses. I am looking forward a deep dive course on AsynIO/threading/concurrency. When will it be available? Thanks!
I have started on one, but no ETA yet.
@@mathbyteacademy Thanks for update. Looking forward to all your Python courses.
is it like what we can do with redis queue?
In the sense that both are queues, then yes. If you need priority queues, then Python asyncio also has that (PriorityQueue)
Fred, what do you say about people who say that Python doesn’t scale well and that it’s slow?
So for enterprises languages like Java, C#, Go, are better for building apps that scale. Is there truth to that? I’m a newbie to all of this so would love to hear your feedback.
I say they have a lot to learn about software engineering still! 😀
I would suggest reading this blog post from Mahmoud Hashemi:
sedimental.org/10_myths_of_enterprise_python.html
@@mathbyteacademy Thank you so much! Also, how would you recommend non-cs degree people new to software engineering to learn computer science fundamentals?
My degree is in chemistry so I haven’t take any CS classes and want to be a great software engineer.
if we were to sum the numbers in the results, where and how would we do that?
Not something that's easy to do (this is sometimes referred to as fork-join), hence why frameworks such Celery are popular. For a more distributed approach, Airflow is a good solution. Basically whatever handles the results queue will need to know what parts of a split task have been completed, and take action when all the parts have been completed (i.e. you have to maintain state somewhere to track that). Not simple enough to explain in detail in a comment, but maybe start thinking along the lines of creating some state (maybe a dictionary), that has entries for each task that is being split (maybe it's just a counter that stores a task identifier of some sorts and the number of sub-tasks, or splits, that need to complete). The message in the queues will need to contain that task identifier, and in the results handler, you not only store the result of the sub-task, but also update the state (like decrementing that counter). Every time you also check if the counter has reached zero in the results handler, and if so, you now gather all the sub-results that you stored somewhere for that task, and do something with all the now completed results (like aggregate, or whatever).
There are other methodologies, probably better suited than the one I describe in the video, to implement fork-join flows. So kind of a different topic altogether.
Hi Fred, I am interested in purchasing Intro to Python course. But the price is a bit high for me as I am a student. May I know the reason for not publishing Intro to Python course on Udemy?
Check if your local library gives free access to Udemy courses.
He already has 4 courses on udemy
just wait a bit for some flash sales, udemy gives them frequently just buy it during that time. i got the course for $5 (INR 455) each.