Hey, Arjan it's been a while now following your design guide videos. But the real problem is that, is it good to place more than one class in a single file, I have been coding in PHP where every class takes it's own file, the code is simple to refactor in that way. What are thoughts?
The naming of the ICECREAM module is a clever play on the abbreviation "IC", which phonetically mirrors "I See" in English. This choice is fitting for a Python debugging tool, as it elegantly underscores its purpose: to illuminate the inner mechanisms of your code, much like peering through a transparent layer in a typically opaque Python environment.
A CLI app is always a good idea when automating boring tasks at work like building reports or extracting data to manipulate. Its quick to develop and fast to execute.
A deep dive video on SQLModel would be great. I like it. On my first pass at learning it, the foreign key constraints were a little confusing and I ended up refactoring some code once I fully understood it. But seeing a video might be a great way to know if there is a better way to define them. I was doing a light project with a SQLite-based database, but ultimately want to get to a project that uses PostgreSQL with geocoding. If you did a video like that I would find it very interesting.
Great video, thanks! Here is a new entry: Ray is a nice framework with a Python API that facilitates code parallelization, whether it is on a single machine or within a designated cluster. Side note for people who might be interested: while Ray offers a user friendly experience, there is a certain degree of syntax overhead, making it challenging to run code in a serial manner without code modifications. To that end, I have build and published a wrapping library (still a work in progress), “ray-ease”, that allows to switch from parallel to serial without requiring code modifications.
Since I've started to learn rust I found the Result type (railway oriented programming) an incredible programming mindset. The basic idea is that many of the exceptions we throw are not really exceptions, but just different paths that our program can take, so we shouldn't treat them differently. If that idea is well integrated into the language, is very powerful and allows to prevent issues.
Learning Rust ATM and I chuckled when Result was discussed, especially with the pattern matching. Should definitely be a language feature in the std library
@@captainwasabi So are for loops, so is all of structured programming. Both exceptions and result types have their uses. E.g. operator overloading is utterly unusable without exceptions, might as well not even try. Exceptions are also much more convenient (and performant) when errors happen several nested layers deep. Also, result types are fundamentally unsuited to dynamic languages like python.
@@ArjanCodes Even better than Argparse and Click is Typer give that a try. Of course it's a layer on top of Click. Typer is made by the same person who made FastAPI.
Click is cool but Typer is also great. Fastapi, SQLModel and Typer are all made by the same author: @tiangolo (Sebastián Ramírez). He's a wizard with leveraging type annotations for a great developer experience!
I write CLIs all the time. I generally design everything like a library so it would be easy to attach any frontend to it, but CLIs are easy to crank out with libraries like Typer. At work, I write a lot of custom tooling for things like: managing/requesting TLS certificates, DNS management for customer domains, and other oft-repeated day to day tasks. At home, I wrote an application to automate organizing my media collections (with remote metadata retrieval). I'm usually in the terminal,and automating GUIs is a pain, so I almost exclusively write CLIs for my libraries and leave the GUI or web service frontend for others to implement.
I adore icecream. Being able to run unit tests and get every variable in colour and very visible on the terminal with just a two letter function when debugging.
One of my go-to libraries for adding 12-factor structured logging to my code is structlog. Works equally well as a replacement or enhancement of the standard logging library, and extremely flexible and extensible.
@@yash1152pretty sure they just mean "structured logging" (logging all the data in a format like JSON) in a "12 factor app" (which is an architectural pattern for containerized/cloud native applications).
I've recently started using structlog as a drop in for the std log library, and I can't go back. It has the features of rich + loguru in an easy to use interface!
We add CLI to almost all of our projects for a couple reasons: 1) it's a quick and easy way to test out what is the proper interface/API to your applications before you invest into a more complex API tooling and 2) our applications are mostly pipelines (orchestrated ML lifecycles), not always-up applications. CLI is just a very robust and controllable entrypoint for that type of applications.
Good to see a recommendation for argparse. Most of these modules are already used by me so I'm glad I'm on a similar track. For the question on command line programs, we create command line programs as the work I do is mainly backend and we try to automate as much of our repeatable and tedious processes as we can. Vendor software provides some functionality and our tools just enhance that to make maintenance seamless and fill the gaps not covered by the vendor.
If you do any calculations that requires units I will always recommend 'units' from 'astropy'. You can basically transform any unit to any unit (like nanometers to feets, if you need that for some reason) in an easy way. Pretty useful in equations where you are not sure if you have converted the units properly as well
FYI, 'pint' is a neat, standalone alternative to 'units' from the excellent 'astropy'. In a bit of testing I found 'unit' somewhat faster BUT either will add considerable overhead when doing computations using quantities with units. I slightly prefer 'pint' in usage. In any case, I recommend one use quantities for I/O function to validate inputs and to provide quantities with units as output, for doing dimensional analyses and validation through the compute chain during development and testing, but to only pass values from input quantities into production computations and then add units back to the outputs. The cool thing is that the computational chain (if implemented with the right abstractions!) should run either quantities or values without any changes.
I build command line interfaces all the time for data science / ETL utility scripts & processes, pulling data from APIs, roll-up aggregate jobs, interacting with cloud services, etc... argparse, tqdm, and pydantic are all super handy.
Thanks for the informative video... A video on SQLModel would be nice! What about a video about integrating between a REST API and a SOAP API using Zeep (Getting the data from both APIs, manipulating and comparing the data between the two to check if you should make a post, update or delete request to the REST API)? I think it should be interesting... It can probably just be 2 REST APIs as well...
I make mostly command line tools. I like to use typer for argument parsing when it doesn't conflict with my requirements otherwise I use argparse. I'm surprised typer wasn't on your list.
Command line interfaces are very useful for astrophysics simulations (at least; I don't want to talk outside my experience), specifically when the physical model you're implementing tries to do as much as possible from first principles. In essence, the whole point of the code is the code itself, not the results; the results only serve as something to compare with observations to evaluate how accurate your physical model is. In this context a good command line interface --- especially when coupled with judicious use of shell scripts --- simultaneously abstracts the concept of a physical model, with (hyper)parameters etc, very naturally, and allows you to easily bring collaborators up to speed, since the code doesn't just document itself, it also explains itself through the story it tells via `tqdm` and friends. More broadly: The closer you are to the bleeding edge of pure research, the more atomic your toolset is. This naturally pushes you into the shell, since cobbling together bespoke pipelines, with your own bits woven in, is much easier there. Hence command line interfaces being useful. (And, of course, a lot of your computing is going to happen on clusters, which often speak POSIX.)
@@iridium1118 To an extent it comes down to personal preference, although I stand by the abstraction argument. I prefer to split the different tasks involved between the shell and python because I feel at home in the shell, and prefer the toolset that it provides for file and task management over those available in python. Of course, these aren't mutually exclusive. I always implement the command line interface inside an if __name_ == "__main__" block, and render my code such that its various parts can be used interactively via ipython; i.e., such that it can be used as a library. (And for the record: I don't mean Jupyter notebooks. Browsers aren't for editing. If my code is usable from inside a Jupyter notebook, that's incidental.)
If you're contemplating creating a follow-up video, I highly recommend including a section on the Typed Argument Parser library. I've recently switched to using it instead of argparse, as it offers a more modern and type-safe alternative to Python's argparse library.
I'd be curious about that too. I have also heard click is great for creating command line apps. I have used it to deal with user prompt inputs and it made a complicated while loop much easier to read and user friendly in just a few lines.
ปีที่แล้ว
First thing I thought of when I heard argparse was TAP! It's such a simplification over argparse, and also allows scripts meant for CLI to be more easily tested and/or used from code, since you can construct the arguments very easily. I've also seen it used alongside click, but I have less experience with that.
I had to use polars for one project due to Pandas being too slow. That project was an exception, and i am typically working on datasets where the speed of Pandas is not an issue. However, I still use polars when I can because I find the syntax so pleasant. I think its going to vary a lot between analysts (most’d rather use SQL than either library) but there’s a chance that for you, its best feature is not its speed but its syntax.
Not a library, but I wanted to mention this: Python Poetry saved me so many headaches at work the past 2 months. I have been been a single dev working on some new infrastructure as code stacks with TerraforCDK and was having a rough time with dependency management. It simplifies virtual envs, dependencies and packaing. Def worth checking it out!
really nice list, thanks for sharing Arjan! i started using argparse recently in a data engineering context, to set up a main job executor and pass ETL jobs and params as arguments. between that and config libraries like omegaconf and hydra, it's completely changed my application design for the better.
Some really cool packages in here that I've never heard of! Please do a video on SQLModel and how to do normal SQL operations such as groupby and aggregations. I tried using this in my code but I found it a lot slower than using Sqlalchemy. But I may have done it wrong. A video would really be amazing!
5:50 I typically build tools with dynamic menus that can show/hide options depending on circumstances. tqdm is one of libraries to show file transfer. For example I built file manager for my photography work that can automatically batch process newest photos into jpgs, save cropped frames and things like that. As for the logging I use my own function to print and write to file with preset formatting. Sometimes I make CSVs out of this so custom method helps. For me it’s more abaut information gathering than pretty colors in console
I write CLIs mainly to fill gaps in automation. Cronjobs, Deployment or Backup scripts etc. I usually write as a CLI, rather then a simple script. This means more work upfront, but is usually worth it once someone needs to execute it manually for the first time, and someone always will. I usually use click in combination with rich, so thanks for this list of libraries. I will check out argparse for smaller CLIs, since it will reduce the overhead, especially in regards to deploying the app.
Honestly happy to see Loguru on this list. Came across it like 2 years ago and much prefer it's simplicity over Pythons built-in logger WHILE still being able to combine both loggers AND maintain complexity as you need it. Brilliant package IMO. 🎉
Yes, I do regularly create commandline python apps, and I use argparse and rich :) The apps are all used exclusively in house in our company, and they are mostly wrappers for internal APIs.
Excellent video! I will definitely include some in my stack. I would like you to make a Sqlmodel video, I think it is an interesting option to consider
We write software with command line interfaces all the time in scientific computing. I use argparse very often. Was glad to learn about tqdm, it looks very slick
Yes, you betcha I use the command line! Specifically, I have dozens of users who use my Makefiles to git/smbclient/git-select/xmllint/Bamboo configure/setup/deploy our developer environments. The Makefile figures out the host OS and installs OS dependencies using one of yum/apt/yast. It'll also use the pipenv files and specific Python selection to create bash shell launcher scripts. I'm particularly interested in rich-cli which give me a nice interface to either prompt a user for a branch or supply them documentation.
@@jeffgruenbaum I'm not sure what you mean by outside of Django? It can be used without using a web server and you can access and manage your db through Django ORM in your own scripts and applications.
I use 'click' when i want to build CLIs - I love the ability to unit-test my code and commands easily, and also test the actual command line itself - and I like the decorator/dispatch model.
Thanks for compiling this list of 15 useful Python libraries! It would have been even more beneficial if you could have included some examples of how each library can be used. That would help beginners and even experienced developers grasp their functionality more effectively. Nonetheless, I appreciate the effort you put into the video. Keep up the good work! 👍
I just found the abseil library for python and so far I really like it. I haven't completely understand all of the build abilities it provides by using bazel, but for building command line Interfaces I like it way more then argparse. You can define arguments as global flags und use or modify them anywhere without having to pass them down accordingly. Found this in a Google Project code and it instantly blew my mind
Python's built ins now handle datetimes with timezones really elegantly with the default Datetime and ZoneInfo modules. Might be worth doing a video on those, a lot of people don't realize ZoneInfo is in the standard library.
Thanks, Arjan! This sort of "speed dating" video is usually lame,but you did the extra work of putting in the index and the links, so it's very useful. I would suggest one improvement, though: add a one-line description (just a phrase, like "Does error logging and adds debugging capabilities') to the libraries, because the names often don't give even a tiny clue as to what they're about. You asked about using CLI. I go back to the days of punched cards, and spent a lot of my career doing embedded systems, so CLI is my "native environment". For me, GUIs are what you use to make your CLI results look pretty ;-) When I took my first run at learning Python several years ago, I tried argparse, but found it confusing because it required a fair amount of understanding of Python,especially what you can do with various data types. I'm sure I'd see it differently now, especially if I were writing code for others. But I have decades of experience with getopt, both as a user and a programmer, so I use it in Python, too.
invoke is my go-to for building CLI tools. Hierarchical configs, zero argument parsing (!!), command chaining, self-documenting. Its so simple I build a little CLI for every project as executable documentation for processes, tasks, etc. Can't understand why invoke is not in every Python developer's toolbox. And fabric adds ssh on top so you can run invoke tasks on remote hosts. Maybe if Arjan did a video on invoke and fabric more people could enjoy the awesome.
Great job with the libraries Arjan!🚀 If anyone’s looking for more Python videos, we’ve released how-to Loguru logging, task scheduling, and more to help the community too 💪
icecream may be a word play on ICE - In Circuit Emulator, a design tool from the past (nowadays In Circuit Debuggers are the norm, because transparent hardware emulation of a fast chip is hard or impossible) for embedded systems development. ICE's were expensive, a luxury way to develop and debug.
I might as well also mention Maturin, which makes working with mixed Python/Rust codebases a breeze. I'm currently working on a general-use project template for such projects using it.
Thank you, very useful information for me. I use the terminal almost always, as a Linux user, I create an alias for the software I did and it's enough for me. But I must say I m over 60s and I just study and work with Python for my own fun, also I m very used to 80/90's terminal environment, so it's way better for me than using a laggy GUI.
For quick deployment, O would like to mention gradio: few lines of code and a user without programming experience is able to interact with a Machine Learning model
@ArjanCodes - I think that it would be very interesting to learn what you experience in other developers as far as weaknesses go. It is likely that a weakness you repeatedly experience is prevalent in other coders as well. Knowing these weaknesses would help other developers potentially learn from you and avoid the same issues.
For PDF's, PyPDF is a good start, but PDFMiner (for parsing) and ReportLab (for building) are much more powerful libraries. Also, if your .env files have anything other than just strings in them, the library you want to use is decouple
Python command-line programs is great when working with docker containers, and when working with containers that process jobs and need input parameters.
Hi Arjan. This video was really cool 👍 I'm happy I know quite a lot of those libraries 😀 Great you mentioned SQLmodel. I'm starting new project with FastAPI and I want to use it, so I can't wait for video about SQLmodel. One thing based on my experience I want to share with you. Argparse? Noo 😅 you should definitely try alternatives like click (from flask author) or typer (from fast API author). This is like comparing datetime to pendulum 😅
Write I am computational biologist, I write many CLI projects. I am a big fan of tqdm and rich. I also came across Textual, but haven't used it much yet
I regularly create cli applications for handling batch changes in cloud applications. It is just easier/faster to use the cli than to point and click through multiple pages on a UI to manipulate ymal files and things of such. My cli module of choice these days are Typer.
Great list! Thank you. Personally, I find Pandas to be indispensable for many problems I have to solve. It is a major reason why I now do a lot of my work in Python. I basically use it to replace eXcel.
Ice cream is likely a pun, or rather two of them: 1. Ice cream sounds like "I scream" which hints that it's a logger, though a bit louder. 2. It's abbreviation, ic is pronounced "i see" as in, ic(foo(bar)) "I see foo bar"
![[MV] A leap of faith...ความเชื่อใจครั้งสุดท้าย (From "Petrichor the series")](http://i.ytimg.com/vi/U1piZH2CNXA/mqdefault.jpg)
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
Hey, Arjan it's been a while now following your design guide videos. But the real problem is that, is it good to place more than one class in a single file, I have been coding in PHP where every class takes it's own file, the code is simple to refactor in that way. What are thoughts?
The naming of the ICECREAM module is a clever play on the abbreviation "IC", which phonetically mirrors "I See" in English. This choice is fitting for a Python debugging tool, as it elegantly underscores its purpose: to illuminate the inner mechanisms of your code, much like peering through a transparent layer in a typically opaque Python environment.
Also "I scream", maybe.
A CLI app is always a good idea when automating boring tasks at work like building reports or extracting data to manipulate. Its quick to develop and fast to execute.
Thanks Arjan ! A full video on SQLModel will be very appreciated..
x2
A deep dive video on SQLModel would be great. I like it. On my first pass at learning it, the foreign key constraints were a little confusing and I ended up refactoring some code once I fully understood it. But seeing a video might be a great way to know if there is a better way to define them. I was doing a light project with a SQLite-based database, but ultimately want to get to a project that uses PostgreSQL with geocoding.
If you did a video like that I would find it very interesting.
Your very informative and nicely composed videos helped me get a job in this tough market. Thank you very much, and keep up the good work!
woow congrats that's really asweome bro , mind if you mentor me in my journy ?
Wow! Thank you so much! I really appreciate it, and I wish you success with your new job! 🎉
Great video, thanks!
Here is a new entry: Ray is a nice framework with a Python API that facilitates code parallelization, whether it is on a single machine or within a designated cluster.
Side note for people who might be interested: while Ray offers a user friendly experience, there is a certain degree of syntax overhead, making it challenging to run code in a serial manner without code modifications. To that end, I have build and published a wrapping library (still a work in progress), “ray-ease”, that allows to switch from parallel to serial without requiring code modifications.
Since I've started to learn rust I found the Result type (railway oriented programming) an incredible programming mindset.
The basic idea is that many of the exceptions we throw are not really exceptions, but just different paths that our program can take, so we shouldn't treat them differently. If that idea is well integrated into the language, is very powerful and allows to prevent issues.
8:45 Result
@@captainwasabi zig language for the win.
Learning Rust ATM and I chuckled when Result was discussed, especially with the pattern matching. Should definitely be a language feature in the std library
@@captainwasabi So are for loops, so is all of structured programming. Both exceptions and result types have their uses. E.g. operator overloading is utterly unusable without exceptions, might as well not even try. Exceptions are also much more convenient (and performant) when errors happen several nested layers deep.
Also, result types are fundamentally unsuited to dynamic languages like python.
it beats my method of always catching exceptions but never bubblling them up and just return something more appropriately informational/checkable
I write quite a lot of CLI applications, and personally I find the docopt library really neat.
Awesome collection. Please create a complete video about SQL Model + FastAPI . Thanks
💯
I have a long video on sql model + fastapi on my channel, feel free to check it out
I love Seaborn! I build CLI for data processing from our data warehouse, primarily using PyODBC.
I really like click as an alternative for argparse. Much simpler to use and very powerful! Only downside is that it adds a dependency.
And even better than click: typer (based on click), developed by the same person as FastAPI and SQLModel :)
Thanks for the suggestion!
@@ArjanCodes Even better than Argparse and Click is Typer give that a try. Of course it's a layer on top of Click. Typer is made by the same person who made FastAPI.
Click is cool but Typer is also great. Fastapi, SQLModel and Typer are all made by the same author: @tiangolo (Sebastián Ramírez). He's a wizard with leveraging type annotations for a great developer experience!
In my experience click makes the code slightly less readable and more confusing. I feel argparse more pythonic.
I write CLIs all the time. I generally design everything like a library so it would be easy to attach any frontend to it, but CLIs are easy to crank out with libraries like Typer.
At work, I write a lot of custom tooling for things like: managing/requesting TLS certificates, DNS management for customer domains, and other oft-repeated day to day tasks. At home, I wrote an application to automate organizing my media collections (with remote metadata retrieval). I'm usually in the terminal,and automating GUIs is a pain, so I almost exclusively write CLIs for my libraries and leave the GUI or web service frontend for others to implement.
Fire is super convenient for simple CLIs!
I love Typer! Another great library from tiangolo
I adore icecream. Being able to run unit tests and get every variable in colour and very visible on the terminal with just a two letter function when debugging.
One of my go-to libraries for adding 12-factor structured logging to my code is structlog. Works equally well as a replacement or enhancement of the standard logging library, and extremely flexible and extensible.
what is 12-factor structured logging?
@@yash1152pretty sure they just mean "structured logging" (logging all the data in a format like JSON) in a "12 factor app" (which is an architectural pattern for containerized/cloud native applications).
hey @@MichaelONeillIrish ! ohkay, thanks a lot :)
I've recently started using structlog as a drop in for the std log library, and I can't go back. It has the features of rich + loguru in an easy to use interface!
@@Jackovasaur but wount that mean that its not lightweight to run?
We add CLI to almost all of our projects for a couple reasons: 1) it's a quick and easy way to test out what is the proper interface/API to your applications before you invest into a more complex API tooling and 2) our applications are mostly pipelines (orchestrated ML lifecycles), not always-up applications. CLI is just a very robust and controllable entrypoint for that type of applications.
Which PYPI project is this ?
Good to see a recommendation for argparse. Most of these modules are already used by me so I'm glad I'm on a similar track.
For the question on command line programs, we create command line programs as the work I do is mainly backend and we try to automate as much of our repeatable and tedious processes as we can. Vendor software provides some functionality and our tools just enhance that to make maintenance seamless and fill the gaps not covered by the vendor.
If you do any calculations that requires units I will always recommend 'units' from 'astropy'.
You can basically transform any unit to any unit (like nanometers to feets, if you need that for some reason) in an easy way. Pretty useful in equations where you are not sure if you have converted the units properly as well
FYI, 'pint' is a neat, standalone alternative to 'units' from the excellent 'astropy'. In a bit of testing I found 'unit' somewhat faster BUT either will add considerable overhead when doing computations using quantities with units. I slightly prefer 'pint' in usage. In any case, I recommend one use quantities for I/O function to validate inputs and to provide quantities with units as output, for doing dimensional analyses and validation through the compute chain during development and testing, but to only pass values from input quantities into production computations and then add units back to the outputs. The cool thing is that the computational chain (if implemented with the right abstractions!) should run either quantities or values without any changes.
I build command line interfaces all the time for data science / ETL utility scripts & processes, pulling data from APIs, roll-up aggregate jobs, interacting with cloud services, etc... argparse, tqdm, and pydantic are all super handy.
Thanks for the informative video... A video on SQLModel would be nice!
What about a video about integrating between a REST API and a SOAP API using Zeep (Getting the data from both APIs, manipulating and comparing the data between the two to check if you should make a post, update or delete request to the REST API)? I think it should be interesting...
It can probably just be 2 REST APIs as well...
For sure, because SQLModel has not finished docs yet and also it is not clear if this tool is production-ready like fastapi or pydantic
I make mostly command line tools. I like to use typer for argument parsing when it doesn't conflict with my requirements otherwise I use argparse. I'm surprised typer wasn't on your list.
Command line interfaces are very useful for astrophysics simulations (at least; I don't want to talk outside my experience), specifically when the physical model you're implementing tries to do as much as possible from first principles. In essence, the whole point of the code is the code itself, not the results; the results only serve as something to compare with observations to evaluate how accurate your physical model is. In this context a good command line interface --- especially when coupled with judicious use of shell scripts --- simultaneously abstracts the concept of a physical model, with (hyper)parameters etc, very naturally, and allows you to easily bring collaborators up to speed, since the code doesn't just document itself, it also explains itself through the story it tells via `tqdm` and friends.
More broadly: The closer you are to the bleeding edge of pure research, the more atomic your toolset is. This naturally pushes you into the shell, since cobbling together bespoke pipelines, with your own bits woven in, is much easier there. Hence command line interfaces being useful. (And, of course, a lot of your computing is going to happen on clusters, which often speak POSIX.)
Why use the shell? Why not use it as a library in Python?
@@iridium1118 To an extent it comes down to personal preference, although I stand by the abstraction argument. I prefer to split the different tasks involved between the shell and python because I feel at home in the shell, and prefer the toolset that it provides for file and task management over those available in python. Of course, these aren't mutually exclusive. I always implement the command line interface inside an if __name_ == "__main__" block, and render my code such that its various parts can be used interactively via ipython; i.e., such that it can be used as a library. (And for the record: I don't mean Jupyter notebooks. Browsers aren't for editing. If my code is usable from inside a Jupyter notebook, that's incidental.)
If you're contemplating creating a follow-up video, I highly recommend including a section on the Typed Argument Parser library. I've recently switched to using it instead of argparse, as it offers a more modern and type-safe alternative to Python's argparse library.
I'd be curious about that too. I have also heard click is great for creating command line apps. I have used it to deal with user prompt inputs and it made a complicated while loop much easier to read and user friendly in just a few lines.
First thing I thought of when I heard argparse was TAP! It's such a simplification over argparse, and also allows scripts meant for CLI to be more easily tested and/or used from code, since you can construct the arguments very easily. I've also seen it used alongside click, but I have less experience with that.
I had to use polars for one project due to Pandas being too slow. That project was an exception, and i am typically working on datasets where the speed of Pandas is not an issue. However, I still use polars when I can because I find the syntax so pleasant. I think its going to vary a lot between analysts (most’d rather use SQL than either library) but there’s a chance that for you, its best feature is not its speed but its syntax.
Pyspark also has similar syntax to Polars and tbh it's made using pandas a bit frustrating!
Not a library, but I wanted to mention this: Python Poetry saved me so many headaches at work the past 2 months. I have been been a single dev working on some new infrastructure as code stacks with TerraforCDK and was having a rough time with dependency management. It simplifies virtual envs, dependencies and packaing. Def worth checking it out!
I really like you to dive into SQLModel and combine it with a easy way to present the underlying data in some sort of templating solution. Thank you!
Contrasting it with SQLAlchemy would be great too!
It's not stable and documentation is not great. What about Tortoise ORM or Piccolo ORM?
Me too, is this similar tooling that is used by low code platforms to generate the database backend?
really nice list, thanks for sharing Arjan! i started using argparse recently in a data engineering context, to set up a main job executor and pass ETL jobs and params as arguments. between that and config libraries like omegaconf and hydra, it's completely changed my application design for the better.
Some really cool packages in here that I've never heard of!
Please do a video on SQLModel and how to do normal SQL operations such as groupby and aggregations. I tried using this in my code but I found it a lot slower than using Sqlalchemy. But I may have done it wrong. A video would really be amazing!
How does it compare to SQLAlchemy?
A comparison between SQLAlchemy and SQLmodel would be great
This is super helpful, especially pydantic and icecream. Thank you so much!
Glad it was helpful!
typer and pydantic-argparse are my favorite CLI parameters libraries 👌
Yes, I'd like a video about SQLModel 🙂
5:50 I typically build tools with dynamic menus that can show/hide options depending on circumstances. tqdm is one of libraries to show file transfer. For example I built file manager for my photography work that can automatically batch process newest photos into jpgs, save cropped frames and things like that.
As for the logging I use my own function to print and write to file with preset formatting. Sometimes I make CSVs out of this so custom method helps. For me it’s more abaut information gathering than pretty colors in console
I write CLIs mainly to fill gaps in automation. Cronjobs, Deployment or Backup scripts etc. I usually write as a CLI, rather then a simple script. This means more work upfront, but is usually worth it once someone needs to execute it manually for the first time, and someone always will.
I usually use click in combination with rich, so thanks for this list of libraries. I will check out argparse for smaller CLIs, since it will reduce the overhead, especially in regards to deploying the app.
I use click on a regular basis. For me it's the best argument parsing library when it comes to big CLI tools.
Result looks very interesting to me, coming from the functional programming world, I could really use a Result monad!
I use python for command line scripts - -replacing bash scripts. Command line tips are appreciated. Thanks and keep them coming.
Honestly happy to see Loguru on this list. Came across it like 2 years ago and much prefer it's simplicity over Pythons built-in logger WHILE still being able to combine both loggers AND maintain complexity as you need it. Brilliant package IMO. 🎉
Yes, I do regularly create commandline python apps, and I use argparse and rich :) The apps are all used exclusively in house in our company, and they are mostly wrappers for internal APIs.
Interested in SQLModel video and how it compares to SQLAlchemy.
In devops we use command line reporting and operation commands. Click and Rich packages are really awesome complements to what we do
Excellent video! I will definitely include some in my stack. I would like you to make a Sqlmodel video, I think it is an interesting option to consider
Thanks for the idea!
Thanks Arjan ,creating a complete video about SQLModel and FastAPI will be help full 💯
"I scream" (bad) -> "Ice cream" (nice). Debugging becomes nice.
in almost all of my projects: ic, pendulum, loguru, python-dotenv, rich. But i also learned some new ones! :)
- loguru can deal with multi-threading/processing. I love it at the first sight
We write software with command line interfaces all the time in scientific computing. I use argparse very often. Was glad to learn about tqdm, it looks very slick
tqdm is great - any iterable object can show its progress. I use it during scraping.
Yes, you betcha I use the command line! Specifically, I have dozens of users who use my Makefiles to git/smbclient/git-select/xmllint/Bamboo configure/setup/deploy our developer environments. The Makefile figures out the host OS and installs OS dependencies using one of yum/apt/yast. It'll also use the pipenv files and specific Python selection to create bash shell launcher scripts.
I'm particularly interested in rich-cli which give me a nice interface to either prompt a user for a branch or supply them documentation.
Great list, thank you Arjan! It would be great to see comparison of SQL Model and Django ORM though.
...and sqlalchemy
These aren’t really comparable since Django ORM can’t be used outside of Django
@@jeffgruenbaum Hey, where is the spoiler alert - this is difference one! :)
@@jeffgruenbaum I'm not sure what you mean by outside of Django? It can be used without using a web server and you can access and manage your db through Django ORM in your own scripts and applications.
I use 'click' when i want to build CLIs - I love the ability to unit-test my code and commands easily, and also test the actual command line itself - and I like the decorator/dispatch model.
I am a data science practitioner and Polars, FastAPI is for sure great. Great list!
Very useful. Thanks for this. Ditto on liking a separate video on SQModel.😀
Excellent overview of libraries. This video has the batteries included.
Thank you for the kind words, haha! Glad you enjoyed the video :)
TH-cam algorithm brought me here, and I found very some useful and interesting libraries unknown to me. Your video is really informative. Thank you!
Thank you for watching! I'm glad it was helpful.
Thanks for compiling this list of 15 useful Python libraries! It would have been even more beneficial if you could have included some examples of how each library can be used. That would help beginners and even experienced developers grasp their functionality more effectively. Nonetheless, I appreciate the effort you put into the video. Keep up the good work! 👍
Awesome libs - if only this popped up for me a few weeks ago, would have been useful for my current project lol.
I just found the abseil library for python and so far I really like it. I haven't completely understand all of the build abilities it provides by using bazel, but for building command line Interfaces I like it way more then argparse. You can define arguments as global flags und use or modify them anywhere without having to pass them down accordingly. Found this in a Google Project code and it instantly blew my mind
Python's built ins now handle datetimes with timezones really elegantly with the default Datetime and ZoneInfo modules. Might be worth doing a video on those, a lot of people don't realize ZoneInfo is in the standard library.
Thanks, Arjan! This sort of "speed dating" video is usually lame,but you did the extra work of putting in the index and the links, so it's very useful. I would suggest one improvement, though: add a one-line description (just a phrase, like "Does error logging and adds debugging capabilities') to the libraries, because the names often don't give even a tiny clue as to what they're about.
You asked about using CLI. I go back to the days of punched cards, and spent a lot of my career doing embedded systems, so CLI is my "native environment". For me, GUIs are what you use to make your CLI results look pretty ;-)
When I took my first run at learning Python several years ago, I tried argparse, but found it confusing because it required a fair amount of understanding of Python,especially what you can do with various data types. I'm sure I'd see it differently now, especially if I were writing code for others. But I have decades of experience with getopt, both as a user and a programmer, so I use it in Python, too.
invoke is my go-to for building CLI tools. Hierarchical configs, zero argument parsing (!!), command chaining, self-documenting. Its so simple I build a little CLI for every project as executable documentation for processes, tasks, etc. Can't understand why invoke is not in every Python developer's toolbox. And fabric adds ssh on top so you can run invoke tasks on remote hosts. Maybe if Arjan did a video on invoke and fabric more people could enjoy the awesome.
First thank you for sharing your knowledge,
a video that includes both SQLmodel and FastAPI would be amazing!
nice selection! My favorite library is more-itertools, using it literally in any module
Great job with the libraries Arjan!🚀 If anyone’s looking for more Python videos, we’ve released how-to Loguru logging, task scheduling, and more to help the community too 💪
Smashing video. Some of these I have never heard about before and will now take a look. Thanks.
Polars is really a good tip and the user manual is written very well!
Right after i see tqdm, I already know this video worth to watch
icecream may be a word play on ICE - In Circuit Emulator, a design tool from the past (nowadays In Circuit Debuggers are the norm, because transparent hardware emulation of a fast chip is hard or impossible) for embedded systems development. ICE's were expensive, a luxury way to develop and debug.
Love to learn about new libraries, you always find something new and exciting. Thank you Arjan!
14:37 I like: plotly, streamlit, pint
thank you for video
I might as well also mention Maturin, which makes working with mixed Python/Rust codebases a breeze. I'm currently working on a general-use project template for such projects using it.
Thank you for revealing Pendulum. It helps me in my work.
I'm happy to hear it was helpful!
Very useful resource! I would definitely would like a video about Fast API & SQL Model. Thanks for all the great work.
"Ice cream" refers to SoftICE debugger, one of the most popular DOS/Windows debugging tools of that era.
I have never heard that monadic exception handling is called railway oriented programming in some languages, thank you for enlightening me.
I like your video a lot, but I would argue that the click package is simpler and easier to use than argparse :)
Hey Arjan, Yes please! Please do a video about SQL Model
Just discovered tqdm and used it in a recent project. Super simple!
Thank you, very useful information for me. I use the terminal almost always, as a Linux user, I create an alias for the software I did and it's enough for me. But I must say I m over 60s and I just study and work with Python for my own fun, also I m very used to 80/90's terminal environment, so it's way better for me than using a laggy GUI.
Very nice selection, many new to me. I actually stopped using argparse a while ago and prefer Fire instead.
Wow, very helpful, thanks.
About CLI I like to play with pyTermTk.
"Icecream" because it sounds like "I scream" and a scream is the typical sound a developer makes while debugging.
For quick deployment, O would like to mention gradio: few lines of code and a user without programming experience is able to interact with a Machine Learning model
@ArjanCodes - I think that it would be very interesting to learn what you experience in other developers as far as weaknesses go. It is likely that a weakness you repeatedly experience is prevalent in other coders as well. Knowing these weaknesses would help other developers potentially learn from you and avoid the same issues.
For PDF's, PyPDF is a good start, but PDFMiner (for parsing) and ReportLab (for building) are much more powerful libraries.
Also, if your .env files have anything other than just strings in them, the library you want to use is decouple
Python command-line programs is great when working with docker containers, and when working with containers that process jobs and need input parameters.
“Result” is most often the one thing from rust that I wish had with me when writing code in just about any other language.
Hi Arjan. This video was really cool 👍
I'm happy I know quite a lot of those libraries 😀
Great you mentioned SQLmodel. I'm starting new project with FastAPI and I want to use it, so I can't wait for video about SQLmodel.
One thing based on my experience I want to share with you. Argparse? Noo 😅 you should definitely try alternatives like click (from flask author) or typer (from fast API author). This is like comparing datetime to pendulum 😅
Write I am computational biologist, I write many CLI projects. I am a big fan of tqdm and rich. I also came across Textual, but haven't used it much yet
Very nice list, indeed. I would include “typer” in the list, I prefer it over “argparse”
If we are talking about command line apps, it is good to mention Typer. I like this library.
tqdm is very well used for AI training to keep progress / benchmark on how fast and well such model is doing
Sweet - picked up a few from the list for myself.
Glad to hear it was helpful!
Maybe late to the party but would like to see a full video on SQL model. Thanks for your videos.
at 1:22 - "Being a human I really appreciate it " lol
I regularly create cli applications for handling batch changes in cloud applications. It is just easier/faster to use the cli than to point and click through multiple pages on a UI to manipulate ymal files and things of such. My cli module of choice these days are Typer.
Only just discovered your channel. Great content keep it up!
Welcome aboard, Dominik! Glad to have you here.
Hi Arjan! Great video! Any libraries or frameworks you would recommend for Python desktop application development?
Yes, would be great a video about SQLModel
Great list! Thank you. Personally, I find Pandas to be indispensable for many problems I have to solve. It is a major reason why I now do a lot of my work in Python. I basically use it to replace eXcel.
Polars, which was mentioned in this list, is like a more modern, faster, better version of Pandas. 😊
Ice cream is likely a pun, or rather two of them:
1. Ice cream sounds like "I scream" which hints that it's a logger, though a bit louder.
2. It's abbreviation, ic is pronounced "i see" as in, ic(foo(bar)) "I see foo bar"