Why is Python so Slow?
ฝัง
- เผยแพร่เมื่อ 24 มิ.ย. 2024
- So why is python such a slow programming language and how can we speed it up? In this video I'll be discussing the slow speed of the python language and why languages like Java, C and C++ can perform between 10x-200x faster.
⭐️ Thanks to Kite for sponsoring this video! Download the best AI automcolplete for python programming for free: kite.com/download/?...
◾◾◾◾◾
💻 Enroll in The Fundamentals of Programming w/ Python
tech-with-tim.teachable.com/p...
📸 Instagram: / tech_with_tim
🌎 Website techwithtim.net
📱 Twitter: / techwithtimm
⭐ Discord: / discord
📝 LinkedIn: / tim-rusci. .
📂 GitHub: github.com/techwithtim
🔊 Podcast: anchor.fm/tech-with-tim
💵 One-Time Donations: www.paypal.com/donate/?token=...
💰 Patreon: / techwithtim
◾◾◾◾◾◾
⚡ Please leave a LIKE and SUBSCRIBE for more content! ⚡
Tags:
- Tech With Tim
- Python Tutorials
- Why is Python so Slow
- Is Python Slow
- Python Speed
#Python #IsPythonToSlow
![[Watch Party] APL 2024 Bacon Time Vs Hong Kong Attitude ชนะเข้ารอบ](http://i.ytimg.com/vi/ikUfW81sDaw/mqdefault.jpg)
Hey guys just wanted to make clear that I believe I may have made some mistakes in the video! Some people that are clearly more educated on this specific topic and have left some polite comments to clear up any confusion. I'd like to apologize if I've misinformed anyone. I believe a majority of what I've said is correct but there may be have been a *more* correct answer to the question. It seems like the dynamic typing is an issue regarding speed but the more influential issue comes from the lack of a JIT compiler by default.
Golang is quite fast and it can be dinamically typed too using the assignment operator :=.
Golang can be both interpreted or compiled!
@@hedleypanama The := assignment operator does not mean you are using dynamic typing, it means that you are letting Golang infer the type of the variable based on the data type that is on the right hand side of the operator. The variables will still not be allowed to have their data types changed however unless explicitly converted to a different data type. And Golang is a compiled language it is not both interpreted or compiled.
Tech With Tim Great video. One correction Java is not low level progamming. en.m.wikipedia.org/wiki/Low-level_programming_language
That mad guy in the corner in the of the room bashing keyboard laughs crazily as he types C and other assembly languages.
.
(cuz he a hacker x)
If I compile a python program and turn it to an exe.....will it run faster?
Python in real life is slow also. Compared with other snakes.
lol i hate you
Thus the brilliant name.
Best comment ever.
😂😂😂
thanks, laughed
Python was not made to be fast...but to make developers fast...
Ik!
Give that man a cookie 🍪
@@Lioneltunisianorevo But let that man know if you're in Europe!
@@zyrohnmng Tunisia :p but why ?
As is java.. Java is easier to program and faster to program, but far slower to run than C/C++.. It's the same for C# it's faster to code, but slower at runtime. That's why C++ is still a thing in 2019, because no other programming language can be as optimize as C/C++ (except the languages that also compile directly in assembly). But C/C++ is slower to code, way slower to code.. You need deep knowleadge of how your computer and language works to make it efficient at runtime. That's why Java and C# are everywhere and C++ is only used for application that required a lot of optimisation.
The problem is that a scripting language has become so popular that we're holding it upto the expectations of compiled languages
I mean is there much of a difference between a compiled language and a shell language? Python is still eventually run in assembly just like C or Java
@@platcrab4890 There's a difference between running interpreted bytecode (Python, Java, etc.) and actual machine code (C/C++). Also, please, do not confuse Machine Code (the thing your CPU runs) and Assembly (the thing you used to write programs in)
Edit: Also, Java certainly does *not* compile into machine code.
The Zucc has spoken
@@TricksterRad how does Java run if not turned into assembly?
@@MJFAN666 Java gets turned into JVM bytecode. When you run a Java app, JVM executes that bytecode.
If java were compiled into machine code, you wouldn't have to install JRE on every computer where you want to run java programs.
The benefits of static typing are not limited to execution speed. It also results in a code base which is considerably easier to refactor years down the line.
how does it make it easier to refactor?
@@Jarrod_C Because there's more meta data embedded into the source code.
It also catches lots of mistakes that can be a bitch to find, like a spelling error in a variable name that creates a new variable at run time instead of an error message at compile time.
It's "horses for courses": use something like Perl or Python for things that run interactively and don't hurt anyone if they fail, but a strict language for programming when failure is not an option.
This hasn't been my experience. Static typing always seems to involve a lot more thinking to refactor, plus a lot more writing.
@@rantalbott6963 that depends on the language design. But then, unless you're using notepad to write code, you're not gonna have many issues with mistyping variables. I've never had any difficult issues from it either way. That will result in some null variable error a few lines down, which will be enough to tell what's wrong pretty easily. (My experience is mostly with Lua, though, I haven't used that much python in the last 4-5 years.)
However, Python does carry some stupid Murphy's Law traps. For example: *x = 10* always defaults to a local *new variable,* and since it's so easy to forget to preface a function with *global x,* you can (and will) find yourself wasting hours on a weird problem caused by that. In Lua this is never a problem; as long as you know to avoid using globals, you're always in direct control of scope.
I graduated with a comp sci degree in 1974. I have programmed in assembler (several different), fortran, basic, C, mumps, cobol, pl/1, apl, lisp, java, C++, javascript, can't even remember them all. I met Donald Knuth and discussed structured programming with him. I was onto OO after reading the issue of Byte magazine that introduced smalltalk to the masses. I met Charles Moore, inventor of Forth, and worked with Jules of Jovial fame (well, maybe not fame). I supported the C++ run time library, cfront, and know LEX/YACC and how the internals of compilers and run times work.
For write only programs, python and similar are quick. If you are writing a corporate product with many parts and many people and care about performance, pleae give me static data typing. C++ and stl give you much of the goodness of python lists. I claim I can program in C++ or java almost as fast as python.
The speed of coding in python is due to the many magical diddles in the language, but that is a two edged sword. I say C++/Java is a little more verbose, but typing speed is not really an issue, and the quick writing is an inevitable tradeoff with clarity, at least to those who are not doing 10 hours a day with python. I would rather see spelled out code than a list comprehension, for instance.
Also, Java and C++ are pretty much what they started out as. Python is continually changing, multiple versions are very different, meaning learning and update time. Makes docs a nightmare of info about how it used to be and how it is now. Much worse than C++ and Java,
Also also, I have been using Kivy, and both kivy and python have documentation that was done quickly and often needs 2 or 3 times as many words, way too terse, seems like it never really does a top down discussion of things. Corporate libraries and languages I used had complete and relatively static documentation in known places. Python is all over the place.
There are tradeoffs, like clarity for brevity. As a silicon valley software architect, for quality, dependability, and clarity you have to pay a price, and I don't want to fly in an airplane whose software has dynamic typing. I know there are those who can read (and code in corporate settings gets read way more than written) python quickly, but for many, slower and less reliably correct.
If you are writing throwaway code, amusing yourself, or ???, okay python. For large projects with many modules, versions, programmers, maintainers, performance demands, etc, I would definitely choose C++ or Java.
Lastly, if you are writing software you expect to maintain and enhance over many years, at some point, you are going to have giant headaches over versions, do you stay with the 10 year old version, or massive convert? No good answer.
jt95124 What is your observation about ADA. Is it still active? Will you choose ADA over C++/C why? Why didn’t ADA fly?
@@sumodmadhavan ada was trying to be c++ but with async multithreading built in. It was developed and pushed by Dept of defense as solution to their issues. C++ does the same thing and grew from the C culture, so had community and momentum.
@@sumodmadhavan languages succeed by large community, Adam has none, so no open source, no tools, no programmers
I'm studying C++ and Python languages right now in my univeristy, which language would you recommend me to develop also and should I?
What about Fortran?
I've casually tried to learn coding many times, python has been the only language I have understood. Seems like a good place to start
Can I ask which languages you tried to learn?
@@ajricherson1099 brainfuck. Joke. It was C or one of its variants
@@DrumRoody You are really comparing two ends of the spectrum then. C is pretty low level language, and Python is one of the higher level programming languages. Just something to think about.
If you do want to learn another language, I would recommend something like Java, C#, or perhaps JS if youre interested in web dev
@@DrumRoody I use C, C++, & Rust. I definitely find them much more expressive & powerful than python. Also. Brainfuck can be used as a test output for compilers. Brainfuck is essentially simplified assembly language.
Lincoln Sand this has to be a joke brainfuck was created to fuck ur brain it’s not useful for anything
REEEE
Kinda wish you mentioned numpy for when you were referring to just raw speed for computing math. Numpy is much much more optimized than the base interpreter and is usually the answer to this type of issue.
Numpy is the reason why Python can be used for so many data analytics and machine learning applications.
Harshit Gupta Agreed. Machine Learning is definitely Pythons sweet spot. I love Java, but none of Java’s data structures comes close to pandas and data frames. Those alone makes stats and ML doable without a Ph.D.
Numpy is fast, I wonder what's the reason behind it omegalul.
Could it be, because, *ehem*, heavy parts of it are coded in coded in C?
@@ShadowaOsu also the most the machine learning bits in python are just wrappers around c/c++ code
@@maxclifford937 yea exactly
> Could it be, because, *ehem*, heavy parts of it are coded in coded in C?
Yes, and so it would have been a great example as an extension someone has already written.
Interpreted VS compiled
Python is still compiled! Just differently than other languages.
@@TechWithTim only when you use bytecode right?
Python is interpreted when you run a .py script
it’s compiled to .pyc then *interpreted as byte code.
@@tobychadderton4209 Python is compiled before running into __pycache__ directories.
But isn't Java interpreted as well?
Absolutely agree, every variable in python is by default an Object Class; in Java you can define variables to be primitive values, which helps it skip python's verification of datatype at runtime. Multithreading is also difficult in python while you can instruct Java to do it specifically when needed; my favourite language is still python for its simplicity and much friendlier user input structure in coding, so usually what I do is write up the code in python to run it, and if I need to reduce runtime for my software, I try to rewrite it in Java and hopefully achieve the same if not similar results.
"I'm learning as well."
Everybody likes a modest person.
Speed is a key factor to me.
That's why I chose Assembly.
what?
Tf
I'm assuming you're joking, but unless you're trying to run your program on a computer from 1989, you probably don't need to go _that_ low level. C and C++ should be plenty fast enough.
@@gavinthecrafter your assumption was right
Hats off
bro if you can make a video on writing C extensions for python, that would be really helpful. Great vid btw, also what are your thoughts regarding Go ?
I mostly write in java, but python is spread everywhere. Learning python, makes you able to make webpages and apps, able to perform data science and machine learning, able to make a stand alone app for desktop. I think it is worth learning python.
It's not just the case that Python uses dynamic typing but because its interpreter uses a stack-based virtual machine to execute python code instead of a register-based virtual machine like Lua uses and Lua is a very quick little scripting language.
Very informative and important at the same time, thx!
Dynamic typing isn't the biggest issue. Lacking a sufficiently powerful JIT is. JS and LuaJIT are bith dynamically typed and are orders of magnitude faster than Python. There are also other strategies like better opcode handling like in PHP's case.
I was thinking this exact thing while I was watching this video.
I mean, python does spend a good 90%+ of it's time doing type checking. A better jit compiler could help, but if you really need speed in python, replacing critical parts with C or Cython will give far superior performance over any jit.
I remember that Graalvm can run python3 and boasts great performance. Haven't got a load to try that but maybe a solution?
No. It's part of the problem. Dynamically typed languages are slow, even auto in C++ is slow. Python has a lot of things that work together to make it slow. If you care about real-time don't use Python.
@@chudchadanstud how is auto slow in c++ if the type is evaluated at compile time?
in my experience multiprocessing is simple enough if you set it up as here's your data, here's your code, come back to me when you're done. Setting it up that way is a lot easier for compute applications though
I am not an expert on this topic but I usually do heavy calculations in python and there's a module called Numba which has most of numpy's functions built in and it allows you to run some functions in python much faster, and even use parallel computation. It has allowed me to speed up my code a lot.
Informative as always thank you!
Thanks for explaining it in a simple way...👍
Keep up the great work, man!
Videos like these seem simple, but in the long run,
it will show case how much of a professional and teacher you are.
:D
Such a great explanation Tim!
Thank you for Sharing, Tim!
You can speed up Python code quite substancially by using PyPy or Cython. Or by using LRE cache dictionary, that function doesnt have to re-calculate all over again in for/while loop
I use Python a lot in my job, thanks for your explanation on why Python is slower than lower-level languages like Java and C.
super clear explanation and answers some of my problems I previously thought of
Good video, Tim! As others have pointed out, nits can be picked over some wording choices, but I like the spirit of what you intended to do here. I have used many computer languages over the years: Basic (several flavors), Fortran, C, C++, Java, Assembly (Intel & Motorola), DOS, BASH, VMS, REXX, Matlab, RT-Works, ObjecTime, Avenue, and now Python. I am not a programming expert, or even a professional programmer, but my coding experience is fairly broad. The design goals of Python's creators focused on issues other than speed. They were more interested in usability and efficiency, as well as some very deep computer science crap ... I mean operability. For anyone who thinks Python was intended to be user-friendly, just try reading the official docs. However, the flexibility and robustness of this language are impressive. Still, it gives you plenty of rope to hang yourself; you can easily write poor code if you lack sufficient skill or discipline. That said, I think Python is a good first language for those who are willing to explore what it offers. It gives you plenty of tools to write good code, and perhaps more important, to develop good programming habits. A language like Visual Basic may be easier to learn, but the skills you can develop are limited, and you are likely to pick up some ugly habits. For more advanced programmers, Python offers a dynamic coding environment that allows you to achieve goals that are difficult to match in other languages. If you are willing to dig deep, you can write intentionally ugly code that will bypass typical limitations found elsewhere. You should not expect Python to perform well on code segments that involve serious number crunching or manual loop structures. But, there are advanced mechanisms available (in the base language) to help improve speed, such as ranges, iterators, maps, comprehensions, closures, etc. Like the 'sorted' built-in function, these code patterns leverage pre-optimized execution, behind the scenes. Without regard to the language/compiler used to build a particular Python implementation, these mechanisms represent a large manpower investment by dedicated people! If you learn how to use them well, they will help you write faster code. And, I think writing C (or Fortran) extensions is quite viable for extreme number crunching. If you like Java, use Jython in place of C-Python, and write Java extensions, instead. Just remember that powerful Python libraries, such as NumPy, are available, as well. If you need to take Fourier Transforms, for example, you might find that NumPy is faster than using a Java routine that you found on the Web -- it depends on array length, precision, etc. Ultimately, I agree with others that no one language will serve all your programming needs.
top notch comment c:
Well very nice comment but compressions don't make your code faster just tidier!
Cheers
I have made essentially a curse word check in Python that checks for curses being used in a word, is simply a curse word, or composed of multiple and/or repeating curse words as well as having different rules for strings after a certain length. It can do a maximum of very roughly over 2 million checks before determining if a curse word is present in less than 1 second. I have recreated the Python code in C to discover that the speed is essentially the same and could not improve the speed in C any further.
In general, Python is slower; however, when creating solutions and algorithms only possible in Python, it quickly becomes one of the fastest.
👏
@@uguree Perhaps. Cheers!
Technically speaking, a python program would be slower than a similar c/c++ program by a constant factor, say 100X, not exponentially. Exponentials blow up really quickly
Actually, no. There cannot exist a functionally-desribed correlation between Python and C speed. Even if you write 100 pairs of absolutely semantically identical pieces of code in Python and C that cover different aspects of programming and measure the execution speed difference for each pair. You will not be able to drive any meaningfull corellation between those two languages outside of Python being generally much slower. Why? For many reasons. One of them is compiler optimisations for C. For example, if you write a simple cycle that adds 5 to variable y 1000 times, in C this code will be compiled to just "y = y + 5000", while Python will execute all 1000 loop iterations effectively executing at least 1000 slower, but if there were only 10 loop iterations then the code would be only 10 times slower. However these 1000 times and 10 time are also not representative, since integers are absolutely different things in those languages and different operations might(will) have different speed ratios.
There are many other reasons, but I think you got an idea)
Actually in same cases it can be far more than exponentially faster than Python.
Python has no contextually knowledge of types and therefore of optimization.
I have optimized a piece of Python code to run 100+ times faster by changing I % 2== 0 into I & 1 == 1 and changing division of multiples of two by shifting right!
These are far more than exponentially faster situations, because a division on an x64 takes 45-93 cycles a shift and bit wise operation 1 cycle!
Changing the run speed of that back batch processor from running 6-7 minutes to 33 seconds. In total since these operations were the crippling factor.
A compiler does these sorts of optimizations for you because it has context. And when I tell a compiler to optimize for speed, it can (and will) un roll loops avoiding many of 100s or even 1000s compare and branch statements. Having a million of the instruction instructions in a row compared to 1 or those and a million compared and branch shorts saved at least 8 cycles per time so 8.000.000 cycles.
But of course every problem is unique so it would not scale quadratically on all factors but most often a C/C++ at least teams 4 to 100 times as fast! And I some cases but 1000s of times.
This is why I advocate that every programmer at least have 1 semester of experience in assembly. Because then you know to avoid those deadly pitfalls.
The main speed issue comes from the lack of a jit compiler. Pypy includes one, and, especially with long running scripts, can see as much as 6 times the performance increase. (note that it can be slower at times, especially with short running scripts)
Great work Tim!! Keep it up 🌷🌹
Out of curiosity, does pre-stating type of variable (varname: int, for example) make Python code more effective?
And, one more question, would it be possible for language to use both types of variables, so you decide whether you want this part of code be more effective, and the other more dynamic?
I have the idea of creating my own language mixing Python simpleness with Java/C effectiveness, but as I'm currently too newbie for it, I'm curious whether this is even possible.
7:07 "whenever you do this you need to shutup"
I was looking for this comment
lmao that got me
thumbnail is misleading, Java isn't 3rd fastest language by any means.
There are faster/more efficient languages, usually those which get compiled into native code, not into vm code.
It’s also misinformative as well. Python is an interpreter while C,C++ and java are all compilers, which provides a massive difference in performance. It’s why I dislike these comparison videos, as it’s comparing different types of how a language is run on something like speed. Compilers are usually faster than interpreters.
Even C isnt the fastest language, assembly and few other languages are much faster than C
@@atomfusion231 Java compiles to bytecode, not to machine code. This means at runtime there is still a interpretation.
@@programmer4047 Since both Assembly and C do compile to machine code, you can't simply say one language is faster than the other. It depends on optimization and the quality of the code.
@@programmer4047 I have to disagree with you. C compilers are very optimized, so the generated machine code is often more efficient than if you would write assembly by hand.
That made sense, thanks!
Tim, what do you think about a language like Julia which is easy to write and at the same time has fast execution?
UNIX: Everything's a file
Python: Everything's an object, even when it doesn't need to be with unnecessarily high amount of levels in our hierarchies
Smalltalk: Everything's an object, we just need reserved protocols and pointers to support our speedy tape measure despite the fact that we don't have an easy way to deploy our code
C: Everything is a pointer, sometimes to functions, so be wary or else you get a segmentation fault for free
In short, C is far superior to anything when you implement and use smart pointers.
@@__jan What is a smart pointer in C?
@@__jan Nothing exists all that pointers relies in memory and usage in typecasting in every single type that you see in pointers just goes through a specific memory address then you can disassemble in x86 where its going to head to in the offset relied in memory when it points you can make it point to another type
Smart Pointers only exist in C++, not in C, as smart pointers heavily rely on destructor methods that are called after an object has gone out of scope and is to be deleted. C doesn't have objects, hence it doesn't have destructors either.
Vivek Yadav worst case make malloc/calloc/realloc/free wrappers maintaining a global array of void pointers thats freed via atexit. It’s janky but works if watch for abort
How to make Java fast - use JIT compiler with "Server" preset
How to make Python fast - use JIT compiler (PyPy) or compile your libraries to C and import them in your project (CPython). Second is more difficult and clashes with Python idea to be elegant and simple.
How to make Lua fast -
Also use JIT compiler.
@techwithtim, does octave language at same case(madlab)? i don't take research about metleb
Can u make a video on how to use C as an extension for python? Love your video as always.
I am not sure about the statement: the python is slow because its dynamic. Why? Lua.
I am sure there is more to that. I am not an expert in python, but I think it's close enough.
Any language requires special API to work with multi-threading, such as locks for shared objects. However certain compilers for statically typed languages might be advanced enough to do it automatically for some algorithms. It's just nobody bothered to implement this in Python so you have to do it all manually...
Static types takes much longer developing? Jesus what a blanket statement. Last time I checked, me writing "int" or "double" was not the major challenge in my coding. If you're doing anything significant you're spending most of your time just thinking.
Indeed
With regards to making "native python" faster, Numba is amazing. It's effectively just a decorator over native python and the acceleration on numerical tasks is comparable to C/Fortran with the simplicity of python.
It also can interact with numpy for those wondering but at that point just use cython and import C headers directly since you're already deviating so much from native
Very important answer! You don't need to write C to make python faster. Let numba do it for you. It's converting your python code to C quite efficiently. You can keep your core code in python with all the convenience python is providing and just decorate the number crunching function that gives you a headache with numba. If it includes a prallelizable for loop you can just use numba's "prange" to even parallelize code.
This is the best comment on this video. There are reasons for why python was developed the way it was and why it is so popular. It feel like this video does not do the topic justice, and misrepresents the power of python. The better way to think of python is a general purpose scripting language for calling optimized libraries. If you are using it any other way you will be disappointed.
That's right, but you have to remember that you can still define the type of variables in Python, and I actually use it a lot, for example:
var: int
That way Python know to inform you that you can't use string functions and methods on this variable, while you CAN use int functions and methods, or that you can or cannot send a certain parameter to a function according to its type.
Can you give another example?
Finally, Thank you 🙏🏽
Does anyone know if when I use static type checking in python, do some of the slowness issues he mentioned go away?
Python Static Types: medium.com/@ageitgey/learn-how-to-use-static-type-checking-in-python-3-6-in-10-minutes-12c86d72677b
Because static keyword locks the object to the memory so when its context or class is destroyed, it's done & you cannot use it w/o passing the compiler back to the line of the declaration of that variable or object ....so it helps in terms of performance and memory but you cannot use this in dynamic cases because if you destroy the thread of class or context having that variable & use that variable in a context you will get a NullPointerException or IllegalStateException & by the way static makes the object creatable only at the runtime of the context only one-time after that you cannot recreate it ...you will have to reRun the context or the application
important to note: For most io bound loads (like running a web server), having n threads does not directly translate to n times performance. Also, single threaded asyncio based webservers frameworks like uvicorn/ node are often surprisingly fast.
If you are IO bound 95% of the time and then you get a sudden spike in things to do that will still translate to a sudden lag spike once in a while and thus bad user experience.
To this one should add that latencies add, even if the work your program does is 2x faster than the IO that still represents 33% of the latency time in your program, wich is quite significant.
"IO slow therefore performance doesn't matter" is a terrible excuse 90% of the time, if you are doing something utterly trivial it might be a good excuse but as long as your program is actually doing something half interesting performance is important, perhaps not triple A video game kind of performance but in some way or form you should care about performance.
Writing main code in Python and importing Fortran/C algorithms is the way to go for me, most if the time at least. For most people speed really does not matter - if your code is written poorly in C there is no benefit to using it.
I had a question does you're pc performance create a difference to the python runtime?
Tim!!!! I can't Install Kite!!! It says Tensorflow requires CPU instruction set AVX which is not mine.. :-(
Love the way you explain things 💞
Really important video for python interview question, nicely explained man 👍
Really interesting man thanks
Could we just as well use C# or C++ to import as an extension? Or does it have to be C specifically?
Python is a great PL for certain purposes. Imagine you need to walk to your barbershop one block away from home. Well you could . time your speed ,call a helicopter for that matter, and wait an hour for it to get to your house. Then find a place to land and so on.. Or you could simply use your Dynamic legs and put them into work.
женя except when speed is necessary
@@amirmousawi9156 Well, he specifically gave a case when you don't. So what's your point?
I dont need to. It is obvious that python runtime is slow compare to languages such as c and c++.
@@matiasrisso5917 I dont need to. It is obvious that python runtime is slow compare to languages such as c and c++. So when speed is important python is not a good choice.
stupid logic 😂
How about cython?
Or PyPy
Nice explaination. 👍🔥
There are also other ways to make Ur python code fast:
JIT compilation. Because JIT compilation skips the whole process that goes on in the PVM and compiles code straight with an LLVM, it executes just as fast as C, or slightly slower, or in some cases (due to caching) faster.
Another way is with Cython, which is a way of compiling python script to C++ code, for this you would need Ur python skills and some extra stuff from static typing to be able to write .pyx files (different from Ur regular .py files) you can achieve C++ speed with this as well. There's also another version called Jython (compiling python code to java)
PS: I've tried all these out (except the Jython). Tho I don't code java, I do code C++.
Concurrency and multiprocessing in python is actually straight forward, people only tend to use them where they don't necessarily apply (just saying) modules like asyncio can enable you convert a single thread python function into a coroutine simply by adding the function in the asyncio event loop or running loop. Also for anyone interested in performing multiprocessing in python I would advise you familiarise urself with mesh reduction.
Also, most python modules / libraries / frameworks have key parts written in C or C++ or both...this is why one of the most computationally expensive tasks (machine learning) is done with python.
And then, with the advancements in computer processors these days, the speed difference in alot of cases cannot even be perceived by humans (as you've already insinuated)
I have a project in mind and I need some advice, will you be able to assist me?
So basically: how you make python faster is just using c. Glad we can agree on that
If I have to write code in C to optimize, then why would I import it in Python and then run it. I can just run it as a C program with the C compiler.
Tell that to tensorflow
If you already have a complex program, but one function is very slow. You can outsource that to the c compiler to optimize. This way you can keep the simplicity and elegance of python code while providing speed on complex requests.
Python is a composing language, a front-end language for gluing together modules like you are composing a solution. That's where python shines.
Great content here. Thx. It think py gained popularity from the Maker folks - stem, rpi, arduino, iot... etc. And to make things worse, the desire for intermediate understanding of deep learning. Think scikit and Tensorflow. Which is crazy since that is mostly linear algebra, matrix math - and intensive list handling. But py makes the approach easy to handle especially for type conversions and containers. So the simplicity of py made coding it so attractive that folks got spoiled with it. Pre-existing cookie-cutter copy-past code and libs for interfacing with hardware and server services made it super easy. See github for copious examples. Attempting to move into C from that background will prove very frustrating. The whole idea that "I have to allocate and free memory for a variable?" mentality is just the beginning of it. And to be honest, some introductions in newer C++ standards are re-creating that lax attitude. Smart, self-maintaining classes like deque, array, and vector remove the responsibility from the programmer to know the actual algorithm/implementation by hiding it in the so-loved obscurity of object abstraction. Sometimes, I think, over encapsulation/abstraction will end up shooting you in the foot. It might just be that you could double the speed of a code section by doing it yourself. Class and template expectations are often large in the final design. Often simple is far better.
Hi Tim,
Say i write a code in kivy. Its slow in python. After i compile it to android will it speed up?
Thanks
Then why is JavaScript so incredibly fast (especially comparing to Python) even though it is also dynamically typed?
I'm not counter-argumenting. Just trying to understand...
pretty sure its all because JavaScript compiles the source code before the program starts, while Python does it one by one line
Same for Lua. It's also dynamically typed but runs just as fast as a C program according to Linux's time command
JavaScript isn't a real language IMHO 🤣
"python in an extremely slow programming language", you're confusing language with implementation there. In this case you're talking about CPython, the reference implementation of Python
CPython is not python. It's C. You're just converting your code to C and compiling.
Holy cow! The first time I wanted only the ad of the begging of the video. I wanna try Kite
I'm currently taking CS50x an online course from harvard, the course progresses with scratch, c, python, sql then you choose tracks web android etc... There was a problem here where we were to create a spellchecker program in C that was so frustrating to make if you're new to c and data structures, then the next lecture we were shown how to implement that same program in python, and damn it was mesmerizing how little code was used, but also we were shown the cost of this little of code. The program we wrote aside from the functionality of spellchecking it was encourage to optimize the program as much as possible, the program also outputs the time it took in the functions and the overall time it took to spellcheck the file. In C the implementation of the staff took 0.6 smth s but in python it took 1.5 smth s the difference was very small but It made me realize there is no best language all of them have their advantages and disadvantages in certain situations, I'm quite bad with python currently but I'm trying to get comfortable with it cause I'm pretty comfy with c because i took a class with c++ before
If you can write fast code with slow language and have fun then you still can write faster code with fast language
can you do a tutorial on using cython to convert python programs to c\c++?
But u haven't mentioned.. What kinda libraries we should import?
Thank you, brother. make a similar on NodeJS and PHP7 please.
Tim can you make a video about importing c code into python. What about c++?
Why not just write C/C++ then?
If you took your pain to write in c
Let your prograam run in a 200 times faster language, don't convert to python
Tell about the different versions of Python such as Pypy, IronPython, Jython and so on.
Oleg Korsunskyi PyPy is fucking fast. Insanely fast. IronPython and Jython I think just allows c# and Java libraries
@@user-ob5hj5vn8c still not fast as c++ or c or luajit
@@xeome5596 True, although compared to regular python it is fast as helllllll
Isn't IronPy dead?
Sorry for noob question, but does that mean JavaScript runs into the same issue too since (I think) it is also dynamically/weakly typed?
No, the powerful JIT compilers major engine vendors use remedy that aspect. JS in many aspects is faster than LuaJIT even.
Js does not works like python, python depend on precompiled executable module, while js is run like a script in an script engine. So that gives js slightly more speed for realtime compilation engine.But if you wanna use js for desktop application speed will be worse than python.
Yes, JS engine runtime is slow..
What's cool about PyTorch is how you can develop your algorithms in python and deploy in C++.
An interpreter that runs slow.,, That ring a bell! OMG, it reminds me of the old BASICA programming languag the 1980's and MS-DOS. Python is the BASIC!
Yes!
Old Basica? That was never popular. I'm surprised you mentioned that one. I think Atari Basic holds the prize for being the slowest of them all. It has a relatively powerful set of commands, but they have to make a lot of sacrifices to fit them into 8K. Later Basics for Atari broke the 8K limit and thus run much faster.
@@simpletongeek I'm comptemplating to either learn a new language like Python, Java, or the like or just keep using some of my older compilers.
@@sidneywinter8952 why would you want to learn a new language? For backend server side development, python is the wrong language to learn. For front end development, that's HTML5, CSS, Javascript.
Java is good for generic usage, so you may want to learn that. I prefer Processing, though. It compiles to Java, and in fact, you will need Java for the really advanced usage of it. Processing is the gateway to Java and Javascript (via P5.js)
If it's for personal use, then anything is fine. I'm currently finding out that Bash is rather powerful all by itself, and that assuming you don't need GUI, it can handle a lot of the old style textual interface programs.
My preference is actually Basic and C, but Perl/Python/Ruby are useful. Just pick one. HTH.
@@simpletongeek I'm retiring next year (62) and I thought about upgrading to the new "languages". I took a course (Java) at a local community college and later saw some tutorials on the languages, so I decided to drop out after a week and get a refund and study at my own pace.
But I'm Data Engineer, not Software Developer.
So I still need to use Python for the most part.
So I guess as a Data Engineer, this is not gonna affect me, right?
Data Engineer only uses Python to get their job done.
No need to adapt in soon future?
Can you plz explain how would I write c extension code in python
that’s awesome, do you have a “deeper understanding” now. Can you make a video if so?
Hey Tim, can you do a video building a C extension? Thanks, love your channel! ♥️
JavaScript is also a dynamic typing language, and it is also much faster than Python...
I was under the impression that the major cost of interpreted languages was that they had to be translated to machine code line by line... But now that I think of it, each line has to check for scope, type and logic of variables and stuff, so yes it makes sense.
Do the external C functions use more than 1 thread?
Good video. Thank you. Python is fast *enough*, but as with anything in programming, you gotta use specific tools for specific task when you have to achieve best results. Im sure some time in the future (maybe python5?) will catch up in speeds.
well the problem is not a problem in code. the actual thing is just python is interpreted and c++ etc are compiled.
I personally noticed that Python was slower when writing some algorithms for image analysis of 3D objects. The algorithm was 1 Order of magnitude slower when written with Python compared to the same algorithm written in Java. But I love Python anyway
I have some doubts about Your multiprocessing statement. As a disclaimer, I am not an expert, I have not written any big commercial python applications, I mostly use it for automation and some auxiliary tasks. However, from my experience there are ways for easy multiprocessing in python. One such way would be to use the starmap from the multiprocessing module. It allows one to call a function for a list of inputs and executes the calls on given number of separate threads. Its use is obviously limited, so it won't help in complex cases, but it's extremely simple and easy to implement.
Nice video regardless :)
Why not take the source to flow chart and recode in assembly?
This is the first time I ever heard someone say java was fast. I usually get to hear stuff like "slow languages like java are so inferior to faster languages like haskell".
Haskell is not a fast language. Neither is java. Haskell has extremely strong typing, but is very high level, it also has a gc. Its gc is better than java's. Java is slow because it uses the jvm & a gc. C & Rust would be good examples of fast languages.
@@lincolnsand5127 In so far as we compare the couple "Language + standard compiler/interpreter", Haskell with GHC is definitely in the fast category. Of course what often makes the difference is the algorithms you use and idioms in your language can influence you to write slow code, such is the case with Haskell if you use the default lists and strings as a panacea (it is very easy to use better data structures without changing your code much thanks to type class but beginners are not always aware of that).
On the other hand I would also place Java in the fast category, comparable to Haskell.
The absolute best languages to write performance critical code are still C, C++, Rust or Fortran (eventually with some Asm) but you very rarely need this kind of performance in general programming and can often make do with fast high-level languages like Java, C#, Haskell, OCaml and so on. C and Fortran should probably be a last resort if you really need the performance, they're much too low level. C++ and Rust are reasonably high-level and Rust even provides a lot of safety compared to C/C++ but they're still more painful to write in than a higher level language.
@@chaddaifouche536 I'll tackle this in pieces. So. For the first part, Haskell is just a bit faster than java but nowhere near C/C++. Haskell is memory efficient and has the best gc I've ever seen. As for what fast is, I think Haskell and Java are slow because I mainly use C++ & Rust. So to me, these languages are slow. If you're using a snail language like Ruby or python, then Java is probably "fast." As for the last part, I somewhat disagree. I regularly use Rust even when I'm not doing performance critical operations because its abstractions and type systems are better than most mainstream high level languages that I've seen. I also think that pointers and consistent object lifetimes are very useful for code design and modeling how code should behave and neither of these are possible in high level languages (usually). Lastly, while C++ *can* be a pain sometimes, I think Rust is nicer to write code in than most high level languages. Also. I agree that Fortran and assembly are painful and should only be used when needed.
What I find most befuddling about this, is how *little* value dynamic typing brings, if any at all.
Never in my years of programming in Python I exclaimed "Gee! I'm sure glad I didn't have to spell out the type of that variable there!". What it instead does, is reducing the clarity of the program not only for the humans writing and maintaining the code, but also the IDEs themselves which often fail to give you meaningful warnings of you fucking up.
And for the cases, where it is supposed to be useful, i.e. having an input variable be of several possible types depending on what the function is supposed to do with it, Union Types a la Typescript do just as well of a job with all the benefits of static typing.
So who thought that this was ever a good idea?
Someone who wanted to monkey patch your class.
And then the best thing: In my python programming class when making a function we had to typehint, aka write out the type. So instead of 'def function(x, y, flag):' we needed to write function(x: int, y: int, flag: bool) -> None:', which greatly negates the 'advantage' of writing dynamically.
They're coming full circle.
I don’t know but I hate dynamically typed languages with a passion. I like my programming language strongly typed with type inference preferably and good code editor to help me out. Dynamically typed language make it too easy for me to create bugs that sometimes manage to hide for years only sometimes noticed and just hellish to find.
because python is an interpreted language, which means it scans the individual lines of code to see the machine instructions, while the others are compiled languages
C++, golang, rust. Or use wrappers like in ML.
Can you make a video of how we can import a c progam in Python as an extension??
Bhai kya bol raha ho
😱
🤣
He's not kidding. It's possible
@@epsilonator it's possible but not the way he describes. First you need to convert to bytecode to implement that
8:00 Python is not written in C, CPython is.
Moreover Python is not slow because of it is interpreted. Pypy is faster than CPython and even PHP is faster than CPython.
CPython is actually slow because the language wasn't develop with Performance in Mind. A number is not only a whole class, but it is by default a BigNumber: it looks like this internally: struct {int total; int* digits}. While in C, long a = 500000000; gets converted to machine code directly and the variable name thrown away. Python will save the variable name, handle the reference counting, split the number into chunks and save them in the struct which is allocated on the heap. Even writing this pains me as a C, C++ dev.
Ok, let's say, you want to anotate your code and do a: string. Python will store the annotation and doesn't actually give a crap if you do a = 2.
This is correct. I don't really know what else to say though.
Python IS written in C
The first implementation was named CPython
BOTH python and cpython are written in C just like 80% of modern programming languages
@@Markkkkkkkkkkkkkkkkk What are you talking about? I worked on the Cpython source code. Python is a programming language. It is not written in anoter language. Just like English is not written in German or Latin. CPython is the default and most popular Python interpreter. When people say Python, they mean Python interpreted by CPython.
By the way modern Programming languages (interpreter/compiler) are written in C++, not C. Take a look at Swift, JavaScript (V8), Node, Julia, Rust.
@@akj7 i am new to tech, but with what i know C is lower than C++ and its closer to the machine why they use C++ for interpreter rather than C and most compiler of C++ is written in C ?
@@qendrimimeri2219 No. C and C++ are on the same level. C++ just has more features. C is not a low level language. C++ adds classes, which get transpiled to their C counterpart: struct, and the methods get turned into normal functions. The issue with C++ is name mangling. When you write a function in C, it has the same name in the binary output however every C++ compiler generates its functions differently (because of namespaces, ...). Due to this, it is harder to create shared libraries with C++, instead nowadays, people write their programs more in C++ instead of C and create C bindings for them.
But how is it, that with this new program I've been using lately, the late version 2.8 of Blender, I can't see any difference in speed, if anything, it loads ultra fast and does all if not almost a better job than some other 3D packages, I'm really not trying to debate the worth or performance of Blender, I'm really just curious about the programming language. So how is it that I don't 'feel' this slowness of Python ? Is it just more visible in web development rather than desktop applications ?
I had to parse lots of binary files using python. Then display it in maplotlib which embedded in tkinter gui.
Python interpreter uses 17GB ram when parsed 200mb of binaries.
Nightmare.
Tbf, there is no numpy or any high-level features were used during development of that parser.
Broadly python became a C/C++ wrapper, because it is too slow.
In that sense, pythons is just a markup language for C/C++ modules.
I heard every language is a wrapper around C or C++. Go was written in C, then they used the old Go to write newer ones. Swift is written in C++, CPython is written in C. PHP in C, Java interpreter is written in C, Rust uses LLVM which is written in C++, JavaScript's Intepreter V8 is written in C++.
Python was just too slow for me so I just used lua, it’s faster and lightweight than python, and is quite similar.
Btw lua is a scripting language but there is luaJIT
What were you doing where python was too slow?
@@patrickcampbell957 (sorry for the very slow response) Iwas trying to re=create the game of life in the python terminal, I eventually transferred and edited all code to be compatible to pygame, the game ran very slow and took ages to load even with cpython and PyPy which are Jit compilers for pyhton t was still quite slow, I quit the project. I then found out about lua and LuaJIT and since moved there
exactly,every languages have their own puposes....no one is best neither worst...they were just built by their creator for specific aims.But after all, i just love python the way it is so simple...some miliseconds doesn matter...good say tim,i am a great fan of yours!!
I'm I the only one that devolps tools for automatic generic typecast in Java?
At first, it is slow, but once you have whats necessary it basically stops being Java, and it stills as efficient.
I develop trading bots as a hobby. I use C for the backtests as the compiled .exes run 50 (!) times faster than the equivalent Python. The code that does the actual online trading is written in Python though as I don't need the speed and Python is waaaayyy nicer to work with, and easier to maintain.
This is a response that shows understanding which tools to use when. Lots of comments seem to want to make the argument that the only thing that matters is execution speed.