วีดีโอ

Good stuff, thanks for the feature :) So interesting to see that results seem to differ from hardware to hardware. on my AMD Ryzen 9 5950X w/ Julia 11.2 I cant reproduce the improvements for the base vec implementation, but glad you can squeeze the lemon even more xD.

That is awesome! I am really glad we could collaborate and learn things. Sorry I couldn't post my solution. Got cuaght up with Holiday travel. Maybe I can post soon. It's really quite fun to optimize these little problems. Again, really enjoy your videos!

Thanks again for yet another very nice and informative video. It is always a good excuse for me to play around with Julia when i see it comes in dead last in a benchmark :) I've created a script for part one on github which achieves around 280x speed improvement on my machine. (26 ms -> 0.095 ms) Sadly I cannot post a link here that leads to my repository. I'll try to post the code in a reply to this comment. Core changes from original version: - Use a Matrix of characters instead of Vector of Strings - Use CartesianIndices to navigate the matrix. This makes sure it iterates "the right way to be fast" (col-major vs row-major memory layout) - Limit memory allocations in hot loop (contrary to popular believe it is not "basically free"). The new version allocates 16 bytes where the original one was > 6 MB - Use StaticArrays for faster array processing

This approach just makes me cringe. It's completely against the rules of the game to abuse try/catch for plain flow control.

Hi! I'm trying to follow your commands in my lancbox tool. I am wondering if it may be the case that I have a version 5.0.3 and operating on Windows and that's why it doesn't follow with, for example, [pos="N.*"]

Cool, thank you for picking up my comment. 🙂

Again ..tq Dr for the information.

You aren't supposed to use try catch, as part of your logic. While on happy path it comes without overhead, on error path it is horrible slow. Julia should perform much better if you don't use the try catch.

maybe add haskell? elixir? f#? rust? golang? maybe it's worth testing them in task 13?

Interesting comparison. Learn a lot from it, thanks!

No need for Python here, just cd to the folder with your pdf files in Terminal (you can just type "cd " and then drag the folder from the Finder into the Terminal window). Then execute ".../pdftotext *.pdf". This will convert all your pdf files. There's lots of things that can be done in the Terminai with the programs that are already installed (e.g. "wc" for word- or linecount, "sed" for replacing text, "awk" for all kinds of text manipulation and frequency count).

I can't find 'PRAAT' menu in the praat objects even though I have downloaded the most recent one. Could you share how we can go on with it?

Dr. EKB you are amazing, it is very informative. may I please request for any guide or manual for Regex Analysis.

That was very cool. Perf_counter() is more accurate for measuring performance btw.

Thanks for the amazing tutorial for us.

Line by line wins if you also consider memory usage. You could easily grab(N) number of lines or parallelize the process

underated content. Subscribed!

For this task you could either make use of pandas: >>> string = "Tilda BAR baz THE geez THE THREE geez Marta Medelin".split() * 8400 >>> import pandas as pd >>> s = pd.Series(data=string) >>> s.value_counts().to_dict() {'foo': 2, 'bar': 1, 'baz': 1} ...or the Counter from the stdlib, which is typically chosen for this kind of tasks and is a subclass of dict: >>> from collections import Counter >>> Counter(string) Counter({'foo': 2, 'bar': 1, 'baz': 1}) The Counter is faster as it employs the counting function implemented in C - _collections__count_elements_impl from the CPython source code. The pandas approach for 84000 words took 3.72 ms, Counter - 2.87 ms on Macbook Pro M2 Max (~3.7 GHz).

1.11 make Array a native julia type instead of a c wrapper, I think Dictonnary now also rely on the new Memory type so that made it even better. 1.12 should allow the compiler to actually use it to say if a vector should be only stack allocated ect ect so maybe another 1.5 x there ? not sure at all

So, a question for you: First; I have some "test feasibility" code in Python that, when implemented" will be "large linked lists of dictionaries of dictionaries, so to speak. What language should I program "commercial grade" code in? My initial strategy was to test code in Python, then "commercial code" in Mojo.

That is not even an order of magnitude difference… and the Python code is not even optimized for speed. Well done Python! 🎉

Can you please explain the log likelihood?

This is awesome, thanks for taking the time to share this

I will touch Mojo as soon as they eliminate the competitive clause from its license.

What would the same task in R look like given the native aggregation functions ?

Nice video, this comes in handy as I was indeed asking myself what use cases warrant reaching to PyO3. I am wondering though, if we convert the call `out_dict.get(w, 0)` to a "dummy" `if w in out_dict` won't it be faster? Something I also find missing here in the video is the memory and CPU (cores) usage. Not that I think Python would do better there, but it would be interesting to check.

Great job, keep it up!

Well I did played around a bit and what found was if you just use mean() from the Statistics package instead of the the home grown straight forward for loop implementation you get an improvement even over the bytes method. Below is how I modified the function makes it a bit more simple and readable as well a performance boost with out the fancy byte stuff. ( Nothing wrong with fancy byte stuff, that was a good catch) function get_mattr(word_list::Vector{String}, window_span::Int = 50) n_words = length(word_list) effective_window_span = min(window_span, n_words) n_windows = n_words - effective_window_span + 1 if n_windows <= 0 return get_ttr(word_list) end mean_ttr = mean(get_ttr(word_list[i:(i + effective_window_span - 1)]) for i in 1:n_windows) return mean_ttr end Here is a link to the data and the output graph I generated. drive.google.com/drive/folders/1-AelwjZZtAPGKf_bLkhkLOC0ZZUWBuTf?usp=sharing

Hey nice vid. I am a Physicist. I work for a photonics quantum computing company and use Julia for modeling in my work. One thing you may consider is using the BenchmarkTools for Julia. I am not sure but the tail at the beginning of your graph might be due to the JIT compiler optimizing. If this something you do often you could precompile the julia code, once optimized and that would negate the JIT start up time. I will play around a little bit and get back with you. I like the attitude of always willing to learn something from some one else. There is so much out there to learn if just listen and not jump to conclusion.

Amazing doucheFace thumbnail! Congrats you look like every unimaginative, lazy creator on YT. Clearly intelligent people choose stupid face thumbnails because looking like an idiot is a huge indicator that your content must be amazing! 😂

When I run it on linux. and click on start. it crashes and closes. I don't know why!

Probably worth noting, Polars is quicker because it's multi-threaded and uses all the cores on the machine, were as Pandas is single threaded

pandas has a join method. It's supposedly faster. You just have to set the join columns as the index before calling.

What is the difference between the original and the X version?

I have been learning rust so looking forward to watching this video!

Great vid!

I really enjoy this channel! I have been picking up rust from python trying to solve bottlenecks from speed

Blue & pink lines are roughly linear. Roughly at 70k, there could be some memory allocation or something else that has dropped the performance. More like a one time thing within each test above 70k. With this I tend to think that it's linear for normal hash reading ( blue & pink ).

I would assume the fastest way to access values is using dict.values() :)

you didn’t access them the way they meant to. You need to iterate keys and access each value using the key. Otherwise hashing is not required to get the map elements. Simply array can do that.

This is just what I was looking for. I am working on business logic and there are a lot of SQL statements. Python takes about 9 seconds to have all 22k entries in a dict, a C++ program I had ChatGPT just write up based on the Python code (it took like 20 iterations of me telling ChatGPT well now I'm getting THIS error) was 10x faster. I won't learn c/++, I'll do some Rust!

Thanks Professor! I'm learning on the job on my own how to implement python & rust together given my interests in software development, data science & biomedical science so this is an interesting series you made!

Great video, but the result is interesting. Mind if I can get eh full main.rs file and dataset. Would love to run the tests my self and perhaps improve upon it

Awesome, I was waiting for that video! I thought that Python's GIL blocking in your last video had a much stronger effect. I guess strings are in all programming languages pretty horrible, because utf-8 doesn't have a fixed byte size, so all programming languages have to use the slow techniques that python uses for all data types. Python is pretty good compared to other languages when talking about dicts and strings. The rest is very slow, but thanks God, it is very simple to speed it up if you need it.

FWIW i got about a 30% speed increase for julia when working on bytes directly (vs strings) and passing @view of byte-vectors. wds = split(txt) bwds = [Vector{UInt8}(word) for word in wds] and then passing the @view of bwds instead of wds into the individual functions note: i also dropped all the println() and I/O operations in my code as i was mostly curios about the speed of julia and not I/O or print. (but fair play if it is included)

Great video, I like your benchmark videos, but from Rust's perspective it's a little unfair. It's not representative when Rust is a slave of Python's GIL. However, I think the speed difference between the language is not that big, because UTF-8 chars don't have a fixed byte size. It would be nice to see a comparison between utf-8 and byte strings (fixed size). You could also add Zig to the benchmarks, I used it a week ago for the first time (RGB search in a picture). It was between 2 and 10(!) times faster than C, but I haven't tested it with strings yet.

Thank you for sharing this!

But only as long Mojo isn't out there to perform python to it's coming new standard limits even faster than c++. The future is going to be fast as hell bro 🎉

Bah, Mojo is not even open source. That's repugnant.

How about using glob as a generator? Would that reduce the gap beteen the os.walk and glob? Because in your code, glob() loads everything into memory and that may adversely impact the runtime.