The Billion Rows Challenge in Rust - an intro to Rust for data engineering

Learn about using Rust for efficiently processing high volumes of data. This live tutorial introduction to the Rust language will explain how to craft utilities for your data engineering pipeline.
The speaker, Tim McNamara, leads a global technology consultancy that helps teams create secure, energy efficient and reliable applications with th Rust programming language.
No experience in Rust required, but you'll greatly benefit from some knowledge of programming. Bring your questions, bring your data. Let's go!
#rustlang #dataengineering #bigdata #easydata

[@rahuldevstech]

"On the planet to build a better planet"! As a Python data engineer with a background in environmental sciences, that resonates a lot with me. Thank you for what you do!

[@mrlectus]

python doesn't help with the environment

[@rahuldevstech]

​​@@mrlectus Agreed. Trying to change the tech stack where I'm at to use Rust 🦀 Needless to say, Tim's content helps me a lot with that!

[@theuberlord7402]

Nice to see a Kiwi leading the charge on Rust education and advocacy.

[@flwi]

Thanks for taking the time to explain your workflow in such great detail! Glad I found your channel just now, because I started learning rust this weekend

[@omaslo]

Thank you for this stream! So many interesting things explained!

[@michaelwagner4801]

The accent character is called an umlaut

[@CarelessMiss]

That would be an Ä

[@sergeykuznetsov7688]

Thanks for the video! While watching I wanted to experiment with so many things so I ended up solving it by myself for both single and multi thread modes. Being quite simple It turns out to be a very educative problem.

[@timClicks]

Nice work! I'm so glad that you have fun playing around.

[@andydataguy]

Ooo can't wait to watch this!

[@sidkang]

wow, really interesting topic, happy to see this. Thanks!

[@knolljo]

Love the challenge, thanks!

[@hanling8489]

like it

[@coolbuddy95able]

Great video! Couple of weeks ago I picked up Rust, and I'm loving it. Would have been good if you could benchmark it in the video. Any chance you did it offline?

[@GlobalYoung7]

thank you😀

[@minciNashu]

else isn't mandatory after if-let.

[@learning_rust]

I'm just curious - if f32 would have been sufficient for the temperature, and if so, would that make any performance improvement?
I come from Python background so naive on this and keen to learn!

[@julians.2597]

Measure, measure, measure. Only way to show performance differences, and then only on specific setups

[@SPeeSimon]

Look at the java examples and what they did there. The fastest ones are working with the raw bytes and not converting it into a string, f32 or f64 type.
It would be great to see if a rust example can achieve the same kind of high speeds. Is it still easy to write and read if you are using multithreaded and direct file reading? Or is that not something a good dev can create as compared to java.

[@JOHNSMITH-ve3rq]

Brother is Aussie?? Cool!

[@timClicks]

New Zealander, but close enough 😂

[@dimitardimitrov3421]

Didn’t finish the whole video yet, but you’re supposed to store the MEAN not the average…
Edit: My bad, I read "mean" and my brain understood it as "median". Sorry about that.

[@Gramini]

What's the difference? From what I gathered they're pretty much identical. It seemed to me that they differ in some technical math terms regarding their history, but that's not not really important in the broader sense.
"the mean is the average of all the data values"

[@dimitardimitrov3421]

@@Gramini I'm sorry, you're absolutely right. For whatever reason I read "mean" and I understood it as "median". A median (which in this case it is NOT, my bad) would change the implementation quite significantly.