nice! i tried with python, but couldnt get the same speed. mine took13.5 seconds overall. most of the time (13s) is spent casting the text into numpy array.
wow nice that's pretty quick for python. Did you get 100% accuracy? What did the casting code look like? perhaps you could make a numpy array of bytes, then convert that to numpy array of floats? Was it pypy? did you use float as the datatype? did you use split or splitlines? I think at the moment Ndarray is missing some optimisations that numpy has / doesn't cover as much hardware etc. but would love to see more data scientists using Rust. btw I thought about trying to init my ndarray directly from the buffer (e.g. zero-copy), but decided to just convert to floats, not sure if working with u8's directly would work. Ideally if the threadpool is big enough, it could pull buffers from the file reader fast enough that the solution runs in the same speed as it takes to read the file.
nice! i tried with python, but couldnt get the same speed. mine took13.5 seconds overall.
most of the time (13s) is spent casting the text into numpy array.
wow nice that's pretty quick for python. Did you get 100% accuracy? What did the casting code look like? perhaps you could make a numpy array of bytes, then convert that to numpy array of floats?
Was it pypy? did you use float as the datatype? did you use split or splitlines? I think at the moment Ndarray is missing some optimisations that numpy has / doesn't cover as much hardware etc. but would love to see more data scientists using Rust.
btw I thought about trying to init my ndarray directly from the buffer (e.g. zero-copy), but decided to just convert to floats, not sure if working with u8's directly would work.
Ideally if the threadpool is big enough, it could pull buffers from the file reader fast enough that the solution runs in the same speed as it takes to read the file.
Nice Rob