Please do a deep dive into optimizing the mutiprocessing. Most people don't understand the difference between threads and processes and how threads are light weight processes that share data, code, and the heap but have their own stack which makes context switching quicker.
Hey Ige, nope, the order has no effect. As mentioned, it was a long known issue in the JDK, I'll need to dig around and find the openjdk bug report for this one.
16:25 Would be interesting to see how you solve the potential race condition when two threads work on identical files. I'm thinking about some kind of synchronized Set which stores the hashes of already processed images, maybe? 🤔
There are many ways to approach this. The easiest one I could think of for now is to let the two threads work on identical files, but render the resulting thumbnail to a randomly named tmpfile. And after rendering has finished, "mv" the tmp file into the final destination. If some other thread has already created the final file, then the mv will just be ignored, the file deleted and that's it. It's a simple solution without elaborated synchronisation on the Java side, just a bit of additional CPU wasted in this corner case.
Concurrency in practice is literally it. It's a bummer that there is no new version out, but reading/working through it will give you the best base possible. You also might want to follow up with the "little book of semaphores"
ImageMagick in fact already is multithreaded, and you can influence that behavior with command line options. Explaining what effect that has is a whole other topic :) as for the Nas, it is self-assembled and I don't quite remember the individual parts - it has been a couple years.
Just interested... what is the rationale behind strictly not using old File API? Performance? For me it looks like big price to have so much technical code copied instead of calling hasher.update(image.toFile()). Thank you.
That is a good question. It is just from past experience that I've seen the file API becoming cumbersome at some point, e.g. testing. And I got into the habit of making sure to use the Path API whenever I can - but sure, for the scope of that prototype we could have also used the "old" file API.
nothing new , most important question were not answered. you ask questions we are here for answers. that is actually a bad thing. we are here looking for the answers not questions. question already raised by ourselves.
Marco, has anyone ever told You that You're doing absolutely fantastic stuff? Because You're doing absolutely amazing stuff
Jacek, thanks a lot for those words!! Much appreciated and a great start for the day for me.
Please do a deep dive into optimizing the mutiprocessing. Most people don't understand the difference between threads and processes and how threads are light weight processes that share data, code, and the heap but have their own stack which makes context switching quicker.
Will see how/when to fit this in!
No need to say we want some content or not, from you it's interesting to see about everything :)
Awesome content as always. Thanks Marco!
Thank you!
Great as always!
Hello Marco,
I believe calling the parallel method before the other operators should work with the old Java version.
Hey Ige, nope, the order has no effect. As mentioned, it was a long known issue in the JDK, I'll need to dig around and find the openjdk bug report for this one.
16:25 Would be interesting to see how you solve the potential race condition when two threads work on identical files. I'm thinking about some kind of synchronized Set which stores the hashes of already processed images, maybe? 🤔
There are many ways to approach this. The easiest one I could think of for now is to let the two threads work on identical files, but render the resulting thumbnail to a randomly named tmpfile. And after rendering has finished, "mv" the tmp file into the final destination. If some other thread has already created the final file, then the mv will just be ignored, the file deleted and that's it. It's a simple solution without elaborated synchronisation on the Java side, just a bit of additional CPU wasted in this corner case.
I learn a lot from this video.
Hi Marco, any good resources on concurrent/multi threading programming in Java apart from concurrency in practice that you have found helpful?
Concurrency in practice is literally it. It's a bummer that there is no new version out, but reading/working through it will give you the best base possible.
You also might want to follow up with the "little book of semaphores"
If ImageMagick was multithreaded would you want to see it using its own thread pool? Also can you say which NAS that you’re running Java on?
ImageMagick in fact already is multithreaded, and you can influence that behavior with command line options. Explaining what effect that has is a whole other topic :) as for the Nas, it is self-assembled and I don't quite remember the individual parts - it has been a couple years.
Just interested... what is the rationale behind strictly not using old File API? Performance? For me it looks like big price to have so much technical code copied instead of calling hasher.update(image.toFile()). Thank you.
That is a good question. It is just from past experience that I've seen the file API becoming cumbersome at some point, e.g. testing. And I got into the habit of making sure to use the Path API whenever I can - but sure, for the scope of that prototype we could have also used the "old" file API.
combine two filter in a private method
yup, possible.
nothing new , most important question were not answered. you ask questions we are here for answers.
that is actually a bad thing. we are here looking for the answers not questions. question already raised by ourselves.
Ironically your nonsensical comment contains not a single question.
@@MarcoCodes His prose follows the style of Gottlob Frege lmao
you are not going to continue this google photos clone
Why?
@@MarcoCodes common its 1st April you can't fall for this
I DID! :D Good one!