Deep image prior: simple code for image restoration with no training data needed

SHORT VERSION: • Simple AI image proces...
Uses a convolutional neural network (CNN) as a deep image prior (DIP) to denoise an image and also remove gaps in an image.
Code is written in Python and PyTorch.

[@yuxijin7883]

It's a fantastic video! The explanation of DIP implementation is crystal clear. Exceptional work! Thank you so much!

[@AndrewJReader]

Thank you so much for your feedback, appreciated! Glad it was helpful.

[@troyfrei2962]

Thank you for your video!!

[@AndrewJReader]

Thanks very much for the feedback! 🙂

[@prot78ap]

great video! could you maybe go through the principal differences between this method and a diffusion techniques? thanks

[@AndrewJReader]

A diffusion model is usually generative - so that a purely noisy random input image is mapped to a random clean image sample from the distribution of images that the diffusion model was trained on. The deep image prior method instead uses absolutey no training data at all, and effectively just trains a CNN to generate a representation of the single image it is fitted to, where the key point is that the training is terminated early to avoid fitting to noise in the single data.

[@sAkIBtHewOlVeRiNe]

Hello Sir, could you please explain " fig1.canvas.manager.window.move(0, 0)" the line or line number 20?
manager has no attributes named window. This line is optional but why you write that which have no existence.

[@AndrewJReader]

For the libraries / packages that I had installed on my system when I did this video, there was no problem at all. So for me at that time there was no error report. Other installations / environments / OS can of course vary. So it did exist for my setup. I used that line to move the window to the top left corner. Hope that helps, and thanks for the feedback.

[@aopengli]

Thank you for your video!I have a question, if the noise is multiple and distributed differently, isn't it harder to reduce the noise

[@AndrewJReader]

Thanks for the feedback and question. This method for denoising is applicable whatever the type or distribution of noise present in an image. But of course, more severe noise levels will always mean it is harder to reliably reduce the noise while preserving the signal.

[@chrisnewman7281]

Thanks again for the very detailed description of your script. I do have a quick question which I hope that you may be able to answer. It is the realm of deep image prior. I have a half tone image that I’ve taken from the newspaper. So my question is in a nutshell can this approach to DIP conver a half tone image to Grayscale. It’s not that the image is lacking detail but there needs to be some kind of conversion from an apparent tonal value which is actually black to corresponding grey scale value.

[@AndrewJReader]

Interesting question, it depends on the sampling of the image. In fact I would simply suggest that you could resize a highly (over)sampled version of the image (just regrid at lower resolution, assigning a value to the lower resolution image pixels according to the sum of corrsponding pixel values in the high res image). Hope that makes sense, no real need for DIP (which can do inpainting, and for a high res half tone image it would possibly just fill the gaps in accordance with the neighbouring values - and if at core the image is binary valued, then the result could be binary as well). It all depends on many things though. I would start with a simple non-deep learning approach for your problem.

[@chrisnewman7281]

@@AndrewJReader that’s gonna keep me fairly busy

[@loko818r]

thank you for the video, I have a question, this process can be applied to image translation? use the z image from one modeality to translate into another modality?

[@AndrewJReader]

This method trains a network that is really only suited to the one single image in hand. Yes, a different z image could be used. So it could convert from one modality to another, but only when you already have both images. It would need an appropriate target image in every new case - implying the target modality needs to be available in all cases (and hence why translate to it, if you have it already?). But if there is some advantage to imperfectly matching the target (as in this video, where the imperfect match allows noise compensation and gap filling), then it could be useful.

[@anutamilrajan3582]

Hi.. Thanks for the video. May I know which network architecture is used here? whether it is U net or something else

[@AndrewJReader]

Hi, thanks for the question. I just used a standard convolutional neural network (CNN), with no downsampling or pooling. Simpler than a U-Net, but serves the purpose for the task shown.

[@adosar7261]

Why the black boxes are removed? If we keep training, should they appear on the reconstructed image (due to overfitting)?

[@AndrewJReader]

The loss function does not include the black boxes in the fitting, and so even if we keep training, the black boxes are unlikely to appear as there is no fitting requirement for those pixels. This selective fitting of only some pixels is achieved by the masking out of pixels not to fit.

[@adosar7261]

@@AndrewJReader Thanks for the clarification!

[@user-sy4os9up5i]

It is a very useful video for me, thank you sharing.Could you share the code for my study? Thank you very much🎉🎉🎉

[@AndrewJReader]

Many thanks for your interest. Did you also send me an email (if that was you, then please see my reply). Many thanks, best wishes.

[@Kalyan_Pallela]

Sir could you please share Source code of this project ❤

[@AndrewJReader]

Thanks for your message - I would simply hope that the code as shown has inspired and helped, but I have not been formally releasing / sharing the code, hope that's ok. Thanks for the request though.