ไม่สามารถเล่นวิดีโอนี้
ขออภัยในความไม่สะดวก
206 - The right way to segment large images by applying a trained U-Net model on smaller patches
ฝัง
- เผยแพร่เมื่อ 3 มี.ค. 2021
- Do not apply a model trained on smaller images to directly segment large images -will not work!!! (Code shared to prove this point).
Divide images into patches of smaller sizes and then perform the prediction.
Code generated in the video can be downloaded from here:
github.com/bns...
Dataset info: Electron microscopy (EM) dataset from
www.epfl.ch/la...
To annotate images and generate labels, you can use APEER (for free):
www.apeer.com
To install patchify: pip install patchify
![[TH] VALORANT Champions Seoul - Playoffs Day 5 - FNC vs SEN](http://i.ytimg.com/vi/ockeLeCB0wI/mqdefault.jpg)
Incredible, exactly what I needed. Don't stop with these remarkable videos!
This video is excellent. I like the negative examples (to prove a point) and the two approaches provided. I was just starting with segmentation with patches training and was struggling with how to handle actual images of various sizes.
Awesome! Sreeni is simple amazing, his videos are way packed with a lot of valuable information and still very entertaining!
One of my favorite channels!
Thanks a ton :)
Great video. If it had also mentioned the solution when the image size is not divisible by 256, that would have been excellent.
I have been using patch-based methods to segment large 3D images. I think the key here is to select the right field of view. Because when you use patches the FOV becomes smaller, many people choose to randomly crop the patches from the original large images. However. there are two issues with too much cropping: 1. The number of training samples will increase too much and some of them have a lot of overlapping so that the training is less efficient. 2. some of the structures need larger context to segment (non-local) and if those important context happens to be cropped out, it's gonna confuse the network. This will not be a big issue if the original images is quite repetitive (same textures like cells). But if it's distinct structure, I don't personally recommend too much random cropping and cascade network might be more useful.
Contents are just amazing! Really appreciated your hard work Sir 👍
big fan of your content, sir
from every video i learn something new
I appreciate that!
Thanks Sreeni, I owe you a great deal!!
You say that patches should be at least 4 times bigger than the objects. But what if the object is not small mitochondria but some bigger object that covers a big part of the image?
Hi @DigitalSreeni, thank you for yet another informative lesson!
I'm wondering though, if you WANT the inference to overlap (so at 21:48 of the video, instead of (256, 256), step = 256, let's say the step = 128), how can you merge those patches of segmentation inference (mask) after the prediction? What would be the correct/efficient way to "reconstructed_image = np.reshape" and "predicted_patches_reshaped = unpatchify"?
Wow this so so helpful even after 2 years! This is a gem of a channel. I've got a question for patch based segmentation though. If i wanted to validate and test my model, should i evaluate the performance based on patches? Or should val set be evaluated by patch and test set by the combined image, or should both val and test sets be evaluated based on the whole image? Would greatly appreciate anyone's insight. Thanks!
Hi Srini ..thanks for uploading this .
Very informative
very good! thanks, this tutorial help me a lot
Such a great technique and awesome tutorial! I have a question though, what happen if the patch numbers do not fit well with the image size?. In your video, the patch is sufficiently divided into integer number. What would happen if it does not and what should we do with the overlap image?
SIZE_X = (image.shape[1]//patch_size)*patch_size #Nearest size divisible by our patch size
SIZE_Y = (image.shape[0]//patch_size)*patch_size #Nearest size divisible by our patch size
image = PIL.Image.crop((0 ,0, SIZE_X, SIZE_Y)) #Crop from top left corner
You're the best 💖
I was about to ask you for that
Thanks
Code is not updated on GitHub for this video
Thank you for this. Great help! 💙
You're so welcome!
My images are 16 bit grayscale and the masks are boolen (0 and 1). Do I need to convert my normalize images to 0 - 255 range?
perfect, it's what I'm looking for
Can you please provide the code in this notebook?
all my code is available on my GitHub page, please check the description under the video.
this tutorial very helpful for me thank you
Hi! could you please elaborate more why setting "None" to input tensors works fine during training and not during inference?
Thanks for the great video. I'm curious as to why feeding the larger size image didnt go so well. From all my readings online, many people are saying that since this is a fully convolutional network, the features would have been learned and should be detectable even if we feed an image that is larger (the features would need to be same dimensions as during training however). Why did it yield such horrible results in this application?
A fully convolutional network can handle images of different size, but it doesn't mean it gets the segmentation right. If you've used objects of various sizes as part of your training, then you would see decent results on images of different sizes. In my case, all objects are of similar size so if a new image is presented where the object size varies (or pixel size), the model fails to segment properly.
Thank you Sreeni Sir for providing such useful tutorials I have been watching them past 2 hours. I have just one question regarding "patch size", what if i have original image has size of 1080*1920 in such case what would you suggest the patch_size gonna be to cover the whole image or is there any other way?. Thank you in advance :)
I think as long as your dimension is divisible by your step size, the whole image will be covered. For example an image with dimension 512 and patch size 256 and step 128 will be covered with 3 patches in each dimension (9 total). Hope that makes sense
Is there a Solution if my image is 5000x5000px. In the edges are problems with the shape and patchify stops working.
How do I define the model if I used Resnet as backbone ( Video 216).. so that I can segment large imge
Is it possible to reconstruct unet multi-class mask results like (3, 512,512))? If yes how can we do that? If that is available, can I use patchify? or needs something custom?
Great video
Would you also make a tutorial on how you upload your codes to Github and manage them?
Fantastic Explanation!!!
Please make a video on liver segmentation from ct scan
Ajarn, You used "Patchify" on the fly........Can we use "Patchify" off line also to patch/crop Training Images and their corresponding masks? In that way, apart from Augmentation, can have more images/masks for training the model......Your views please.....
Really struggling to get this running with RGB image patches. Someone kindly help me out? Thanks.
How do I train the model on the patches?
Hi sir,
How edge cells predicted can be taken, if it is predicted as different class in multi-class system ?
Can you show how we can segment out each cells and save it in a folder?
Hello Screeni. I'm trying to remake your patching solution for large grayscale images. I want it to work with ordinary color images, because trained my own U-net with inception backbone based on your tutorial. So I can't make it work with color images, I used your manual solution and your patchify library solution, but I always get different error about wrong shapes. It worked once for me, but just for the color image with equal height and width (square image) and the result was not informative. Is it possible to make it work with color images or it would be better to retrain my network with grayscale images as training data?
Color images should work especially with inception as it uses 299x299x3 (color images). I am not sure about what the exact issue you are facing. I don't think it is handling color images but rather about input dimensions. If you are using Inception you may be forced to crop your images to 299x299x3 size. Please have a look at their documentation: github.com/qubvel/segmentation_models/blob/master/segmentation_models/backbones/inception_v3.py
Sir can you say on how to train the model with patched images and reconstructing them
if We do this on satellite image after converting tif to jpg then will it not change lat long ?
What should I do if the dataset has images of different sizes? Should I resize them to a particular size and the divide into patches?
Crop them to same size. I do not recommend resizing for semantic segmentation.
Hi Sreeni, Thanks for your highly informative content. I have only 250 images for my segmentation experiment whether I can try U-net or go with ML approaches like Random forest/Xgboost. Kindly suggest your experience
Hi, did you have the occasion to test both ? I am currently in the same situation
hey , great work done , when will upload the code in github
Does patchify work on 23 band images? I have a tif file which is 4000x4000x23
I haven't tried it on such images. Just give it a try and see, I have a suspicion that it may work. Can you share the file with me, in case it is ok with you for me to use it in one of my videos? I will give it a try with patchify and if it works I will make a training video so others can benefit.
@@DigitalSreeni Sure! how can I dm you for the information?
14:49 35th line is not sure what is the purpose?
session crashed due to memory usuage in googlecolab,how can i solve
Depends on why it crashes. If the dataset size is too large you obviously need more resources, so you may need colab pro. Or you can try to train on a subset of images each epoch.
When I run the program using "patchify" and "unpatchify", I get this error. 'window_shape` is incompatible with `arr_in.shape`. Can anyone solve this problem?
It appears that the window shape you are using is not compatible with your image size. If your image is 520x520 pixels and your window size is 256x256, the math does not work as 256x2 = 512. There will be some pixels left over. Pick a window size that you can divide your image into.
You either use grey images or you may need to add channels to your patches size: (256, 256, 3)
Best VIdeo. Thank you
Great video!
I'm slowing becoming a fan.
I'm having a bit of an issue using patchify on an image with 3 channels. I'm getting an array shape of (4992, 1, 252, 252, 3) and I expect a shape of (4992, 252, 252, 3)
Yes, this is correct for 3 channel images. Just drop the 1 from your array by reshaping.
@@DigitalSreeni Ohh great. I'll do that
@@busayoawobade7177 Hey, same issue. How'd did you get rid of the 1?
@@georgemiller6010 You have to reshape it.
I ltook another approach tho. I can't remember how i exactly solved this particular one. But reshaping it is what you have to do.
@@busayoawobade7177 @DigitalSreeni I got the same error. Any solution?
There is a better way if you use more patches and smoothing between them. Take a look at the github project Vooban /
Smoothly-Blend-Image-Patches.
Hi! Great tutorial as always! I'd like to ask why predictions on the large image without resizing don't work. The network is fully convolutional and technically can be applied to an image of arbitrary size higher then the receptive field. If you resize it, you change the scale and all your filters become useless, because you introduce images from a distribution that they never were trained on. but if the scale isn't changed, why is it dependent on the image size?
Edge effects most likely. When doing spatial convolutions / pooling, zero or reflective padding is used. When the surface to circumference ratio is changed, the distribution of what you find in your receptive field changes.
Thank you!!
please how can i use patches with pytorch ?
Sorry, I am not very much familiar with pytorch. Hopefully, someone else can help.
Something like this:
import patchify
import einops
import numpy as np
img = np.zeros((1024,1024)) # mock image
# patchify
patched_image = patchify.patchify( img, (128,128,step=56)
# restack with einops
patched_image_for_pytorch = einops.rearrange(pactched_image, "N M Y X -> (N M) () Y X" )
patched_image_for_pytorch = nn.Tensor(patched_image_for_pytorch)
[didn't check this of course, but something like this should work]
Hi, make a video for YOLO algorithm.