ไม่สามารถเล่นวิดีโอนี้
ขออภัยในความไม่สะดวก
228 - Semantic segmentation of aerial (satellite) imagery using U-net
ฝัง
- เผยแพร่เมื่อ 18 ส.ค. 2024
- This video demonstrates the process of pre-processing aerial imagery (satellite) data, including RGB labels to get them ready for U-net. The video also demonstrates the process of training a U-net and making predictions.
Code generated in the video can be downloaded from here:
github.com/bns...
My Github repo link:
github.com/bns...
Dataset from: www.kaggle.com...
The dataset consists of aerial imagery of Dubai obtained by MBRSC satellites and annotated with pixel-wise semantic segmentation in 6 classes. The total volume of the dataset is 72 images grouped into 6 larger tiles. The classes are:
Building: #3C1098
Land (unpaved area): #8429F6
Road: #6EC1E4
Vegetation: #FEDD3A
Water: #E2A929
Unlabeled: #9B9B9B
Images come in many sizes: 797x644, 509x544, 682x658, 1099x846, 1126x1058, 859x838, 1817x2061, 2149x1479
Need to preprocess so we can capture all images into numpy arrays.
Crop to a size divisible by 256 and extract patches.
Masks are RGB and information provided as HEX color code.
Need to convert HEX to RGB values and then convert RGB labels to integer values and then to one hot encoded.
Predicted (segmented) images need to converted back into original RGB colors.
Predicted tiles need to be merged into a large image by minimizing blending artefacts (smooth blending). (Next video)
![[TH] VALORANT Champions Seoul - Playoffs Day 5 - FNC vs SEN](http://i.ytimg.com/vi/ockeLeCB0wI/mqdefault.jpg)
Thanks for this. I work with aerial/satellite images everyday and it's the reason I started to watch your videos. Great work as always!!!
Glad to hear it!
wow!! THANK YOU SO MUCH!! I wish all youtubers (and professors!!!) explain things with this level of detail! this is the true democratization of education
Thanks
You are the first CS and one of the first data sci teachers I have ever had that is thorough and clear. Thank you!
Great point about resizing the labels, I used to resize and never paid attention to the interpolation that may be happening!! Thank you
You are so welcome!
Thank you so much for this channel sir...it is a one stop channel for learning ML image processing from scratch...I am recomending this to all my friends
Thank you very much.
Thank you very much for this and also thank you for your 'painfully slow pace' ;-)
I will be eternally grateful for your explanation 👍🏾
Glad to hear that!
Hey, I have been trying to get into Semantic Segmentation for a while now, and really your videos have helped me a tons lot. Thank you so much!!
If you want semantic segmentation information then my channel is a gold mine. Please go through all relevant videos.
@@DigitalSreeni Yes I am going through the playlist "U-net for image segmentation" right now as I comment. This is great content., thank you so much.
Thanks Sreeni, since my job is about cities for sure I will make sure to follow this video carefully. Keep growing, big fan of you : )
My pleasure 😊
Another great video. If your viewership is growing keep explaining it with details and/or add some link to one of your other videos if the detail is to much of a detour.
Thanks for the tip!
@@DigitalSreeni Just something i notice from other AI content creators (also python, statistics and math in general). I'm about to try that anotation tool from Zeeis to get "enough" images to try this U-net magic you have been "selling me" in your videos. I was able to succefully identify/isolate what i want from my images with tradicional color thresholding segmentation and some other morphological filtering to reduse noise, close holes, smooth contours, etc (identify granular red "marks" on a particular polished surface). Now i'm looking to see what result i can get with these methods. Also, is there any pre-processing aspect related to the quality of the pictures (white balance, glare....and other GIMP fearures) that you ever tryed to optimize the results in these ML segmentation technics? I know the convolutions kernels, poolings, etc bring the features "to the surface" but....at least in the thresholding segementation the quality of the pictures really affects the results (from excelent to really bad).
thank you for the video, very helpful!
really impressed how you explain... even a small bit of code. Thanks! please keep up the good work.
Glad it helped!
amazing work Shreeni......Needed such a extensive tutorial ......will see ur entire playlist for more information......
I must say your videos have really taught a lot in the last weeks I have watched them. You have become my favorite teacher. Thanks a lot and God bless. I am currently working on a startup idea and wish I can reach out to ask some questions or get directions if that is okay?
Please message me on LinkedIn. Hopefully I can offer some insights.
Thank you for such a detailed explanation. I learned a lot from your videos.
I am just want say a lot of thank to you
Joining other people's comment, it would be wonderful if you can provide us with some tutorials for segmenting satellite images with more than 3 channels. 🙏🙏
how are these images are labelled?, can anyone tell me please if i want to use my own images how do i label and mask and align it together, like the one in the video?
Thanks for your efforts in teaching. This one was particularly interesting because the sizes were not perfectly matched.
I wondered if there could be a case where the matching could leave an empty row or column at the edge. What would the code do with it?
First I had to check if the case actually existed.
What you did was replace the label value with 0,1,2,3,4,5 depending if the pixel was Building, Land etc.
What I did was to initialize the array with value of 6, and then run your code.
I used unique to see if I picked up a 6. In a minority of cases it does happen.
Looking at the plot I see no change, but looking at the matrix values I could see the 6's in the top row.
I think I will change my code to initialize to 5 (Unlabeled) and then ignore your check for unlabeled points.
Since you initialize to 0 (Buildings), I think unlabeled is a better choice.
In any case, thanks again for all your efforts.
Ilan
I have to edit my post because Tile 4 masks are a total disaster. The masks there are not RGB but u8bit codes. I could deal with that
but I needed to verify what the codes were in fact (and then translate to the RGB generated codes) The first image/mask in Tile 4 was the same as your code.
The following mask values were pretty much random. For example Road is value 2 for maps 1 and 2, but after that started to use value 4.
Between maps 1 and 2 the water and vegetation values swapped between themselves. If you train on them it is "garbage in, garbage out".
I wonder if your source data had RGB for the masks to Tile 4 or if was u8bit data?
I did not encounter any issues and I did verify by plotting random images. My results were also very good so I didn't have any reason to question the labels. Hopefully you'll find that it is some minor mistake in your code; I make many such mistakes.
@@DigitalSreeni
You are an excellent teacher, showing all the nuts and bolts, which allows hands on. My experience is with Fiji/ImageJ using java, where I have a PET/CT plugin. So from you I learn both machine learning and python. I make LOTS of mistakes, which is the only real way to learn.
I continued on to verify that in this case I have no known errors. Using ImageJ if you look at the mask folder of Tile 4, you will see that the files are not RGB but rather uint8. ImageJ allows me to move my cursor over the mask image and see the numerical values.
I could see from your code that you deal only with RGB masks, so without a special case for Tile 4, your code cannot work. Never mind, the idea is to learn, so I dealt with the case. In dealing with the case, I decided to check if the codes are the same as what you assigned to buildings, roads etc. To my total amazement the codes vary from image to image. The first image did indeed correspond to your code choices, but each image had a different set of codes. I could have corrected them, but I chose to ignore all images above the first one.
Since I know you are a very serious teacher, I wondered how you solved the problem? One possible answer could be that you saw a different data set than I did. It is possible that you saw only RGB and someone worked on the masks of Tile 4 and essentially destroyed them, by writing back garbage data.
I was also curious about a much smaller problem: what about the different data sizes within the tile data? There could be strange data in the missing row or column data and I wanted to see if I could detect it. I did indeed detect it as zero pixels, and to my surprise there were also some non defined pixels inside the mask data. This was for the RGB mask tiles where you replace the RGB data with the predefined codes. Since you fill with a default code of zero, you define these pixels as buildings. A better choice would have been to fill with a value corresponding to an undefined pixel (at least that is my default choice).
Again, I really appreciate your efforts in teaching and I have 2 new areas where I have learned from you.
Thanks for the great tutorial
Thanks that's a great tutorial
Glad you think so!
You are a genius. Thanks a lot
Thank you for the videos. They are amazing and very helpful. I saw some other people with the same problem as me. Could you make a video considering more than 3 channels (RGB)? Like for 7, 8… It would be very helpful for my research. Best regards.
your channel is awesome
Alhamdulillah (All praises are due to Allah) what a tutorial!, thank you Mr. Sreeni for this quality content.
It's my pleasure.
Thanks for this video
Well explained!
Hi
Your videos are amazing. So easy to learn and understand. I could properly understand U-net from your video series. Thank you so much 🥳 Is it possible you could make videos on U2-net or U-squared net? Thank you again 😄
I covered U-net variations as part of my Segmentation Models library videos. There is no point in writing every code from scratch. It makes sense to copy existing code and customize it. Of course, please provide proper reference to the original code.
Thanks, super job!
My pleasure!
Sir, please make a video on ROI segmentation
Grazie!
I followed this tutorial line by line to preprocess the images and mask. But when i did the plot for sanity check, i don't get the image and mask patch to be aligned. They are out of order. Could you suggest me some tips to get it aligned ?
I am loving your content. I am wondering if you would consider doing a tutorial on urban scene segmentation e.g. using something like the Cityscapes dataset? I’m looking at urban scene semantic segmentation at the moment for a project but there are no good, clear end to end tutorials out there so I’m having to piece it together using your tutorials for other applications.
That sounds great..Do you have another labeled dataset for that? If so, could you send it to me?
@@NourSoltani-mh1xh Hi, I had to give up on this endeavour, unfortunately, as I didn't have sufficient time to figure out how to use the Cityscapes dataset. I've parked the idea for the moment and will go back when I have more time
Thanks for all the great videos. Can you make a video on how to use LBP/HOG/SIFT type of feature extractors for image classification? Any suggestions would be highly appreciated.
You're videos have been very helpful, thank you. I don't follow why you choose minmax scaler instead of dividing by 255 ?
Your videos are very instructive for me. Thanks for informations. Can you upload a video on Medial axis transform with deep learning on LiDAR data?
Thank you
hellow! I'm a beginner.Ihave a question. If my data are not enough,how can I adding the data generator in this code.I has been try ,but not success.🙏
Thanks a lot you're always amazing, Would you please to make videos of creating masks of these areal photographs!, because in this tutorial masks are already available,
that's would be great before jumping foreword to the next topic.
I use www.apeer.com to create my masks. In fact, there are many tools out there. You just need something that lets you paint over the image and save the overlay as a mask, which is what APEER does.
Hello! I love your videos and they helped me so much during my master thesis and now during my phd project. I have a question, which noone can answer so far. How is it possible to implement data augmentation for multiclass semantic segmentation regarding the masks? For binary segmentation the lambda function with a threshold value has been used. Do you have any ideas?
thanks so much for all your videos! my satellite images however are also all different sizes, but I wouldn't be able to crop them into all similar sizes as that would eliminate important context. Is it really necessary? are there other ways to feed these images to the UNet, not using 1 single numpy array?
Thank you very much for your fantastic videos. Sir I am new to this, Deep learning, I request you to send the videos for creating our own dataset for unet (LULC) and its implementation.Pl.. sir. It would be very greatful help from ur side.
You can create your own labels / annotations for Unet using many tools out there. I annotate my data using www.apeer.com because this is what my team at work develops and it makes the process simple.
hello sir...can you make a tutorial on how to do object detection for UAV image using R , glcm
Great explanation, really help me to understand how to proceed with such related work, thank you for that. I just need to ask, whether georeference information will be available after the processing?
All metadata will be lost during the conversion of images into numpy arrays. You need to carry the metadata over and include it after all your processing. For OMETIFF files, often used in microscopy, the metadata can be saved as XML and then re-embedded later. I am not sure about geotiff files but there must be a way to easily handle it.
Can you please provide the saved model file of yours for this, I have trained my model following this tutorial but results are not coming as expected
I do not save trained models form my tutorials as they are usually trained for only a few epochs using partial data.
Hello sir, thank you for these great videos. I have a question : I want to build a U-Net (from scratch) to segment images that are not squares (720*464 pixels). Is it possible, or is it necessary to have 256*256 images ? I did not find much information or examples showing that...
Thank you
Good content
Thanks for the video. Wondering why/how you decided to make patches for images with different sizes. Did you try resizing all images to a common size by any chance?
Thank you for the fantastic explanation. The thing i did not get is why you have converted the mask images from bgr to rgb but not the images?? Can you please explain.
when crop image it can cut some details like what happen in vit network corping image cut some importatnt details . , can you explain this ?? please
Great content. How did you change the colors of the predicted masks to match up with the original masks? Thank you Sreeni
Hi sreeni. Thank you so much for the informative videos. They are helping me a lot. My Ph.D work is related to semantic segmentation. when i executed this code, I got empty sets for print(image_dataset), print(mask_dataset) and also encountered an error "IndexError: index 0 is out of bounds for axis 0 with size 0" while doing sanity check. can you please help me out in fixing this error.
Your code is not reading images. Please make sure the path is given correctly.
Hi do you think instead of unet can we SAM model...? any suggestion would be welcome
Thank you sir for all the videos..
Satellite images are always interesting, because they don't just have 3 RGB channels. It would be interesting to do it with more than 3 channels. And they also have not only 8 bit image size, but also 16 bit or 32 bit.
Do you have any suggestion to do it with u-net?
Great explanation Sir. I follow all your lectures and it has helped me to understand Deep Learning from scratch.
I have a doubt: What if the size of all the images and the masks in my dataset is the same, for example, 6000x4000. Should I use patchify here also? Though I am running it on Google Colab Pro with High-RAM settings, the session is crashing due to very high RAM demand. Any alternate solution to this is most obliged. Waiting for your response.
You want to work with 6kx4k images. Let us see how practical that would be.
Normally you'd load a batch of images into memory. Since you want to work with large image, let us set our batch size to 1.
Let us calculate the memory required for some steps and I assume the input image is RGB.
Step 1: Convert your large image to float as you need to normalize the pixel values.
6000x4000x3 float32 image - 275MB
Step 2: Apply the model by taking this input image. Let us only look at a couple of layers. First one with 64 filters and the second with 128 filters. Please remember that your network applies many more filters by gradually reducing the image size. All that information need to be stored in the memory.
Step2a: 6000x4000x64 - 5.7GB
Step 2b: 6000x4000x128 - 11.4GB
Just in step 2 with only 2 layers in the model you need about 20GB memory. If you use Colab GPU, you'd get about 12GB memory. I do not see a practical way of working with these large images. If you do the same match for a batch of 16 images, each 256x256.
Step 1: 256x256x3x16 to store float32 - 12MB
Step2a: 256x256x64x16 to store float32 - 256MB
Step2b: 256x256x128x16 to store float32 - 0.5GB (512 MB)
We can handle the above data on most systems.
PS: I used this image size calculator. svi.nl/ImageSizeCalculator
@@DigitalSreeni Sir, Thank you so much for making me understand the reason behind the memory issues. I will continue to work by following your guidelines. I really appreciate you for giving an elaborate and clear answer to my question.
No problem Sachin. Thanks for your question, I will make a short video on this topic so others also realize the limitations and resource demands of deep learning.
@@DigitalSreeni That's great Sir!! Please do it ... It will help us to more smartly design the architectures!
Sure, will do as part of my Tips and Tricks series that I release on Mondays.
35:32 In this part I receive an error: AttributeError: module 'keras.api.backend' has no attribute 'sum' ... can some1 help me with this?
Thank you so much for this! Your video makes it very easy to understand and learn semantic segmentation for satellite imagery but I want to ask about something. I have tried this code in Google Colab and it causes the RAM Memory in colab to quickly run out or there is a memory leak. It happened when I tried to preprocess the data. It jumped from 3 GB to 15 GB quickly and then it crashed. Can I do something to prevent this? And I'm curious as to which code causes RAM to run out quickly. But again, thank you for this video!
Could you explain how to make data for a testing label it is hard to find the tutorial to make a testing label or an annotation dataset?
Hi Sreeni. Your videos are amazing and very informative. What if the satellite image is labelled using label studio, how would i hot encode the mask??
But can you explain why after converting the masked images from rgb to labels , the colours in the masked images is not as the same colours as the designated labelled colour ? Can you give the solution on how to resolve it ?
Nice job! Thank you! May I ask you what GPU re you using? I use was but it seems that tensorflow cannot pass the info from CPU to GPU because of memory issues. I got this error
InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.
SyntaxError: invalid syntax on from segmentation_model as sm what to now
Hello Sreeni. when i executed this code for 100 epochs, i got memory exhausted error and now i upgraded my system to 16GB RAM and i7 processor and executed the program for 20 epochs, now my output is IoU graph and after it i am encountering the error " load_ model raise IOError( OSError: No file or directory found at models/satellite_standard_unet_100epochs.hdf5" . I don't have any graphic card in my system. can you please help me out to resolve this issue as well please suggest the configuration required to run this successfully. Thank you.
Respected Sir,
How have you prepared the dataset so well? I am trying to find the end in this code but it is not happening.When I manually label multiclass annotations. I am unable to distinguish between the built-up classes and the vegetation because the two classes are being mixed together. Kindly advise me on how to mark the annotations accurately using satellite images to prepare my mask data for the UNet model.
How compute recall ,precision,f1 for segment aerial images?
when i am trying to run the first part it shows me the wrong set of images and btw my patchifying version is 0.2.3
is the problem becuase of this because all the dataset and code are same as yours
Thanks
Welcome
I have been having version issues, I know it's silly but can anyone please share the requirements.txt for this specific code
Thanks a lot for your great videos. By any chance do you have a video where you have segmented tree crown on an aerial image or plan to make a video on that?
That is highly specific - No, I haven't made a video on that topic.
@@DigitalSreeni I have issues running multi_unet_model function in colab. What could be the solution for this?
I have the following error while compiling model with loss = total_loss "ValueError: Dimensions must be equal, but are 8 and 6 for '{{node dice_loss_plus_1focal_loss/mul_4}} = Mul[T=DT_FLOAT](dice_loss_plus_1focal_loss/truediv, dice_loss_plus_1focal_loss/mul_4/y)' with input shapes: [8], [6]." please help me
What happens if you change shuffle to shuffle=true?
Sir, please make a video on 1D signal semantic segmentation using unet
what if i had an input of 11 bands, instead of RGB ? how would i have to modify my uNET thank you for the content, is extremely helpful
Sir i am getting errors at semantic segmentation import and keras
Hi Sreeni. when i executed this code for 100 epochs, i got memory exhausted error and output is only test image and corresponding mask. predicted image is not shown. when i repeated the same for 5 epochs, again i couldn't see predicted image and received a mesage that "Allocation of 134217728 exceeds 10% of free system memory". My RAM is 8GB and i5 processor. I don't have any graphic card in my system. GPU is 3.9GB. can you please help me out with this issue as well please suggest the configuration required to run this successfully. Thank you.
Hi!! Great works! I have a question. Why I got error when load model. It says "ValueError: Unknown loss function:dice_loss_plus_1focal_loss" even though I already add the custom objects. The thing is my error says "dice_loss_plus_1focal_loss" while it should be "dice_loss_plus_2focal_loss". When I changed the custom objects into "dice_loss_plus_1focal_loss" then I was able to load the pre-trained model. Why I got this error? Thank you!!
I received error
model = get_model()
TypeError: super(type, obj): obj must be an instance or subtype of type
is there a known method to do it if we do not have ground truth labels and we do not want to label the data manually ?
Unsupervised techniques such as k-means and GMM.
if i try to train the model it crashes every time . i have 6gb gpu and 16 gb ram . is there any way to make dataloader( like pytorch dataloader) ?? @DigitalSreeni
I am trying to execute it but it is taking too much time and in between the runtime gets disconnected on Google colab, kindly let me know where I am going wrong.
Thank you very much for this amazing videos. Sir to get the aerial photo or any satellite imagery i can do but can you help with videos / steps on how you created your mask? cos most of the mask i have (i.e. multiclass labelled National land cover) is 2D and not RGB. Also the Labels are not Hex....Please help
hello sreeni, the explanation for this video is wonderful, while working with arial image segmentation, we encountered the error "No module named 'simple_multi_unet_model" . Can you please suggest us where to find the code for this.
U got an answer 4 that problem? I encounter the same problem. :/
In the multi_unet_model.py you took n_classes=4 , can i know how that works becouse we have 6 classes all together?
Thanks for a great video! can you please add requirements.txt files so we be able to test on your code as sometimes it gives alot of compatibility issues :)
Hi, how much RAM do you have on your machine? My kernel crashes whenever I train the model.
I have drone images of Paddy field, captured by DJI MAVIC MINI. Can I do such work for those images?
I am having problem to run this code I am getting error unable to allocate array. I have 65 image data set. What should I do? Any help?
Hi sir, I am watching your videos and run the codes but every time I got error related to libraries, when I upgrade one library then the other library give me error , please make a video on basic python(3.7, 3.6.....) and Libraries versions(tensorflow, keras, opencv.......) Example which TF library is working with which version of keras and python. Thank you
i want to know your python version and other libraries version.
thank you
Hello! Incredible video and very useful. Just one question, I opened the classes.json (which comes with the dataset) and the labels are different. For example, the label for building in your script is #3C1098 and in the json file is #D0021B. Why they are different? The json file is wrong? Thanks!!
#D0021B is a shade of Red and I do not see any red in their masks. I am not sure what the issue is with their JSON file, I was using the colors from their description on the data page (link below). All their described colors match the masks, so no issues.
www.kaggle.com/humansintheloop/semantic-segmentation-of-aerial-imagery
@@DigitalSreeni Thank you so much for your reply. Probably, they made a mistake when they were made the JSON file. Also, I want to ask you (hope I am not bothering you with my questions), but i have the feeling that if I use tensorflow fit method instead of custom training (using tf.GradientTape) is faster. Do you have any experience with this? Maybe i am setting wrong my custom training. (Sorry I haven't watched all your videos but probably I will 😁) thanks for your help and your comments.
Hey prof, I am getting util error while installing segmentation model
Please show some light on this area
How to create that mask file?
Did you got an answer on that topic? I had the same question :/
for total_loss=dice _loss +(1*focal_loss)
why (1*focal_loss) and not just focal_loss?
Because I am weighting focal loss by a factor of 1. You can change it to weight it differently.
My image is in tif extension and it's not reading my images
Plz hlp
What if we have unknown number of classes and the image is in the RGB format and we also do not know the hex code for each of the classes in the image how do we go about solving this problem
That sounds like something for unsupervised approach. You may want to try k-means or some unsupervised approach. I am not sure what you mean by not knowing hex code, you can use many tools to find out what the hex code for each color in your image.
Thank you so much sir, can segmentation help in any way for the classification.
Classification tasks are usually easier compared to segmentation. I am not aware of using segmentation to enable classification. I guess we are limited by our need and creativity so it doesn't hurt to try :)
Is it possible to segment objects such as trees, cars, bicycles and roads from photos taken with a drone?
Yes, of course.
where to find this - "models/satellite_standard_unet_100epochs.hdf5"???
I do not share my trained models and full data on github as it doesn't allow large data storage. Moreover, I just train models for the video purposes. This means the quality of my trained model would not be good anyway. You can easily train your own model using the code I shared.
Hi, Do you know how to do segmentation of SAR images?
Great video as always. Thanks.
Instead of segmentation, If I want to classify each pixel against a specific class, how can I do that. Please comment on this.
You basically gave the definition of segmentation - classify every pixel in an image to a specific class.
@@DigitalSreeni thanks for your reply. Please explain a bit further this topic in your coming videos.