L9.5.2 Custom DataLoaders in PyTorch --Code Example

Sebastian's books: sebastianrasch...
Slides: sebastianrasch...
Link to the code: github.com/ras...
-------
This video is part of my Introduction of Deep Learning course.
Next video: • L10.0 Regularization M...
The complete playlist: • Intro to Deep Learning...
A handy overview page with links to the materials: sebastianrasch...
-------
If you want to be notified about future videos, please consider subscribing to my channel: / sebastianraschka

[@cvresearch9218]

Great tutorial. It is useful to use PIL consistently since Pytorch Dataloader expects PIL or Numpy array.

[@melanieschaller5343]

Great explanation. I think the error with worker=1 comes from the velocity of loading. If multiprocessing is called there could be a chance, that two workers get the same idx. Therefore you should use the worker_init_fn in the beginning to avoid this. This should work out.

[@SebastianRaschka]

Actually, I never toyed around with worker_init_fn. Thanks for the hint!

[@kenbobcorn]

Just a suggestion: if you want Jupyter to autoreload your imported packages instead of you having to restart the kernel and re-run the notebook, run this in a cell
%load_ext autoreload
%autoreload 2

[@SebastianRaschka]

Ah yes, thanks, that's a good one!

[@amk2298]

Thank you for this awesome playlist, i have watched all of your videos and found it extremely useful, Thank you again .☺

[@repli5869]

Just what I wanted. Good Vid

[@PoojaKumari-ic7zh]

Hey Sebastian, I am really enjoying your lectures. Thank you for that. I use .json files always. Can you please make video on that. For example I have .json of kepoint labels. Thank you.

[@SebastianRaschka]

Oh interesting. I never worked with JSON files in the context of DataLoaders. But yeah, I suppose you would go about it similar to CSVs. Btw I recently wrote a blog article about PyTorch's new DataPipes that might be interesting: sebastianraschka.com/blog/2022/datapipes.html (there is a specific JSON parser IterDataPipe actually: pytorch.org/data/0.3.0/generated/torchdata.datapipes.iter.JsonParser.html#torchdata.datapipes.iter.JsonParser). Should make a video about that as well some time!

[@PoojaKumari-ic7zh]

@@SebastianRaschka Thank you 🙂

[@PoojaKumari-ic7zh]

I also saw your lectures on LSTM. It is very basic. I need some hints regarding LSTM training. I am working on a project same as action recognition using keypoint detection(animals). I am not able to find any material on How to train LSTM or RNN models using Video material. What should be the data format for training LSTM. I converted Videos into Images and Annotated keypoints on everyimage. Got annotation file. With this, I can only do keypoint detection. LSTM training is still confusion. Some people suggested to deetct keypoints(using rcnn or similar nets) and directly give to LSTM. But before that LSTM should be trained, If I am right, but how? I am sorry for long doubt. I am following your lectures and Your books only. So just thought if you could give some hints.

[@SebastianRaschka]

@@PoojaKumari-ic7zh Typically, you use a CNN for the feature extraction layers in the LSTM. I think if you search for CNN-RNN Video Classification you should be able to find some code examples. E.g., this here: github.com/pranoyr/cnn-lstm

[@PoojaKumari-ic7zh]

@Sebastian Raschka It worked and I have completed my project successfully. Thanku so much for your guidance. Big fan of your work. Thank you again. Currently I am watching your series on ML and it is in more details than DL series.

[@jreas_]

Great video! Thanks

[@cedrickcalib-og3463]

Can I use this code for creating custom dataset for Prototypical Network (Few-Shot Learning)? I am using EasyFSL

[@SebastianRaschka]

You'd have to make some adjustments, but generally, yes. PS: It's been many years, but my first PhD student was doing tons of few-shot learning in PyTorch

[@kritiohri558]

Sir please make a tutorial on balancing the training dataset using pytorch.

[@akshat407motu]

Hello, I am getting a constant error: "raise KeyError(key) from err KeyError: 'File Name' " seems like there is an issue with the line self.img_names under the MyDataset class. Can you please help me with this?

[@SebastianRaschka]

Hm, maybe you need to modify the get_dataloader function to get it the absolute image paths.
def get_dataloaders(batch_size,
csv_dir='.',
img_dir='.',
num_workers=0,
batch_size_factor_eval=10,
train_transforms=None,
test_transforms=None):
I.e, setting csv_dir and img_dir paths

[@shubhamkapoor8756]

I have 4 zip files of MNIST data marked as train-images-idx3-ubyte and three more .I am not very sure how to load them in dataloader.Can you suggest a way to do it?

[@SebastianRaschka]

It's actually how the internal MNIST/FashionMNIST dataloaders in PyTorch/torchvision work. They are also loading the from the ubyte files. You can take a look for inspiration here: github.com/pytorch/vision/blob/main/torchvision/datasets/mnist.py
Together with that, I have a section on loading MNIST from the ubyte files in my book, the code can be found here: github.com/rasbt/python-machine-learning-book-3rd-edition/blob/master/ch12/ch12.ipynb

[@shubhamkapoor8756]

@@SebastianRaschka thanks it was a great read.

[@chidiedim3166]

how can i get the dataset

[@SebastianRaschka]

It's inside the repository here: github.com/rasbt/stat453-deep-learning-ss21/tree/main/L09/code/custom-dataloader