Very useful video! I have a question about the third part, what is the portion of dataset that can i use to test the best model (characterized by the best hyperparameter)?
Hey good question. You can have - Modelling set: This would be broken down into a training and validation sets as part of the CV. - Test set: A dataset to evaluate your final model across. In reality, you don't really need the test set for final evaluation. The CV on the modelling set is doing the evaluation process anyway, Hope this makes sense?
Hi Egor, do we need to separate the dataset into train, validate sets? Meaning that I would use a subset to train and validate with walk forward and THEN apply the model with the best hyperparameters to a held out/unused test set? I see different variations of this approach and a am bit confused
Hi Maja, thanks for the question! I understand your question and why it may seem confusing. What I do, is I have a TEST set which purely for testing and no model training. Then I have MODELLING set which is for modelling. This MODELLING set is then often broke down to TRAIN and VALIDATE sets using cross-validation (walk forward for time series of course). I hope that makes sense? In reality it's just the normal train-test split but I am doing CV on my train set, so I often name it something different. :)
So I guess the final step would be to train using all the "modeling set", with the params found during optimization. My question would be how many epochs would you use?
After doing the cross validation, are we seeing the average values for the confusion matrix if I build a classification model? My question is not regarding Time Series data.
Hey, I am not sure if I understand your question? For classification, you would typically hyperparameter tune using cross validation. For each set of hyperparameters, you would take the mean across the cross-validation folds you had. Hope this makes sense?
@@egorhowell My question is not about the time series model building. It is about how and when to use the confusion matrix for a classification model. I may have confused you, asking a question apart from the content of the video.
I still don't full understand? You would use the confusion matrix to work out your precision and recall scores. Depending on the context of what you are trying to predict, will dictate which one is more important to you.
Hi Thanks for the feedback. You are aright, I am aware I naturally speak quickly. In my recent videos, I have slowed done a bit you noticed? But its an on going learning for me!
This video is great. Just what I needed. Thank you!
Glad it was helpful!
Really good explanation Egor!
Thanks David, glad you liked it!
Very useful video! I have a question about the third part, what is the portion of dataset that can i use to test the best model (characterized by the best hyperparameter)?
Hey good question. You can have
- Modelling set: This would be broken down into a training and validation sets as part of the CV.
- Test set: A dataset to evaluate your final model across.
In reality, you don't really need the test set for final evaluation. The CV on the modelling set is doing the evaluation process anyway,
Hope this makes sense?
Hi Egor, do we need to separate the dataset into train, validate sets? Meaning that I would use a subset to train and validate with walk forward and THEN apply the model with the best hyperparameters to a held out/unused test set? I see different variations of this approach and a am bit confused
Hi Maja, thanks for the question!
I understand your question and why it may seem confusing. What I do, is I have a TEST set which purely for testing and no model training. Then I have MODELLING set which is for modelling.
This MODELLING set is then often broke down to TRAIN and VALIDATE sets using cross-validation (walk forward for time series of course).
I hope that makes sense? In reality it's just the normal train-test split but I am doing CV on my train set, so I often name it something different.
:)
So I guess the final step would be to train using all the "modeling set", with the params found during optimization. My question would be how many epochs would you use?
Thats right for the first part. I don't understand the second bit of your question. What do you mean by epochs?
After doing the cross validation, are we seeing the average values for the confusion matrix if I build a classification model? My question is not regarding Time Series data.
Hey, I am not sure if I understand your question? For classification, you would typically hyperparameter tune using cross validation. For each set of hyperparameters, you would take the mean across the cross-validation folds you had. Hope this makes sense?
@@egorhowell My question is not about the time series model building. It is about how and when to use the confusion matrix for a classification model. I may have confused you, asking a question apart from the content of the video.
I still don't full understand? You would use the confusion matrix to work out your precision and recall scores. Depending on the context of what you are trying to predict, will dictate which one is more important to you.
can we use it for ARIMA?
yes of course! you can use it to find the correct number of coefficient values for the autoregressive and moving average components.
3:50
???
@@egorhowell hahah its the part i am interested to remember lol
I saved your video
ah haha
Reduce your speed If you were targeting a large audience
Hi Thanks for the feedback. You are aright, I am aware I naturally speak quickly. In my recent videos, I have slowed done a bit you noticed? But its an on going learning for me!