Great video. I will add a wise take on outliers..... “Treat outliers like children …… correct them when necessary, but never throw them out.” Ed Gilroy, formerly Statistician at United States Geological Survey
Wow, what an amazing tutorial. I've been using R for 5 years, and I've never used the dlookr package before. Your explanation was simple, focused and directed to the point, just as usual. Thank you so much for your great videos. I really appreciate your work. 😊😊😊😊😊😊.
Thank you very much for nice feedback! I am glad you liked it. I have since then produced reviews on even more useful packages, "gtsummary" is one of the best. You might like it too.
I love your videos and screencasts! They are very educational and still very high-level. Could you please make a video on performing meta-analysis in R using for example the metafor or other packages? Especially covering the choices one has to make concerning types of meta-analysis: e.g. fixed-effects, random-effects or bayesian meta-analysis of different types of response variables, modulators and outcomes. It could also be cool if you covered the considerations of multivariate and multilevel meta-analysis as well as composite outcome meta-analysis, such as the concept of "borrowing of strength" (BoS) in meta-analysis when dealing with multiple outcomes or studies with small sample sizes.
Thanks for such a lovely feedback! The meta-analysis videos are definetely on my list! And they will be done. But the list still has a few topics before it, like all kind of models. Thus, please, don't expect them too soon, but stay tuned as they are important for my work and will be produced in the future!
Thanks, Robert! I also love dlookr. I actually have only done reviews on packages I do enjoy and use everyday. So, you might find my other package reviews also useful. I thing the gtsummary is one of the most capable.
Wow! Thank you! So many important informations. I have to watch this video several times. But one question: In which order would you use the packages "janitor" and "dlookr". Would be interesting to teach people how to load and handle "dirty" excel table, fix some excel problems (e.g. date as numbers or entries like "no data" in numerical columns etc) and if those problems are fixed to use "dlookr" to diagnose, explore and repair the data.
Thanks a lot for your nice reply, Hendrik! I would use janitor first and dlookr on top. I guess you already have seen the janitor video on my channel. If not, feel free ;) I also have one video on tidy data, where I show the dirty table, but there is not much of R programming. Thanks for watching!
Thanks for the feedback, Haraldur! You might also like the Deeper Exploratory Analysis Video. It's long, but very dense with lots of nice functions, including and similar to dlookr
Yury, I noticed that your code for imputing outliers in the diamonds data repeats and is thus prime for a for loop, apply or map function (this is roughly 14 minutes into the video). I did not try to get too fancy so I wrote a short for loop to iterate over the methods. The function generates the plots one after the other. I thought I might share this with you and your viewers. Here is my rather crude code: imp_na_method
Thanks for the for loop, mate! I have to say, nowadays I mostly use missRanger, because it's a very fancy and multiple imputation. I also always check the imputed values and they never disappointed so far.
Amazing video. This will be a great package for my EDA work. Many thanks. Is everything ok with your website in the video description? I keep getting a 404 Site Not Found error page. Same result when I try similar links in a few of your other videos.... ?
Thanks, I did a follow up with many more packages „deep exploratory analysis“ . My blog was shut down because they want me to pay for increasing traffic. I refuse to pay for doing something good for the world in an open source software. So, it might take me some time to find the alternative. But that’s not a problem because the blog is actually the script for videos word by word. And TH-cam is still free.
@@yuzaR-Data-Science makes total sense. Have you considered moving your blog over to Github Pages, which is free and should play nicely with your script as blog or code or code on blog pages?
I actually even tried to move it to github pages myself, but something was off, and I could not go online. I am not an IT guy, so I am waiting for a friend to have a look at it and may be help me to solve it. but as you can imagine we both me and my mate have an normal everyday job and life, so the priorities are often not on the blog. anyway, thanks for your support!
8:19 plot_normality() Is there a way of displaying qq plots too for the log and sqrt transformations? Can we request other fancier transforms such as Box-Cox or Yeo-Johnson? Separate but related question. What package looks at a predictor and returns the transformation that best normalizes its distribution?
and yes, sure, there is any transformation possible. Just type "?plot_normality()" and look inside. Hier is an example: mtcars %>% plot_normality(mpg, right = "Box-Cox")
To your last question: I am not aware of any package for that, but there might be one. However, I am not a big fan of transforming the data because you kill interpretability. Log-transform is the most harmless in my opinion. I think it's a better way to use the correct model to fit your distribution.
Yury, I have duplicated your code and it basically reproduces with a few exceptions. For instance the correlation plot is not a matrix with ellipses but rather a colored chart with the r values. I guess as the package gets updated we will see some variations. Still good stuff - thanks. P. S. I ran my code in normal r-session with script rather than in RMarkdown.
Thanks for the feedback! Sorry for that, man! Netlify shut down my blog since they want me to pay for increased traffic. I refuse to pay for doing something useful for the world (without earning absolutely nothing) and since R is open source. But I want to reopen it ASAP, as soon as I find an alternative for Netlify. It'll take some time though, because I am not an IT guy. FYI: my blog is actually the script for the video, word by word, code by code. Thanks for understanding! But if you want the access quicker, consider to join the channel and becoming a member. For members I provide the code immediately. Cheers!
I got an error: "Error in html_paged_target_numerical(reportData, targetVariable, base_family = base_family) : object 'index' not found" when I ran airquality %>%
I don't know what's wrong with this dataset ... may be it's a numeric variable Temp. It does not work at my computer either. However, I tried iris dataset, and it worked flawlessly: iris %>% eda_paged_report( target="Species", output_format = "html" ) If you don't figure it out by yourself, report a bug to the package github page
Great video. I will add a wise take on outliers..... “Treat outliers like children …… correct them when necessary, but never throw them out.” Ed Gilroy, formerly Statistician at United States Geological Survey
That’s a brilliant insight! 😀 thanks mate 🙏 I’ll use it to tell my colleagues who take outliers either too seriously or too not-seriously 👍
Wow, what an amazing tutorial. I've been using R for 5 years, and I've never used the dlookr package before. Your explanation was simple, focused and directed to the point, just as usual.
Thank you so much for your great videos. I really appreciate your work.
😊😊😊😊😊😊.
Thank you very much for nice feedback! I am glad you liked it. I have since then produced reviews on even more useful packages, "gtsummary" is one of the best. You might like it too.
Eso es lo maravilloso del universo de R, uno nunca sabes si te encontrarás con un dinosaurio.
Outstanding. Tremendous functionality accessible with relatively few, simple commands. Top notch graphics as well. Keep up the good work!
Many thanks for a positive feedback, William! That's why I love R, it makes life easier 😊 really happy that it is useful to more people than just me.
Why didn't I came across you channel till now? This is phenomenal
Thanks a ton, Sandip! That means a lot! My channel is still very young and I hope I'll produce more useful content. Cheers!
I love your videos and screencasts! They are very educational and still very high-level.
Could you please make a video on performing meta-analysis in R using for example the metafor or other packages? Especially covering the choices one has to make concerning types of meta-analysis: e.g. fixed-effects, random-effects or bayesian meta-analysis of different types of response variables, modulators and outcomes. It could also be cool if you covered the considerations of multivariate and multilevel meta-analysis as well as composite outcome meta-analysis, such as the concept of "borrowing of strength" (BoS) in meta-analysis when dealing with multiple outcomes or studies with small sample sizes.
Thanks for such a lovely feedback! The meta-analysis videos are definetely on my list! And they will be done. But the list still has a few topics before it, like all kind of models. Thus, please, don't expect them too soon, but stay tuned as they are important for my work and will be produced in the future!
An overall thanks for all the videos uploaded into this channel!
Thanks a lot 🙏 glad it’s useful!
This (specially this!) is marvelous, but also the rest of the series of explanatory videos. Congrats!
Glad you like them! Since they are already a bit old, you might find the more recent videos also useful. Thanks for watching!
I've been using data explorer and dlookr more in my learning journey thanks to you sir.
Glad to hear that! More exiting package will follow! I just do it slowly due to my day job. Thanks for your support and Cheers 😊
I really hope your channel will grow in the future. Your videos are very helpful to me.
I am very glad to hear that! Thanks! And thanks for watching, it's the best support!
What a great presentation. I love this package. Thank you for introducing it and descdribing it in such an easy to follow presentation.
Thanks, Robert! I also love dlookr. I actually have only done reviews on packages I do enjoy and use everyday. So, you might find my other package reviews also useful. I thing the gtsummary is one of the most capable.
Wow! Thank you! So many important informations. I have to watch this video several times. But one question: In which order would you use the packages "janitor" and "dlookr". Would be interesting to teach people how to load and handle "dirty" excel table, fix some excel problems (e.g. date as numbers or entries like "no data" in numerical columns etc) and if those problems are fixed to use "dlookr" to diagnose, explore and repair the data.
Thanks a lot for your nice reply, Hendrik! I would use janitor first and dlookr on top. I guess you already have seen the janitor video on my channel. If not, feel free ;) I also have one video on tidy data, where I show the dirty table, but there is not much of R programming. Thanks for watching!
Thanks a lot Dr. Yury. Nice and helpful video.
Thank Ange! Glad it was helpful!
This is great stuff!
Thanks for the feedback, Haraldur! You might also like the Deeper Exploratory Analysis Video. It's long, but very dense with lots of nice functions, including and similar to dlookr
Great! I love your videos. Please cover mixed models 🙏🤓
Thanks a lot, landoska! Noted! Funny enough, I do lots of mixed models in my job - medicine statistician.
Fanatastic video and code examples.
Glad you liked it!
Yury,
I noticed that your code for imputing outliers in the diamonds data repeats and is thus prime for a for loop, apply or map function (this is roughly 14 minutes into the video). I did not try to get too fancy so I wrote a short for loop to iterate over the methods. The function generates the plots one after the other. I thought I might share this with you and your viewers. Here is my rather crude code:
imp_na_method
Thanks for the for loop, mate! I have to say, nowadays I mostly use missRanger, because it's a very fancy and multiple imputation. I also always check the imputed values and they never disappointed so far.
And thanks a lot for a nice feedback! I am glad videos are useful :)
Great video and great analysis ! Thank you very much!
I also like package("recipes") and package("vtreat").
You are welcome. Thanks for the tipps, I'll check them out
Thank you so much for the video
Glad you enjoyed it! Thanks for watching!
Thank you very much for the videos!
These are life changing indeed.
You are very welcome, Sadat! :)
How do you get the results in the same window as your code? And the ability to preview graphs?
Via R-Markdown document instead of R script
Excelente
Thanks Martin!
Many thanks for your help
You are welcome. Thanks for watching!
Amazing video. This will be a great package for my EDA work. Many thanks.
Is everything ok with your website in the video description? I keep getting a 404 Site Not Found error page. Same result when I try similar links in a few of your other videos.... ?
Thanks, I did a follow up with many more packages „deep exploratory analysis“ .
My blog was shut down because they want me to pay for increasing traffic. I refuse to pay for doing something good for the world in an open source software. So, it might take me some time to find the alternative. But that’s not a problem because the blog is actually the script for videos word by word. And TH-cam is still free.
@@yuzaR-Data-Science makes total sense. Have you considered moving your blog over to Github Pages, which is free and should play nicely with your script as blog or code or code on blog pages?
I actually even tried to move it to github pages myself, but something was off, and I could not go online. I am not an IT guy, so I am waiting for a friend to have a look at it and may be help me to solve it. but as you can imagine we both me and my mate have an normal everyday job and life, so the priorities are often not on the blog. anyway, thanks for your support!
Just awesome 👍
Thanks a lot 😊 I love dlookr too! You might also like the gtsummary review, if you did not see my video on it already. Thanks for watching!
Thanks for this one
You are welcome!
I have a question plz, Why did we put “temp” as a predictor to imputate missing values in Ozone variable ?
simply as an example of a predictor
@@yuzaR-Data-Science Am sorry, I cant get it !
sorry, what exactly can't you get?
sorry, what exactly can't you get?
@@yuzaR-Data-Science Do we have to put a predictor to impute missing values in a variable?
8:19 plot_normality()
Is there a way of displaying qq plots too for the log and sqrt transformations? Can we request other fancier transforms such as Box-Cox or Yeo-Johnson?
Separate but related question. What package looks at a predictor and returns the transformation that best normalizes its distribution?
hey, sure, I'd use:
> ggpubr::ggqqplot(log(mtcars$mpg))
> ggpubr::ggqqplot(sqrt(mtcars$mpg))
and yes, sure, there is any transformation possible. Just type "?plot_normality()" and look inside. Hier is an example: mtcars %>% plot_normality(mpg, right = "Box-Cox")
these are transformations: "log", "sqrt", "log+1", "log+a", "1/x", "x^2", "x^3", "Box-Cox", "Yeo-Johnson" possible with plot_normality().
To your last question: I am not aware of any package for that, but there might be one. However, I am not a big fan of transforming the data because you kill interpretability. Log-transform is the most harmless in my opinion. I think it's a better way to use the correct model to fit your distribution.
@@yuzaR-Data-Science Thanks a lot. Look forward to a vid on Decision Trees and Random Forests with R packages... 🧁
This a awe·some package.
Thanks David! I hoped this would be helpful not only to me!
Is dlookr the best package for outlier diagnostics and correction?
What we call "the best", depends on a lot ... but, I think, it's certainly useful enough to stop looking for other packages :)
@@yuzaR-Data-Science What's the bestest, most prettiest decision-tree visualization package? Rpart and the like all look pretty industrial.
@@chacmool2581 Don't know, because I don't use decision-trees
great package, kudos and keep the work!
Thanks a lot Sergio for such a nice feedback! I will continue! Cheers!
Yury,
I have duplicated your code and it basically reproduces with a few exceptions. For instance the correlation plot is not a matrix with ellipses but rather a colored chart with the r values. I guess as the package gets updated we will see some variations. Still good stuff - thanks.
P. S. I ran my code in normal r-session with script rather than in RMarkdown.
Sure, usually, they get better. It's just amazing, that this is open source :)
Really Really love your tutorial！！
Thanks a lot Yao! Your feedback is very encouraging!
Amazing video!! thanks a lot!!!! but the code link is broke D:
Thanks for the feedback! Sorry for that, man! Netlify shut down my blog since they want me to pay for increased traffic. I refuse to pay for doing something useful for the world (without earning absolutely nothing) and since R is open source. But I want to reopen it ASAP, as soon as I find an alternative for Netlify. It'll take some time though, because I am not an IT guy. FYI: my blog is actually the script for the video, word by word, code by code. Thanks for understanding! But if you want the access quicker, consider to join the channel and becoming a member. For members I provide the code immediately. Cheers!
I got an error: "Error in html_paged_target_numerical(reportData, targetVariable, base_family = base_family) :
object 'index' not found" when I ran
airquality %>%
eda_paged_report(
target="Temp",
output_format = "html"
)
I don't know what's wrong with this dataset ... may be it's a numeric variable Temp. It does not work at my computer either. However, I tried iris dataset, and it worked flawlessly:
iris %>%
eda_paged_report(
target="Species",
output_format = "html"
)
If you don't figure it out by yourself, report a bug to the package github page
@@yuzaR-Data-Science Thanks, Yuza.
Happy to help
Doesnt work with Quarto, sadly.
Oh, good to know! I didn’t try it with quarto
What parts (still) do not work with Quarto?