Amazing video you have put together here. I enjoyed how clear you were as well as the pace you took to go through the steps and explain everything. I am new to this kind of thing so does anyone have resources on where I can learn how to interpret cluster graphs
Hi Andy, this was a great tutorial as it's something I would like to try on a csv file with various metrics in the design of a pharmaceutical. I have one question though: I will be wanting to use 5-7 columns on the csv file for clustering - how do you go about visually representing this? I can't think of a good way to do it. Thanks!
hi , thanks for the video, but could you please direct me that which file in your github is the jupyter notebook for this video? I could not find it. thanks
pretty cool. I have used K-means and DBSCAN to identify electrofacies, but I am still working on a way to optimize this task. It would be grade to see the Well Plots (depth Vs logs) with each point identified by its own cluster.
Thanks Jose. I did have a section of code for displaying the facies data on a log plot but I did not include it in the video. The full plotting code can be found here: towardsdatascience.com/how-to-use-unsupervised-learning-to-cluster-well-log-data-using-python-a552713748b5
Hi, I have one question about scettering in 13:21. Why were 'NHPI' and 'RHOB' written in 'plt.scatter()' when all calculations were done according to scaled data (I mean 'NHPI_T' and 'RHOB_T')? I am just trying to learn it. Could you please help me?
Using the scaled data within certain algorithms can reduce the effect of different data ranges (e.g feature1 ranges from 0 to 1, and feature2 ranges from 0.1 to 10,000), and scaling can also help speed things up. Some algorithms such as decision trees/random forests don't really need scaling whereas Neural Networks and even clustering can benefit from this process. Plotting the data using the original curves allows us to see how the calculated clusters align with the original data. If we were using scaled data, then the numbers on the axes wouldn't make too much sense for petrophysical interpretation. Hope that helps :)
@@AndyMcDonald42 Yes. It helps. :) Thank you very much. Also I have other question. Is there any way to get information about point in the graph by click using mouse to see which point belongs to which data?
@@ahmetatasever8315 Yes, there certainly us, The plot shown in this video was done with matplotlib, which is used to create a basic and static figure. You could easily swap that out for Plotly, which will have the extra interactivity and give extra info on hover.
Andy, thanks for sharing. I can’t find the notebook for this specific exercise. I am trying to follow along with a different dataset but I am getting an error “name ‘means’ is not defined” when trying to determine the number of clusters.
Hi Timothy, did you manage to resolve this? If not, I would go back and check you have ran all of the cells before trying to determine the number of clusters.
Thanks! I have been doing this on resistivity and seismic values on different profiles in a catchment. However, everytime I get same trend but clusters change in their places. Would like to know about this issue...
I will be jumping between some Python topics and machine learning topics over the future episodes. Is there any particular algorithms you would like to see covered?
Can I say that at the end of the day, the way of interpreting the clusters is kind of subjective especially when the dataset gets more complex? Since the results could vary quite a lot as you apply different clustering algorithms or tuning some of their parameters. So it could be quite subjective, no?
Yes. That is very true. It is down to you or the person doing the interpretation to understand what the cluster may represent. If another person does there own interpretation they may have their own understanding of what the clusters represent
Thank you so much, Andy! I really find your video helpful. I am just wondering whether it would be possible for us to draw the scatter plot in multi-dimensions? Cuz I followed all of your steps but could not continue the step after the elbow plot when using my 500 columns dataframe.
Thanks Patty. You would only be able to draw the scatter plot up to 3 dimensions (X, Y and Z). However, you could look at using Seaborn's Pairplot to view 2d scatter plots of each of the variables versus the others: th-cam.com/video/D5DPZyge31g/w-d-xo.html I would be wary though of using 500 features with this plot as it will become unwieldy. I would be asking myself the following in your situation: - Do I require all 500 columns? - Are all of the columns relevant? - Can I reduce them manually or look at algorithms such as PCA to reduce the dimensionality of the dataset.
Thank you so much for this video. I downloaded the data you used and found a negative relationship between RHOB and NPHI. Can tell me how your scatterplot shows a positive relationship between them? Thank you.
No problem. You are correct that NPHI and RHOB are usually anti-correlated. In petrophysics, we normally display RHOB on an inverted scale, often on the Y-axis. As RHOB values get lower, we likely have a higher porosity, and the values will plot higher up on the y-axis. For higher NPHI (neutron porosity) values, the points will plot further to the right. If we have a case where both NPHI and RHOB are high, they will then plot in the top right. It's a nice and easy way to visualise and identify potential reservoir intervals.
@@AndyMcDonald42 Thank you so much. I am using it to cluster customer data, but I wanted to make sure I could replicate yours before trying. Thank you again for the explanation and such an awesome tutorial.
In the .fit() call at 12:00 you would pass in more variables. I have just used 2 for this example to illustrate what the output is like. Hope that helps :)
Sorry for the late reply. I realised I hadn't uploaded the file to the repo. You can find it here: github.com/andymcdgeo/Petrophysics-Python-Series It is Notebook 18.
I'm working on clustering energy consumption profiles of a group of households, how should the starting dataset be structured? For each apartment I'm given the annual energy consumption profile (15 minutes frequency for 1 year), the number of appliances and the number of rooms
Sounds like an interesting task 🙂 If I understand correctly, you have a continuous variable for the energy consumption and then fixed variables for the rest? Have you considered clustering based on the profiles alone and grouping them into something like high energy users and low energy users or early birds and night owls? After that you could then try to use the other properties to gain more insights
hello Andy, thanks for well-explained session,but on the final part can you assist to explain as to which features or measures differentiate one cluster from other,Thanks again
Thanks Dominic. One way would be to use a facet grid plot from seaborn and split by the clusters. You could then view the data by histograms, scatter plots and other plot types. That way you can see how the data features vary per cluster
I have trouble using kemans.labels_ at the end it keeps showing this error: 'numpy.ndarray' object has no attribute 'labels_' can someone help me with this? Thank you!
Andy, i get some NaN value on the datasets.. and then when i try to run the "df.dropna(inplace = True)", all of the datasets become empty (zero). How to handle this? Thankyou
I believe this may have been my original notebook. It contains much more detail than what I covered in the video. I hope this helps. github.com/andymcdgeo/Petrophysics-Python-Series/blob/master/18%20-%20Unsupervised%20Clustering%20for%20Lithofacies.ipynb
![K-means Clustering From Scratch In Python [Machine Learning Tutorial]](http://i.ytimg.com/vi/lX-3nGHDhQg/mqdefault.jpg)
![[UNCUT]“ฅนตื่นธรรม”ปะทะ“หมอปลา-ปู่มหามุนี-อั๋นโอกิ”สะเทือนกระทรวงเวทมนตร์ Iคนดังนั่งเคลียร์I10 ตค.67](http://i.ytimg.com/vi/peMD_SAkwNc/mqdefault.jpg)
Thanks ! I am geoscientist just starting my data sciences journey and I find your videos very helpful
Please can you help me I want to know more about data sciences applying in geosciences
Your fluency and skill, simply superb! Keep it up!
Thank you! 😃
hi. thank you for this wonderful tutorial. where do you recommend choosing data sets from?
That was the best explanation what i watch for KClustering thank you 😊
Great presentation. The clearest I've seen on TH-cam, to date. 👍
Very nice, simple, clear and to the point. Thank you for sharing.
Amazing video you have put together here. I enjoyed how clear you were as well as the pace you took to go through the steps and explain everything. I am new to this kind of thing so does anyone have resources on where I can learn how to interpret cluster graphs
Thank you! The example script is a huge help
Excellent tutorial! Thank you very much for your time
very useful thank you! I'm midway through a data analysis apprenticeship and this helped me alot!
You're very welcome! I am glad to hear it has been helpful.
Explained this better than my professor. Big W
This was such a fantastic tutorial, thank you for putting quality content out there.
Glad you liked it!
Andy - Your videos are very helpful and informative! Thank you!
Glad you like them! Thanks!
Hi Andy, this was a great tutorial as it's something I would like to try on a csv file with various metrics in the design of a pharmaceutical. I have one question though: I will be wanting to use 5-7 columns on the csv file for clustering - how do you go about visually representing this? I can't think of a good way to do it. Thanks!
Precise and clear👍👍plz explain naive based, Support vector machine & decision tree as well
hi , thanks for the video, but could you please direct me that which file in your github is the jupyter notebook for this video? I could not find it. thanks
Thanks buddy, your lesson helped me a lot
pretty cool. I have used K-means and DBSCAN to identify electrofacies, but I am still working on a way to optimize this task.
It would be grade to see the Well Plots (depth Vs logs) with each point identified by its own cluster.
Thanks Jose. I did have a section of code for displaying the facies data on a log plot but I did not include it in the video. The full plotting code can be found here: towardsdatascience.com/how-to-use-unsupervised-learning-to-cluster-well-log-data-using-python-a552713748b5
great tutorial, thank you.
Excellent presentation and explanation
is there a place from where I see the code you have written for this as that would help me in learning. Thanks
do we only use 2 features of a data while using k means clustering or did you do it for visualization purposes?
Hi, I have one question about scettering in 13:21. Why were 'NHPI' and 'RHOB' written in 'plt.scatter()' when all calculations were done according to scaled data (I mean 'NHPI_T' and 'RHOB_T')? I am just trying to learn it. Could you please help me?
Using the scaled data within certain algorithms can reduce the effect of different data ranges (e.g feature1 ranges from 0 to 1, and feature2 ranges from 0.1 to 10,000), and scaling can also help speed things up. Some algorithms such as decision trees/random forests don't really need scaling whereas Neural Networks and even clustering can benefit from this process.
Plotting the data using the original curves allows us to see how the calculated clusters align with the original data. If we were using scaled data, then the numbers on the axes wouldn't make too much sense for petrophysical interpretation.
Hope that helps :)
@@AndyMcDonald42 Yes. It helps. :) Thank you very much. Also I have other question. Is there any way to get information about point in the graph by click using mouse to see which point belongs to which data?
@@ahmetatasever8315 Yes, there certainly us, The plot shown in this video was done with matplotlib, which is used to create a basic and static figure. You could easily swap that out for Plotly, which will have the extra interactivity and give extra info on hover.
@@AndyMcDonald42 Thank you again :)
Andy, thanks for sharing. I can’t find the notebook for this specific exercise. I am trying to follow along with a different dataset but I am getting an error “name ‘means’ is not defined” when trying to determine the number of clusters.
Hi Timothy, did you manage to resolve this?
If not, I would go back and check you have ran all of the cells before trying to determine the number of clusters.
Thanks! I have been doing this on resistivity and seismic values on different profiles in a catchment. However, everytime I get same trend but clusters change in their places. Would like to know about this issue...
Thanks again for the content, Andy! You're a great teacher!
Thanks Allan. Glad to hear you are enjoying the content.
Solid video :)
Btw, where is your accent from?
Hey! great video, only one question. What if I want to set my own centroids?
Hi Andy I think you start machine learning topic and it's my favorite topic thank you 🙏🙏
I will be jumping between some Python topics and machine learning topics over the future episodes. Is there any particular algorithms you would like to see covered?
Can I say that at the end of the day, the way of interpreting the clusters is kind of subjective especially when the dataset gets more complex? Since the results could vary quite a lot as you apply different clustering algorithms or tuning some of their parameters. So it could be quite subjective, no?
Yes. That is very true. It is down to you or the person doing the interpretation to understand what the cluster may represent. If another person does there own interpretation they may have their own understanding of what the clusters represent
Thank you so much, Andy! I really find your video helpful. I am just wondering whether it would be possible for us to draw the scatter plot in multi-dimensions? Cuz I followed all of your steps but could not continue the step after the elbow plot when using my 500 columns dataframe.
Thanks Patty.
You would only be able to draw the scatter plot up to 3 dimensions (X, Y and Z). However, you could look at using Seaborn's Pairplot to view 2d scatter plots of each of the variables versus the others: th-cam.com/video/D5DPZyge31g/w-d-xo.html
I would be wary though of using 500 features with this plot as it will become unwieldy.
I would be asking myself the following in your situation:
- Do I require all 500 columns?
- Are all of the columns relevant?
- Can I reduce them manually or look at algorithms such as PCA to reduce the dimensionality of the dataset.
Absolutely useful. Thank you Andy
Great to hear!
Thank you so much for this video. I downloaded the data you used and found a negative relationship between RHOB and NPHI. Can tell me how your scatterplot shows a positive relationship between them? Thank you.
No problem. You are correct that NPHI and RHOB are usually anti-correlated. In petrophysics, we normally display RHOB on an inverted scale, often on the Y-axis. As RHOB values get lower, we likely have a higher porosity, and the values will plot higher up on the y-axis. For higher NPHI (neutron porosity) values, the points will plot further to the right. If we have a case where both NPHI and RHOB are high, they will then plot in the top right. It's a nice and easy way to visualise and identify potential reservoir intervals.
@@AndyMcDonald42 Thank you so much. I am using it to cluster customer data, but I wanted to make sure I could replicate yours before trying. Thank you again for the explanation and such an awesome tutorial.
Thank you Andy, great video! What if I want to cluster more than 2 variables?
In the .fit() call at 12:00 you would pass in more variables. I have just used 2 for this example to illustrate what the output is like.
Hope that helps :)
I’ve searched for this file in the github repository and I didn’t find this tutorial’s code file
Okay. So how to draw conclusion from these clusters ? I mean, what are your insights from this model ?
Thank you Andy! I just want to ask you where can I find this notebook to download and work with it? Thanks again!
Sorry for the late reply. I realised I hadn't uploaded the file to the repo. You can find it here: github.com/andymcdgeo/Petrophysics-Python-Series
It is Notebook 18.
@@AndyMcDonald42 thank you!! Please, keep on doing videos like this, I've been learning a lot!
You are a hero!
Amazing video, thank you Sir
I'm working on clustering energy consumption profiles of a group of households, how should the starting dataset be structured?
For each apartment I'm given the annual energy consumption profile (15 minutes frequency for 1 year), the number of appliances and the number of rooms
Sounds like an interesting task 🙂
If I understand correctly, you have a continuous variable for the energy consumption and then fixed variables for the rest?
Have you considered clustering based on the profiles alone and grouping them into something like high energy users and low energy users or early birds and night owls?
After that you could then try to use the other properties to gain more insights
Maybe have a look at time series clustering techniques for grouping the profiles
Could you please share the link to get the dataset?
I have problem when trying calculate using excel, the result is different with code, what can i do to fix it?
THANKS YOUUUUU AHHHHH SO HAPPY I DID IT
hello Andy, thanks for well-explained session,but on the final part can you assist to explain as to which features or measures differentiate one cluster from other,Thanks again
Thanks Dominic.
One way would be to use a facet grid plot from seaborn and split by the clusters. You could then view the data by histograms, scatter plots and other plot types. That way you can see how the data features vary per cluster
@@AndyMcDonald42 thank Andy,this is useful,I real appriciate
thank you andy for your sharing 🙏🙏
My pleasure
An error is raised after writing (kmeans_3) while plotting (NPHI vs. RHOB)
I have trouble using kemans.labels_ at the end it keeps showing this error: 'numpy.ndarray' object has no attribute 'labels_' can someone help me with this? Thank you!
very helpful . If you could use example that can be easily understandable for non-science community would be extra helpful!!!
What a great tutorial, thanks a lot🥰🥰
Glad you like it!
Yasss! A fellow Scot!!!
Andy, i get some NaN value on the datasets.. and then when i try to run the "df.dropna(inplace = True)", all of the datasets become empty (zero). How to handle this? Thankyou
I would check if one or more columns are entirely nan.
i liked it, had to hit that belllll
Thanks alot for your helpful videos..
Glad you like them!
Thank you, Andy, I could not find the notebook in your github.
I believe this may have been my original notebook. It contains much more detail than what I covered in the video. I hope this helps.
github.com/andymcdgeo/Petrophysics-Python-Series/blob/master/18%20-%20Unsupervised%20Clustering%20for%20Lithofacies.ipynb
I cannot find notebook file of this video in your git
Thank you so much !!
sir, how to clustering data 2d with size(512,512), please help me sir tq
Excellent thanks
You are welcome
how to create input and output lines? pls help
great content
Thanks a lot!
No problem 👍
thanks a lot
Too good
It keeps saying name means not defined :(
finally, a non-indian accent speaker