Hi sir, thank you for this great tutorial! I am struggling trying to estimate individual home ranges for marsupial populations and I was wondering if you would suggest if I should set up the analysis tools differently, as individuals have very different points density (as few how them have been predated sooner, so I have actually less points for them). When I get the kernel density I have very different values for individuals (like for example 0 to 2167 for one, 0 to 42861 for another) and it becomes really difficult for me to decide how to select the contour lines because they obviously reflects different information and I cannot (can't I?) pick the same density level as a standard. I was thinking about approaching the data in percentage, so selecting contour intervals by 10% (so it is individual-specific) and then select the third line of the contour or similar. Would you recommend something like this or might you have any other advice? Thank you in advance!
Yeah, I hear exactly what you are saying (and my apologies for taking so long to respond!). There is a solution (described in link below), but this is definitely a weakness of the ArcGIS density tools. So far they haven't offered a way to automatically convert the density values to anything describing the relative volume under the kernel density surface, so it is very difficult to choose a contour level that will describe the same activity level or usage for different animals. I've gotten around this in ArcMap by writing custom ArcObjects code to make true "Proportion Under Surface" rasters, from which it is easy to generate contours at any percentage-of-use level. You can get a good approximation of this without using any special code, though. It's a little tedious but you can do it with a combination of ArcGIS Pro and Excel. The basic strategy is to first identify the total volume under the curve, and then determine what density levels correspond with the percentile levels you are interested in. I've written up some instructions with illustrations here: docs.google.com/document/d/12W_8TYul1P35h17mZNc7HYBsg2Irk5dm/edit?usp=sharing&ouid=113601434494391531497&rtpof=true&sd=true Hope this helps! Take care - Jeff
Good question Alexandria, and this is a weakness in the ArcGIS version of Kernel Densities. This isn't an option in the basic tool. This is something I've only been able to do in code, and as of today (December 2022) I haven't figured out how to do it in ArcGIS Pro. I'm still doing it in ArcMap. But the basic strategy is this: 1) Make an array of all the Kernel Density values in your Kernel Density raster. If your raster is 1,000 rows by 1,000 columns, then this means you'll have an array of 1,000,000 values. 2) Add them all up so you have a single sum. 3) Sort them in order of smallest to largest, and then... 4) Step through the array from the smallest to the largest and incrementally add them up. So, for example, when you've hit the 1,000th value in the array, you'll now have the sum of the first 1,000 values. 5) At each increment, divide that running sum by the total sum. Once you cross 5%, then you've got your 95% threshold value (meaning you've eliminated the lowest 5% of the density values). For example, if the density value from your sorted array is 100 pts per sq. km when your running sum crosses 5%, then you'll use the value "100" as your density level that captures 95% of the points. Once you cross 50%, then you've got your 50% density threshold. And of course you can choose any threshold values you like; there's no set-in-stone rule that says 50% and 95% are the only defensible options. 6) Finally, you just generate contours at those threshold values. It sounds like a lot of work and a hassle, and the first time it is. But if you get it down in code, it's as easy to run as any other tool. And you could speed it up and simplify it a bit if you're willing to accept an estimate of the threshold values instead of the exact values. In this alternative, you could convert your ArcGIS Kernel Density to an integer format so you can get an attribute table of values, and then you could export the table to a CSV file or something. This table will contain the density values plus a "Count" field, so you'll know how many cases of each value there are. Therefore it's a table of values plus weights. Then you could just follow the steps above on your table of values, or maybe bring it into R to get the percentile values you're interested in. Then, the last step will again be to generate contours at those final threshold values. Sorry I don't have a simpler answer Alexandria! Most of the best things you can do in ArcGIS come down to writing code, I've found. It's worth the effort when you can do things like this, though. Hope this helps! Take care -
@@JennessEnterprises Excellent video, thanks for making it. How do you code the 50% and 95% options? python? would you be willing to share that code? I've used R and it looks something like this but would like to see how you do it in python, if that's what you used. Many thanks. library(adehabitatHR) library(tidyverse) library(sp) library(rgdal) library(scales) library(glue) library(rgeos) df
Thanks! I'm afraid I haven't used Adehabitat enough to be familiar with it, but I know the author (Clement Calenge) has made a really popular and useful tool. Adehabitat even won an award from the Wildlife Society's Spatial Ecology and Telemetry Working Group, so I'm pretty confident the method you describe should be a good one. Thank you for sharing! One of these days I want to explore that tool more. I'm doing something simpler in ArcMap. I'm just identifying the innermost X% of the volume underneath the kernel density surface, using the only kernel shape that ArcGIS offers. The algorithm is not that complicated, but I'm afraid it is not in Python. This was written for ArcMap, using VBA to interact with ArcObjects functions. ArcObjects gives much deeper access into the internal ArcMap functions than Python so I've always written my code using those libraries. If you're interested, you can download the code here: drive.google.com/file/d/1kPWk1Ddygv02cpkeM8ERhVLWweA_XxpL/view?usp=share_link A few caveats: First, since this was written for ArcMap, it won't come close to working in ArcGIS Pro. It would help a lot if you are familiar with VBA or Visual Basic 6, and ArcObjects. Second, I wrote this function specifically to create percentile contours for a large number of rasters, and also to do a few other things. Therefore there's a lot of stuff in there that has nothing to do with what you're interested in. I tried to delete most of the major extraneous code, but I'm afraid I don't really have time to rewrite a completely clean function for you. If you are comfortable with code, I'm hoping you'll be able to make sense of this. If you open the function in Notepad++, then the code that extracts the non-zero raster density values and puts them into a numeric (double-precision) array extents from line 147 to 191. Then that array needs to be sorted. Lines 201 - 242 creates a 2-dimensional reference array containing the probability levels I want to calculate (which would correspond with the 50% and 95% you mention; I actually have it calculating every 5%. Then lines 245 - 259 go through the sorted list of raster values and identify what the density level is at each of my probability levels. The remaining code creates the contour feature classes and symbolizes the raster. Hope this helps! Take care - Jeff
Hi Shahrul, Sure thing, and good question! There are actually a few options for the output units when you run the tool, but the most common way (at least in my experience) is as "Density" values. This means the value gives you an estimate of the number of locations per unit area within the neighborhood around each cell. If you're a wildlife biologist like me, and you go and map a bunch of spotted owl locations, then a cell value of "10" might be interpreted as "10 owls per square kilometer within the local region surrounding that cell". This can be a little confusing, though, because it's only loosely related to the actual number of locations on the landscape, and points are counted differently based on how close they are to the cell center. For example, a raster cell might have 5 spotted owl locations within your search radius (aka "bandwidth"), but the actual kernel density value will be higher if those 5 owls are close to the cell center than if they are far away. The "Kernel" is the algorithm we use to count points higher if they are closer to the cell center. Picture a three-dimensional bell-shaped curve positioned over the cell center. If an owl is close to the cell center, then the height of that bell-shaped curve at the owl's location will be high, and therefore that owl will contribute more to the final density value. Owls farther away will sit lower on the tails of the bell-shaped curve, and consequently contribute less to the final density value. ArcGIS only gives you one option for the kernel shape, and it looks a lot like the normal bell-shaped curve. One important difference is that the ArcGIS bell-shaped kernel drops off to zero at the bandwidth distance, while the normal curve never quite reaches zero. Other software packages like R let you choose different kernel shapes (for example, see www.researchgate.net/figure/Various-kernel-shapes-for-Kernel-Density-Estimation_fig2_283575639), which would change the overall kernel density surface a little. And at the end, the final density values are rescaled according to the area units you prefer. If you generate a density surface using Hectares as your area unit, and then generate another density surface on the same data using Square Kilometers as your area unit, then you'll find the density values in your first raster are all exactly 100x the values in the second raster because there are exactly 100 hectares in a square kilometer. If you really want to get into the weeds on this, take a look at B. W. Silverman's book "Density Estimation for Statistics and Data Analysis" (Silverman 1986). He discusses different kernel shapes you can use, and the Esri documentation on Kernel Densities mentions that they use Silverman's equation 4.5, on p. 76 of his manuscript. I hope this helps Shahrul! Take care - Jeff - - - - - - - - Silverman, B. W. 1986. Density estimation for statistics and data analysis. Volume 26. Monographs on Statistics and Applied Probability, CRC press.
At this point I only believe. To know I'd have to go find the rasters I generated in that lab exercise and look at the metadata, and that would take some time. The parameters I used would be recorded in the geoprocessing history. I suspect that you really care more about the units of your own analysis than my example in this video, so remember that you choose the output units you want when you run the tool. So there really is no mystery about the output, other than the general complexity of kernel density analysis itself. Of course, it is easy to forget what units you chose after you've run the tool (as well as the bandwidth you chose, or any other parameter). This is definitely a common problem. Fortunately all that information is stored in the kernel density raster metadata. Regarding the metadata: ArcGIS Pro stores a great history of all geoprocessing operations that have been applied to a dataset, in a section called "Geoprocessing History". However, by default ArcGIS Pro only shows you a small subset of all the metadata available on that dataset, and that subset does not include the geoprocessing history. You need to change your metadata style to anything other than "Item Description" in order to see the geoprocessing history (see roughly 5:50 in the video th-cam.com/video/4IVuyEqtOeA/w-d-xo.html for a demonstration on how to change your metadata style). I hope this helps Rajat! Take care - Jeff
Hi Farhan, I'm afraid I'm not familiar enough with QGIS to know! I'm sure they do Kernel Density analysis, but I don't know if they tell you the kernel density results in terms of probabilities. You might take a look at sporock's comment above, though, to see a solution for R. Sorry I don't have a better answer for you! Take care - Jeff
Hi sir, thank you for this great tutorial! I am struggling trying to estimate individual home ranges for marsupial populations and I was wondering if you would suggest if I should set up the analysis tools differently, as individuals have very different points density (as few how them have been predated sooner, so I have actually less points for them). When I get the kernel density I have very different values for individuals (like for example 0 to 2167 for one, 0 to 42861 for another) and it becomes really difficult for me to decide how to select the contour lines because they obviously reflects different information and I cannot (can't I?) pick the same density level as a standard. I was thinking about approaching the data in percentage, so selecting contour intervals by 10% (so it is individual-specific) and then select the third line of the contour or similar. Would you recommend something like this or might you have any other advice?
Thank you in advance!
Yeah, I hear exactly what you are saying (and my apologies for taking so long to respond!). There is a solution (described in link below), but this is definitely a weakness of the ArcGIS density tools. So far they haven't offered a way to automatically convert the density values to anything describing the relative volume under the kernel density surface, so it is very difficult to choose a contour level that will describe the same activity level or usage for different animals. I've gotten around this in ArcMap by writing custom ArcObjects code to make true "Proportion Under Surface" rasters, from which it is easy to generate contours at any percentage-of-use level.
You can get a good approximation of this without using any special code, though. It's a little tedious but you can do it with a combination of ArcGIS Pro and Excel. The basic strategy is to first identify the total volume under the curve, and then determine what density levels correspond with the percentile levels you are interested in. I've written up some instructions with illustrations here:
docs.google.com/document/d/12W_8TYul1P35h17mZNc7HYBsg2Irk5dm/edit?usp=sharing&ouid=113601434494391531497&rtpof=true&sd=true
Hope this helps! Take care -
Jeff
I've read in some papers that they chose to use a 50% (core) and 95% (range) in their KDE, how would you specify these in this procedure?
Good question Alexandria, and this is a weakness in the ArcGIS version of Kernel Densities. This isn't an option in the basic tool.
This is something I've only been able to do in code, and as of today (December 2022) I haven't figured out how to do it in ArcGIS Pro. I'm still doing it in ArcMap. But the basic strategy is this:
1) Make an array of all the Kernel Density values in your Kernel Density raster. If your raster is 1,000 rows by 1,000 columns, then this means you'll have an array of 1,000,000 values.
2) Add them all up so you have a single sum.
3) Sort them in order of smallest to largest, and then...
4) Step through the array from the smallest to the largest and incrementally add them up. So, for example, when you've hit the 1,000th value in the array, you'll now have the sum of the first 1,000 values.
5) At each increment, divide that running sum by the total sum. Once you cross 5%, then you've got your 95% threshold value (meaning you've eliminated the lowest 5% of the density values). For example, if the density value from your sorted array is 100 pts per sq. km when your running sum crosses 5%, then you'll use the value "100" as your density level that captures 95% of the points. Once you cross 50%, then you've got your 50% density threshold. And of course you can choose any threshold values you like; there's no set-in-stone rule that says 50% and 95% are the only defensible options.
6) Finally, you just generate contours at those threshold values.
It sounds like a lot of work and a hassle, and the first time it is. But if you get it down in code, it's as easy to run as any other tool. And you could speed it up and simplify it a bit if you're willing to accept an estimate of the threshold values instead of the exact values. In this alternative, you could convert your ArcGIS Kernel Density to an integer format so you can get an attribute table of values, and then you could export the table to a CSV file or something. This table will contain the density values plus a "Count" field, so you'll know how many cases of each value there are. Therefore it's a table of values plus weights. Then you could just follow the steps above on your table of values, or maybe bring it into R to get the percentile values you're interested in. Then, the last step will again be to generate contours at those final threshold values.
Sorry I don't have a simpler answer Alexandria! Most of the best things you can do in ArcGIS come down to writing code, I've found. It's worth the effort when you can do things like this, though.
Hope this helps! Take care -
@@JennessEnterprises Excellent video, thanks for making it. How do you code the 50% and 95% options? python? would you be willing to share that code? I've used R and it looks something like this but would like to see how you do it in python, if that's what you used. Many thanks.
library(adehabitatHR)
library(tidyverse)
library(sp)
library(rgdal)
library(scales)
library(glue)
library(rgeos)
df
Thanks!
I'm afraid I haven't used Adehabitat enough to be familiar with it, but I know the author (Clement Calenge) has made a really popular and useful tool. Adehabitat even won an award from the Wildlife Society's Spatial Ecology and Telemetry Working Group, so I'm pretty confident the method you describe should be a good one. Thank you for sharing! One of these days I want to explore that tool more.
I'm doing something simpler in ArcMap. I'm just identifying the innermost X% of the volume underneath the kernel density surface, using the only kernel shape that ArcGIS offers. The algorithm is not that complicated, but I'm afraid it is not in Python. This was written for ArcMap, using VBA to interact with ArcObjects functions. ArcObjects gives much deeper access into the internal ArcMap functions than Python so I've always written my code using those libraries.
If you're interested, you can download the code here: drive.google.com/file/d/1kPWk1Ddygv02cpkeM8ERhVLWweA_XxpL/view?usp=share_link
A few caveats:
First, since this was written for ArcMap, it won't come close to working in ArcGIS Pro. It would help a lot if you are familiar with VBA or Visual Basic 6, and ArcObjects.
Second, I wrote this function specifically to create percentile contours for a large number of rasters, and also to do a few other things. Therefore there's a lot of stuff in there that has nothing to do with what you're interested in. I tried to delete most of the major extraneous code, but I'm afraid I don't really have time to rewrite a completely clean function for you. If you are comfortable with code, I'm hoping you'll be able to make sense of this.
If you open the function in Notepad++, then the code that extracts the non-zero raster density values and puts them into a numeric (double-precision) array extents from line 147 to 191. Then that array needs to be sorted.
Lines 201 - 242 creates a 2-dimensional reference array containing the probability levels I want to calculate (which would correspond with the 50% and 95% you mention; I actually have it calculating every 5%.
Then lines 245 - 259 go through the sorted list of raster values and identify what the density level is at each of my probability levels.
The remaining code creates the contour feature classes and symbolizes the raster.
Hope this helps! Take care -
Jeff
Hi, I have a hard time understanding the value when kernel density is run. What does the value refer to actually?
Hi Shahrul,
Sure thing, and good question! There are actually a few options for the output units when you run the tool, but the most common way (at least in my experience) is as "Density" values. This means the value gives you an estimate of the number of locations per unit area within the neighborhood around each cell. If you're a wildlife biologist like me, and you go and map a bunch of spotted owl locations, then a cell value of "10" might be interpreted as "10 owls per square kilometer within the local region surrounding that cell".
This can be a little confusing, though, because it's only loosely related to the actual number of locations on the landscape, and points are counted differently based on how close they are to the cell center. For example, a raster cell might have 5 spotted owl locations within your search radius (aka "bandwidth"), but the actual kernel density value will be higher if those 5 owls are close to the cell center than if they are far away.
The "Kernel" is the algorithm we use to count points higher if they are closer to the cell center. Picture a three-dimensional bell-shaped curve positioned over the cell center. If an owl is close to the cell center, then the height of that bell-shaped curve at the owl's location will be high, and therefore that owl will contribute more to the final density value. Owls farther away will sit lower on the tails of the bell-shaped curve, and consequently contribute less to the final density value.
ArcGIS only gives you one option for the kernel shape, and it looks a lot like the normal bell-shaped curve. One important difference is that the ArcGIS bell-shaped kernel drops off to zero at the bandwidth distance, while the normal curve never quite reaches zero. Other software packages like R let you choose different kernel shapes (for example, see www.researchgate.net/figure/Various-kernel-shapes-for-Kernel-Density-Estimation_fig2_283575639), which would change the overall kernel density surface a little.
And at the end, the final density values are rescaled according to the area units you prefer. If you generate a density surface using Hectares as your area unit, and then generate another density surface on the same data using Square Kilometers as your area unit, then you'll find the density values in your first raster are all exactly 100x the values in the second raster because there are exactly 100 hectares in a square kilometer.
If you really want to get into the weeds on this, take a look at B. W. Silverman's book "Density Estimation for Statistics and Data Analysis" (Silverman 1986). He discusses different kernel shapes you can use, and the Esri documentation on Kernel Densities mentions that they use Silverman's equation 4.5, on p. 76 of his manuscript.
I hope this helps Shahrul! Take care -
Jeff
- - - - - - - -
Silverman, B. W. 1986. Density estimation for statistics and data analysis. Volume 26. Monographs on Statistics and Applied Probability, CRC press.
can you please mention what will be the unit of the output density data?
Certainly Rajat. I believe these videos calculated density in units of Points per Square Kilometer.
@@JennessEnterprises as do I, but there is a difference between believing and knowing, So do you believe or know?
At this point I only believe. To know I'd have to go find the rasters I generated in that lab exercise and look at the metadata, and that would take some time. The parameters I used would be recorded in the geoprocessing history.
I suspect that you really care more about the units of your own analysis than my example in this video, so remember that you choose the output units you want when you run the tool. So there really is no mystery about the output, other than the general complexity of kernel density analysis itself.
Of course, it is easy to forget what units you chose after you've run the tool (as well as the bandwidth you chose, or any other parameter). This is definitely a common problem. Fortunately all that information is stored in the kernel density raster metadata.
Regarding the metadata: ArcGIS Pro stores a great history of all geoprocessing operations that have been applied to a dataset, in a section called "Geoprocessing History". However, by default ArcGIS Pro only shows you a small subset of all the metadata available on that dataset, and that subset does not include the geoprocessing history. You need to change your metadata style to anything other than "Item Description" in order to see the geoprocessing history (see roughly 5:50 in the video th-cam.com/video/4IVuyEqtOeA/w-d-xo.html for a demonstration on how to change your metadata style).
I hope this helps Rajat! Take care -
Jeff
@@JennessEnterprises surely did help. Thank you Jeff.
JUST BRILLIANT! THANKS FOR SHARING
Thank you so much Mariola! Your comment lightened my day!
How I can convert HGBDR format to Tif in arc gis
Hmm, I'm not sure! I'm not familiar with that format. Can you tell me more about it?
sir, can this tool work for regular arcgis or qgis?
Hi Farhan,
I'm afraid I'm not familiar enough with QGIS to know! I'm sure they do Kernel Density analysis, but I don't know if they tell you the kernel density results in terms of probabilities. You might take a look at sporock's comment above, though, to see a solution for R.
Sorry I don't have a better answer for you! Take care -
Jeff