Market data seems to be hard to get as "website" or "refreshable picture", I mean like finviz heat maps or sectors. I am trying other sites one by one but I am mostly interested in whole charts not price points.
i've tried scraping the Total pages printed off my HP printer webinterface as it is not an entity in the IPP integration, but for 1. it uses a PIN login, couldn't get the webform publish to work it, 2nd, the css selector off the site doesn't get me results on the inspector console or HA (lots off css error on the page reported by firefox inspector). I'm working on a video on how i managed to fetch the number thru SNMP...
Great video! Made a scraper for swedish namedays. A tip for getting the correct selector in the inspection view (which I had some trouble with): right click the selector you want to scrape, pick Copy and then Copy selector. Then you get the selector correctly. Like so: body > div.header > div.text-vertical-center > h1
Enorm bedankt voor je video. Het heeft me geholpen de scrape sensor eindelijk werkende te krijgen en heeft me meer inzicht in deze sensor gegeven. Een dikke pluim en duim omhoog voor je goede werk :)
Hi Ed, as usual, very well explained. As you said in the video, I'll play with that a bit to get the hang of it, but your video is a good guide. Thanks for sharing this.
now that I have a temperature sensor, brought it outside and realized the humidity values for my area are way off. so I'm scrapping it from a local weather station I just found that actually matches my readings and my temp sensor can get back indoors.
I had already come up with such a plan and was very happy when I saw your video. Only my energy supplier (Greenchoice) does not offer a rate sheet via the public internet, I must first log in to the My Greenchoice environment to view the prices of my contract. I've tried it myself with the multiscraper but can't figure it out. Could you also make a video for such situations?
I wanted a calendar for USPS Holidays and so I'm scraping a list from a website and using that to flag when it's a USPS Holiday or not. I couldn't find an iCal file that didn't include Valentine's Day which is ordinarily a delivery day so I had to scrape.
Great...I have few of my scrappers running in node red...which gives more flexibility to write manual codes in funtion node.. I mostly scrap from websites which requires logins... Could be great if you do 2nd part of this video.. And include websites which require login authentication
Most modern sites do not use basic or digest. For token-based or oauth, you'd better to login manually with a browser and copy the token you got, and use that in your scrappers.
I need a way to get dynamic data or rather a dynamic amount of items with sub attributes I am trying to get data from a list that has a huge amount of items
Here in South Africa we have frequent power outages due to the power company unable to meet the demand, those outages are published on a website to help people plan around the power outages. Knowing when my area will not have power can help me manage the remaining power left on my solar system.
it's fantastic , thanks I have scraped the bus time table real time.. but how to access to USERNAME and PASSWORD website? I tried so many time but no result
@@SmartHomeJunkie Thanks for your quick response. But how do I enable debugging logs from the multi scraper. Where to put the logger command's in the configuration files?
Thanks for the video, got me going on scraping. But where is the split function shown around 9:00 documented? I'm trying to better understand it so I can use it in other use cases. Thanks!
ok, so i'm very new to HA. I understand most of this and getting ready to try it. Could you show how you would use/display said information scraped from a site? (or point me to the video that you've already done)
I love your video, I am trying to preproduce it. 1) Why do you only start with "Chart content" ? 2) Why can you use "col-md-6" and not "col-xs-12" as you do further below with "media-body" ? I am so confused.
Same :) VERY new to scraping, and trying to understand where you start from in a structure and why there is # and . in front of some of the tags in example two... I keep on searching for clues :)
I am struggling with exactly the same sort of thing. Also if you use the copy selector for the field (which several sites suggest is the easy way to parse it) you get: "#chart-content > div:nth-child(1) > div:nth-child(1) > div > a > span > h4" - so confused!!!
the element I want to scrape is this one 32 °C the value I want to scrape is the degrees, once the sun is gone the degrees wil be - I have watched the video several times but cannot find out how to do the unit of measurement and value template as there is no letter or text to do the split as in your examples
Thank you so much. Very interesting! I have a question because I’m trying to read some energy data but I have some problem. The website request a password to enter and then insert the data. I’m able to enter with my credential but when I read the data the value is 01/01/2015, very old! This is very strange because when I use chrome to copy selector I see the correct data! How can I insert the actual data before scraping the data from website? Thank you
With the latest release, April 2023, this setup, which worked WONDERFULLY for me, no longer does, as Scrape is now integrated into HA. Can you show me how to use the latest version? The UI is different, and I'm looking to re-create what I already had working perfectly from the HACS version you illustrate here. Basically, we need an update now. :) Thank you as always for your excellent work!
@@SmartHomeJunkie It's been on-warning that the HACS would be removed since December 2022. It's been working nonetheless, but with this update, it's finally dead.
@@SmartHomeJunkie Welp, now that you mention it with a link, I'm not sure any more. LOL. I'll do some digging... Anyway, there's now a Scrape Integration under Settings >> Devices & Services >> Add Integration >> Scrape. No HACS needed for that one. Might be a future video idea for you. I appreciate it again. I'll doublecheck your github link and compare it to what I had. I'm now doubting it was the same thing. 😕
Great video. I never thought of doing that. It sounds very useful. As a different exercise, can you get share prices from somewhere too? I guess this couldn't be done by scraping but I assume there is some way to do it.
Thank you. If there's a website somewhere that shows these prices, then it might be possible. As long as the website contains HTML, it should be possible. Sometimes it's not possible if a website contains a lot of javascript objects that are created on the fly.
Hi, it would be amazing if you considered making a video on how to get data from Home Assistant to display on Kustom KLWP, or into Tasker. Love your videos but can't find anything to help
I "think" you will only see that the entity is unavailable. So, you can create an automation that triggers on the state "unavailable" for that entity and send you a message maybe?
Nice, I have modified the value template to make sure the values are integers and that you can calculate with it. value_template: '{{ value.split("€")[1] | replace(",",".")}}'
@@SmartHomeJunkie Thanks for your fast reply!! Within the first table I can scrape every value using your example, but since my knowledge of css/html is none I don't have a clue as how to point to another table/use the correct css tag on a webpage.
Hello Smart Home Junkie, In the new release of HA 2022/12 a webscraper integration has been added. How is this new integration working compared to HACS multiscraper?
There was already a scrape function in Home Assistant, but it couldn’t scrape multiple values in one call. The HACS version can. I didn’t look into this new version yet. Might be that it supports that now too.
@@SmartHomeJunkie as far as I can see it is possible to create multiple sensors with the 2012/12 scraper integration. I spend some time but was not able te create a working scrape sensor
Hi, thats really cool!, how would you use this method to get past a login screen, I assume there's a way to enter your username and password first and then go to the source page after the login page?
This tutorial has helped me a lot to set up a scrape sensor for scraping the daily Belgian gas price from my energy provider. Unfortunately the have changed the website last Thursday and now I can not get it working again. I am looking into the issue already for 3 days, and cannot find a solution. Although I always had a lot of advice and support from the Home Assistant Facebook pages, scraping is something hardly anyone provides help with. Is there any possibility I could ask your assistance with setting up the sensor again? Thanks in advance. Kind regards.
@@SmartHomeJunkie Ik zal erover nadenken, maar kan je me a u.b. een schatting van het tijdsbestek geven? 75 € exclusief btw is namelijk een enorm bedrag voor iets dat maximaal 10 minuten duurt en al voorbereid is
@@SmartHomeJunkie ik heb via het formulier contact opgenomen met een situatieschets. Voorlopig enkel informatief voor dat ik een finale beslissing maak. Alvast bedankt voor de snelle reactie
Hi@@SmartHomeJunkie, I'm also busy with this, but I have 1 issue here. To see/ scrape our Engie dynamic energy prices, we need to login via our personal Engie account, and here we have already the first problem, they are using CAPTCHA in the backend of the login process. And till now, I'm not able to use Multiscrape for this, because, correct me if I'm wrong, it is not possible with Multiscrape?
What does the scraper put into the entities when the web page changes? Assuming it doesn’t just find something weird, but doesn’t find anything any more? Does it just keep the existing stale data or more usefully some kind of error message?
It depends if the structure of the webpage is still the same. In that case other "false" data will be shown. If the structure changed, then it will probably show either unknown or unavailable.
Can not get this to work at all, it always unavaiable. nothing works tried so many ways. Like copying the entire css selector path and all. While scraping with home automation integrator works flawlessly .
Hi, I need to scrape 4 data energy. One for example is the energy of solar panel that have this selector: #pSolareMon Now I see 600W inside website but I read 0.0kW using scraping. Very strange! For this reason I try to scrape other variable not necessary, for example the date and I noticed that was different! Inside website is present a box to choose the date and probably is necessary set the date before scrape. Boooo!
i tought to myselfe, using a scraping tool i shoud be abel to get my energie price info. looked up a random tutorial. turns out its about energy scraping. Super bedankt, DOE ZO VERDER !
I’d love to be able to scrape tweets from a specific account on Twitter (like my daughters school) that have specific text in them (like “all schools closed” or “paved pickup”).
@@SmartHomeJunkie LOL, you’re telling me. I thought this would be a fairly easy thing with the Twitter integration. But that’s more for posting tweets than compiling/scraping them.
Nice tutorial! To use the prices in your Energy Dashboard the following should be changed/add per identity: unit_of_measurement: "€/kWh" value_template: "{{ value.replace(',','.') }}" I used this ☝for my Powerpeers prices. To get the scraper ('select'), simple got to the site and right click with you mouse on the price. Select 'Inspect' (in Chrome). Now the devtool opens. right click and select 'Copy' > 'Copy selector' and put it to you select: of your entity in the multiscrape.yaml file.
any idea how to solve this? It keeps giving this message when i want to use the gas price for example inside the energy dashboard: Entity has non-numeric state The following entities have a state that cannot be parsed as a number edit: Maybe i got it working now with: value_template: "{{ value.split('€')[1]|replace(',','.') }}"
What data do you (want to) scrape from what websites? Let me know in the comments!
Market data seems to be hard to get as "website" or "refreshable picture", I mean like finviz heat maps or sectors. I am trying other sites one by one but I am mostly interested in whole charts not price points.
i've tried scraping the Total pages printed off my HP printer webinterface as it is not an entity in the IPP integration, but for 1. it uses a PIN login, couldn't get the webform publish to work it, 2nd, the css selector off the site doesn't get me results on the inspector console or HA (lots off css error on the page reported by firefox inspector).
I'm working on a video on how i managed to fetch the number thru SNMP...
Instagram Followers. And I'm not making it... How can I extract it from
@@StereoSatellite Ah yes, I tried that as well. I even tried this with the API, but couldn't get it to work either. 😢
Follow the tesla cars prices in the inventory. I search a lot of time but it don't work...😔
Great video! Made a scraper for swedish namedays. A tip for getting the correct selector in the inspection view (which I had some trouble with): right click the selector you want to scrape, pick Copy and then Copy selector. Then you get the selector correctly. Like so: body > div.header > div.text-vertical-center > h1
This is where I failed - thank you.
That is a perfect tip, thank you so much!
This is an ESSENTIAL tip.
Hi Ed
This is a wonderful tutorial! Very clear and step-by-step with good examples. Thank you so much!
Thank you!
That worked better than expected. Added local sea temperature as a sensor, perfect for swimming season!
Danke!
Please do bring more multi dimension of HA with different use cases like these
Enorm bedankt voor je video. Het heeft me geholpen de scrape sensor eindelijk werkende te krijgen en heeft me meer inzicht in deze sensor gegeven. Een dikke pluim en duim omhoog voor je goede werk :)
Thx!!!
Hi Ed, as usual, very well explained.
As you said in the video, I'll play with that a bit to get the hang of it, but your video is a good guide.
Thanks for sharing this.
now that I have a temperature sensor, brought it outside and realized the humidity values for my area are way off. so I'm scrapping it from a local weather station I just found that actually matches my readings and my temp sensor can get back indoors.
I had already come up with such a plan and was very happy when I saw your video. Only my energy supplier (Greenchoice) does not offer a rate sheet via the public internet, I must first log in to the My Greenchoice environment to view the prices of my contract. I've tried it myself with the multiscraper but can't figure it out. Could you also make a video for such situations?
Hi Robert,
Sometimes it's not possible if you have to log in first. It really depends on how the site is built.
I wanted a calendar for USPS Holidays and so I'm scraping a list from a website and using that to flag when it's a USPS Holiday or not. I couldn't find an iCal file that didn't include Valentine's Day which is ordinarily a delivery day so I had to scrape.
Thank you. You're an excellent teacher. I've learned a lot from your videos. 👍
Thank you for your compliments! 🙏🏻
Great...I have few of my scrappers running in node red...which gives more flexibility to write manual codes in funtion node.. I mostly scrap from websites which requires logins... Could be great if you do 2nd part of this video.. And include websites which require login authentication
You can use logins too. See the manual of multiscrape for this. It's just adding two fields to your scraper.
Most modern sites do not use basic or digest. For token-based or oauth, you'd better to login manually with a browser and copy the token you got, and use that in your scrappers.
I need a way to get dynamic data or rather a dynamic amount of items with sub attributes
I am trying to get data from a list that has a huge amount of items
Here in South Africa we have frequent power outages due to the power company unable to meet the demand, those outages are published on a website to help people plan around the power outages. Knowing when my area will not have power can help me manage the remaining power left on my solar system.
That sounds like a very nice use case. 👍🏻👍🏻
it's fantastic , thanks I have scraped the bus time table real time.. but how to access to USERNAME and PASSWORD website? I tried so many time but no result
That should be possible with this integration too. I will investigate this too!
Great video.
I'm pretty new to HA, so bare over with my question. How can I turn on the logger to see what is wrong. Thanks
Look at Settings > System > Logs
@@SmartHomeJunkie Thanks for your quick response. But how do I enable debugging logs from the multi scraper. Where to put the logger command's in the configuration files?
Thanks for the video, got me going on scraping. But where is the split function shown around 9:00 documented? I'm trying to better understand it so I can use it in other use cases. Thanks!
Thanks. This might help you: www.middlewareinventory.com/blog/ansible-split-examples/
Thanks!
Thanks for all your awesome and helpful videos. Just getting into this and your videos have been so helpful
Is this possible to use in local network? Trying to get data from my solarpanel inverter
ok, so i'm very new to HA. I understand most of this and getting ready to try it. Could you show how you would use/display said information scraped from a site? (or point me to the video that you've already done)
I love your video, I am trying to preproduce it.
1) Why do you only start with "Chart content" ?
2) Why can you use "col-md-6" and not "col-xs-12" as you do further below with "media-body" ? I am so confused.
Same :) VERY new to scraping, and trying to understand where you start from in a structure and why there is # and . in front of some of the tags in example two... I keep on searching for clues :)
I am struggling with exactly the same sort of thing. Also if you use the copy selector for the field (which several sites suggest is the easy way to parse it) you get: "#chart-content > div:nth-child(1) > div:nth-child(1) > div > a > span > h4" - so confused!!!
the element I want to scrape is this one 32 °C the value I want to scrape is the degrees, once the sun is gone the degrees wil be - I have watched the video several times but cannot find out how to do the unit of measurement and value template as there is no letter or text to do the split as in your examples
rightclick (most browser provide this) and copy xpath and paste in the yaml to make it easier?
Yes, that will also work.
Thank you so much. Very interesting!
I have a question because I’m trying to read some energy data but I have some problem.
The website request a password to enter and then insert the data.
I’m able to enter with my credential but when I read the data the value is 01/01/2015, very old!
This is very strange because when I use chrome to copy selector I see the correct data!
How can I insert the actual data before scraping the data from website?
Thank you
Scraping does not work for every site. Sometimes a webpage is build up by Javascript and in those cases it may not work.
With the latest release, April 2023, this setup, which worked WONDERFULLY for me, no longer does, as Scrape is now integrated into HA. Can you show me how to use the latest version? The UI is different, and I'm looking to re-create what I already had working perfectly from the HACS version you illustrate here. Basically, we need an update now. :) Thank you as always for your excellent work!
Probably the HACS integration need an update. Give it a week or so so that the developer can catch up with the update of HA.
@@SmartHomeJunkie It's been on-warning that the HACS would be removed since December 2022. It's been working nonetheless, but with this update, it's finally dead.
@@JasonPhipps Are you sure? I use this one in the video and it's still maintained: github.com/danieldotnl/ha-multiscrape
@@SmartHomeJunkie Welp, now that you mention it with a link, I'm not sure any more. LOL. I'll do some digging...
Anyway, there's now a Scrape Integration under Settings >> Devices & Services >> Add Integration >> Scrape. No HACS needed for that one. Might be a future video idea for you.
I appreciate it again. I'll doublecheck your github link and compare it to what I had. I'm now doubting it was the same thing. 😕
@@JasonPhipps I know, but the HACS scraper is a multiscraper which has a lot of advantages over the standard scrape. But both work.
Thanks a lot! How can I use the value.split to select a value between 2 different characters?
thanks Ed, i need to focus more to learn this, very interesting function to automat
Yeah, there's a lot of information in this video to digest. ;)
Thanks for the video. It's east to understand what you explain.
Thank you for the compliment!
Is it possible for me to connect the code with google home, for example ask google what the 3 top box office movies are ?
Great video. I never thought of doing that. It sounds very useful. As a different exercise, can you get share prices from somewhere too? I guess this couldn't be done by scraping but I assume there is some way to do it.
Thank you. If there's a website somewhere that shows these prices, then it might be possible. As long as the website contains HTML, it should be possible. Sometimes it's not possible if a website contains a lot of javascript objects that are created on the fly.
Hi, it would be amazing if you considered making a video on how to get data from Home Assistant to display on Kustom KLWP, or into Tasker. Love your videos but can't find anything to help
I do not use Android, so I think I won't be able to do that. If it's even possible.
What would be a good way to detect when a scraper is failing so a notification can be shown in HA?
I "think" you will only see that the entity is unavailable. So, you can create an automation that triggers on the state "unavailable" for that entity and send you a message maybe?
Nice, I have modified the value template to make sure the values are integers and that you can calculate with it.
value_template: '{{ value.split("€")[1] | replace(",",".")}}'
This was needed for me to make it useful in the Energy Dashboard. Works like a charm!
Liked the video a lot, but what to do when the value is not in the first table on the webpage, but in a 5th table?
The trick is to see what the correct css tag is and use that tag.
@@SmartHomeJunkie Thanks for your fast reply!! Within the first table I can scrape every value using your example, but since my knowledge of css/html is none I don't have a clue as how to point to another table/use the correct css tag on a webpage.
Hello Smart Home Junkie,
In the new release of HA 2022/12 a webscraper integration has been added.
How is this new integration working compared to HACS multiscraper?
There was already a scrape function in Home Assistant, but it couldn’t scrape multiple values in one call. The HACS version can. I didn’t look into this new version yet. Might be that it supports that now too.
@@SmartHomeJunkie as far as I can see it is possible to create multiple sensors with the 2012/12 scraper integration.
I spend some time but was not able te create a working scrape sensor
@@WilcoKakkenberg Not all sites can be scraped. It really depends on how they are built.
As far as I see, it's not a new integration, but just the UI interface is added.
@@SmartHomeJunkie managed to get my values through the scrape UI integration. Thnx for your replies.
Hi, thats really cool!, how would you use this method to get past a login screen, I assume there's a way to enter your username and password first and then go to the source page after the login page?
Yes, there is. See the manual of the integration for this.
your sample code is not working anymore, something changed in HA?
It works fine here, but I understood that some sites cannot be scraped that easily.
This tutorial has helped me a lot to set up a scrape sensor for scraping the daily Belgian gas price from my energy provider. Unfortunately the have changed the website last Thursday and now I can not get it working again. I am looking into the issue already for 3 days, and cannot find a solution. Although I always had a lot of advice and support from the Home Assistant Facebook pages, scraping is something hardly anyone provides help with. Is there any possibility I could ask your assistance with setting up the sensor again? Thanks in advance. Kind regards.
You can find more info about my services at www.smarthomejunkie.net/services/
@@SmartHomeJunkie Ik zal erover nadenken, maar kan je me a u.b. een schatting van het tijdsbestek geven? 75 € exclusief btw is namelijk een enorm bedrag voor iets dat maximaal 10 minuten duurt en al voorbereid is
@@RobinDeBecker Je kan contact met mij opnemen via het contactformulier op de site.
@@SmartHomeJunkie ik heb via het formulier contact opgenomen met een situatieschets. Voorlopig enkel informatief voor dat ik een finale beslissing maak. Alvast bedankt voor de snelle reactie
@@RobinDeBecker Hmmm. Ik heb geen mail binnengekregen. Dat is niet best. Ik kijk het morgen na.
Hi, is it also possible to add the scarped prices to the energy dashboard?
Yes, you can add them to a helper and use that helper in your energy dashboard.
@@SmartHomeJunkie Nice! now i can set the high rate and low rate in the energy dashboard :)
very interesting to 'capture' online consumption but how to for belgian people about Engie or Fluvius ?
I'd say, check those sites and scrape them using the info in this tutorial.
Hi@@SmartHomeJunkie, I'm also busy with this, but I have 1 issue here. To see/ scrape our Engie dynamic energy prices, we need to login via our personal Engie account, and here we have already the first problem, they are using CAPTCHA in the backend of the login process. And till now, I'm not able to use Multiscrape for this, because, correct me if I'm wrong, it is not possible with Multiscrape?
What does the scraper put into the entities when the web page changes? Assuming it doesn’t just find something weird, but doesn’t find anything any more? Does it just keep the existing stale data or more usefully some kind of error message?
It depends if the structure of the webpage is still the same. In that case other "false" data will be shown. If the structure changed, then it will probably show either unknown or unavailable.
Oke, briljant tut again, now i have to find the Vattenfall prices haha, Oh and the soccermatches (like from flashscore) :)
Ah, that sounds like a perfect use case indeed!
Can not get this to work at all, it always unavaiable. nothing works tried so many ways. Like copying the entire css selector path and all.
While scraping with home automation integrator works flawlessly .
That's unfortunate. Not all sites allow scraping.
@@SmartHomeJunkie But the thing is that it does work, with the build in scraping. It just doesn't work with multiscrape.
@@mikej2489 That's strange. It should work with both.
@@SmartHomeJunkie Yeah i am sure i am doing something wrong just don't know what haha. Even used the full path with css selector.
Hi, I need to scrape 4 data energy. One for example is the energy of solar panel that have this selector: #pSolareMon
Now I see 600W inside website but I read 0.0kW using scraping. Very strange!
For this reason I try to scrape other variable not necessary, for example the date and I noticed that was different!
Inside website is present a box to choose the date and probably is necessary set the date before scrape. Boooo!
I'm afraid so yes.
i tought to myselfe, using a scraping tool i shoud be abel to get my energie price info. looked up a random tutorial. turns out its about energy scraping. Super bedankt, DOE ZO VERDER !
holyyyyyyyy cowww!!! I have now so much ideas hahaha thank you sir
Heel mooi en bedankt. Vraag: Waarom staat het € teken achter de prijs?
That is a layout thing in Home Assistant. You can change that I believe.
@@SmartHomeJunkie Okay, I will take a look a that. 👍
I would like to scrape the solar radiation reading off an Ambient Weather Station down the road
Whoahh ! Great job !
Really interesting, thanks
I’d love to be able to scrape tweets from a specific account on Twitter (like my daughters school) that have specific text in them (like “all schools closed” or “paved pickup”).
That could be a challenge...
@@SmartHomeJunkie LOL, you’re telling me. I thought this would be a fairly easy thing with the Twitter integration. But that’s more for posting tweets than compiling/scraping them.
@@RyanDuffy Yes indeed. The Twitter integration is just for posting things on Twitter.
thanks!
cheap flights... prices for the next month(s) between 2 airports
Super Cool
amazing ' thanks
I want to scrape the temperatures of many lakes near me so I know when it's perfect to swim where 😉
Too bad my gas company puts it in a pdf...
Nice tutorial! To use the prices in your Energy Dashboard the following should be changed/add per identity:
unit_of_measurement: "€/kWh"
value_template: "{{ value.replace(',','.') }}"
I used this ☝for my Powerpeers prices. To get the scraper ('select'), simple got to the site and right click with you mouse on the price. Select 'Inspect' (in Chrome). Now the devtool opens. right click and select 'Copy' > 'Copy selector' and put it to you select: of your entity in the multiscrape.yaml file.
any idea how to solve this? It keeps giving this message when i want to use the gas price for example inside the energy dashboard:
Entity has non-numeric state
The following entities have a state that cannot be parsed as a number
edit: Maybe i got it working now with:
value_template: "{{ value.split('€')[1]|replace(',','.') }}"