A big use case for this (at least for me) is scholarship/grant websites. They tend to rarely be updated until they become available for the current scholarship round. For example, there is one scholarship I am looking at that still has the "2023 deadline passed" banner, and I am waiting for it to be updated for 2024. This will be incredibly useful. Thanks for the video!
I recently read a forum post from someone who was using a Docker tool like this one to track the availability of 3D printing filament, and I meant to ask them which tool they were using or do some research myself. But here you are, delivering exactly what I was looking for, as if you have a sixth sense for these things 😂 Thanks, Tim!
This is a great tool and I have used it quite a bit in the past for specific purposes. As mentioned - be sure to avoid an overly aggressive scanning interval, especially if running this from home. Many sites will block suspected bot scanners - and that is a very bad thing when your home IP is what is being blocked! Running from a cloud instance or through a proxy should at least avoid the problem of getting blocked at home, but it is best to avoid the situation altogether.
I tried to configure Huginn about 2 years when I first started using Linux. Did not understand head or tail. Came across ChangeDetection a little over a month ago. Most practical piece of software I host.
Huginn is really good if you need something very specific or you need to do a bunch of post-processing on the data. Huginn helped me buy a Raspberry Pi during the supply shortage
Firstly, thank you Tim for your amazing videos. They have been very helpful to me setting up my homelab. Is anyone else getting this error "Exception: Target page, context or browser has been closed" for some pages? I have 13 sites it's looking at. The error moves around these sites but I usually have 4 or 5 at a time. I've changed the chrome version from latest to 1.54, 1.49, 1.45, 1.44, 1.43, 1.42 and 1.41. Some are better than others. I've also increased the timeout times and the number of concurrent sessions.
Thanks for the video. Wonderful idea. I'm playing with it. It's quite the touchy little creature isn't it?! It's favourite thing to say is "Exception: Target page, context or browser has been closed". Still trying to track this down but it's an interesting journey.
@@GitPush1 I've been playing further and it seemed to be much worse when I used Tim's github link that he showed for a test. Perhaps loads of people are using it in the example and Github is getting annoyed? Just a thought. After taking that test out, it seems to have settled down a bit now. Are you using the example too? It may not be related but it was just a thought.
I had this initially - I removed a few of the playwright-chrome environment variables so these only contained those on the dev's sample docker-compose stack - this seemed to fix it for me.
I was almost positive I'd updated this but apparently not! I removed the link to Tim's Website that he showed us in the example and everything settled down. Perhaps it was because so many people were using the example that it caused issues? We'll never know.
Good ol XPATH queries. Id be curious how often the elgato recorded steps get broken by Elgato redeploying their site. But it is nice that they give you the option to customize it to be more resilient. Awesome awesome video! And a fantastic tool that I dont know what Ill use it for, but definitely going to add it to my arsenal.
I'm curious to know why you don't use docker volumes for the datastore, since you don't have to access/ edit those files... to me it feels that, in this particular case, bind mounts are just a needless complication (and going with volumes will unlock gitops and remove the need for ssh, but I get that it's an advanced topic). That said, I will definitely try this service!
Thanks! Unfortunately this is also where your sites are stored as well as your dashboard config. Agreed you can use a Docker volume but I've found bind mounting it to the file system when using Docker (vs. k8s) is much easier to manage, back up, and not accidentally delete 😅
Tim, this is fantastic! Great job on this video! 😍 I love self-hosting services and this is certainly one that's being added to the "stack". 🤓(See what I did there?) 😉
Great video, biggest issue I'm seeing is it takes forever to check. Running on a proxmox docker lxc container with 8gb of ram. Noticing this only after adding Playwright
Can you plot the price detected on a graph or do you need to parse the data to Grafana or whatever? Also, I would have liked to see the chrome extension. I know I can check the doc but I choose to watch a video, so...
I use puppeteer to get my water consumption from the council website every day as they have installed wireless water meters which send the consumption to the council. Similar to what you are doing.
Is there a way to monitor an account balance? I have to log into that page and distil isn't seeing any value for the field. I think it is just looking at the publicly accessible version of the page. I need to monitor my privately displayed account balance. Thank you.
Just ran this up myself, noticed you always cut your video when you do a fresh check - is that because you came across this error "Exception: Target page, context or browser has been closed" and pretty consistently needs to be run a couple of times to get a clean check. Or were your checks all clean and just needed a few moments to run?
I usually cut out long pauses of the video. I did run into that initially when using the web driver + selenium but not since moving to playwright and chrome. You may need to adjust some of the settings but if it only happens on the first initial runs it might be hard to track down unless you look at the logs when it happens.
I think you might be out of luck with that unless you can find a way to automate getting that code and supplying it, unless you only need to supply it once. If that's the case, you can drive the browser to supply it once and then hopefully the cookies are preserved so that the next time it runs it uses those. If not see if there's a way to persist those cookies between runs.
@@TechnoTim I was thinking the way alexa media player integration works in home assistant, it register as a 2fa client... But that means if someone breaks into your ha, has your full Amazon account
No. I subscribe to you as I can follow your instructions. The VScode section with adding in the SSH extension was not digestible for my small brain like most of your videos are. You skipped so many necessary pieces there. Im sad.
![[TH] 2024 PMSL SEA Finals D1 | Fall | ยิงต้องโหด ให้เหมือนโกรธกันมาก่อน](http://i.ytimg.com/vi/OnTdfGySmds/mqdefault.jpg)
A big use case for this (at least for me) is scholarship/grant websites. They tend to rarely be updated until they become available for the current scholarship round. For example, there is one scholarship I am looking at that still has the "2023 deadline passed" banner, and I am waiting for it to be updated for 2024. This will be incredibly useful. Thanks for the video!
Great idea!
@@TechnoTim docker network npm+dns+traefik how guide please
I recently read a forum post from someone who was using a Docker tool like this one to track the availability of 3D printing filament, and I meant to ask them which tool they were using or do some research myself. But here you are, delivering exactly what I was looking for, as if you have a sixth sense for these things 😂 Thanks, Tim!
Sounds like this will work great!
I plan on using this to monitor the county website for people trying to steal my property with a forged deed so I can fight it and fix it quickly.
I've used this for years. It's an incredible tool and a real must-have.
This is a great tool and I have used it quite a bit in the past for specific purposes. As mentioned - be sure to avoid an overly aggressive scanning interval, especially if running this from home. Many sites will block suspected bot scanners - and that is a very bad thing when your home IP is what is being blocked! Running from a cloud instance or through a proxy should at least avoid the problem of getting blocked at home, but it is best to avoid the situation altogether.
Well said!
I tried to configure Huginn about 2 years when I first started using Linux. Did not understand head or tail. Came across ChangeDetection a little over a month ago. Most practical piece of software I host.
Huginn is really good if you need something very specific or you need to do a bunch of post-processing on the data. Huginn helped me buy a Raspberry Pi during the supply shortage
Firstly, thank you Tim for your amazing videos. They have been very helpful to me setting up my homelab. Is anyone else getting this error "Exception: Target page, context or browser has been closed" for some pages? I have 13 sites it's looking at. The error moves around these sites but I usually have 4 or 5 at a time. I've changed the chrome version from latest to 1.54, 1.49, 1.45, 1.44, 1.43, 1.42 and 1.41. Some are better than others. I've also increased the timeout times and the number of concurrent sessions.
Might have missed it but "Visual Filter Selector" is a great option too.
Think it shows up after the 1st screenshot grab.
Level1Techs spotted 👀
If Wendell is Level1, I am more like Level 0.01
Thanks for the video. Wonderful idea. I'm playing with it. It's quite the touchy little creature isn't it?! It's favourite thing to say is "Exception: Target page, context or browser has been closed". Still trying to track this down but it's an interesting journey.
I'm having the same issue on an Unraid server. Still trying to track it down
@@GitPush1 I've been playing further and it seemed to be much worse when I used Tim's github link that he showed for a test. Perhaps loads of people are using it in the example and Github is getting annoyed? Just a thought. After taking that test out, it seems to have settled down a bit now. Are you using the example too? It may not be related but it was just a thought.
I had this initially - I removed a few of the playwright-chrome environment variables so these only contained those on the dev's sample docker-compose stack - this seemed to fix it for me.
I was almost positive I'd updated this but apparently not! I removed the link to Tim's Website that he showed us in the example and everything settled down. Perhaps it was because so many people were using the example that it caused issues? We'll never know.
I moved to a different container. Playwright-chromium IIRC
The timing of this video and me seeing a comment in an unrelated Discord server about this service.
Good ol XPATH queries. Id be curious how often the elgato recorded steps get broken by Elgato redeploying their site. But it is nice that they give you the option to customize it to be more resilient.
Awesome awesome video! And a fantastic tool that I dont know what Ill use it for, but definitely going to add it to my arsenal.
Thanks! I thought my xpath days were over 😅
Love this video. Anyway you can do more software installs for those that use Synology Nas systems?
Thank you! What are you looking for specifically for Synology? I don't have one but assume they just use a web form to fill out?
I'm curious to know why you don't use docker volumes for the datastore, since you don't have to access/ edit those files... to me it feels that, in this particular case, bind mounts are just a needless complication (and going with volumes will unlock gitops and remove the need for ssh, but I get that it's an advanced topic). That said, I will definitely try this service!
Thanks! Unfortunately this is also where your sites are stored as well as your dashboard config. Agreed you can use a Docker volume but I've found bind mounting it to the file system when using Docker (vs. k8s) is much easier to manage, back up, and not accidentally delete 😅
Dang, this is huge. I was looking for this for a while ago now. Thank!
Now this is a tool I can use great find ❤
Wat a great bit of software with so many uses. Thank you for great breakdown
Tim, this is fantastic! Great job on this video! 😍 I love self-hosting services and this is certainly one that's being added to the "stack". 🤓(See what I did there?) 😉
Great video. Can I ask what terminal are you using in the video?
Great video, biggest issue I'm seeing is it takes forever to check. Running on a proxmox docker lxc container with 8gb of ram. Noticing this only after adding Playwright
Did you get this to work? I have the same issue (not running on Proxmox) ,and I don't think playwright is working correctly.
I've been self hosting this for a few months now to monitor Vanguard 7 day SEC yield on VMFXX. Great piece of software.
Awesome use case!
Is there an LLM integration? What we really want is a vision model like llava2 to navigate the steps, not xpath
Can you plot the price detected on a graph or do you need to parse the data to Grafana or whatever?
Also, I would have liked to see the chrome extension. I know I can check the doc but I choose to watch a video, so...
I use puppeteer to get my water consumption from the council website every day as they have installed wireless water meters which send the consumption to the council. Similar to what you are doing.
I love puppeteer. Use it all the time in CI.
Playwright is basically a fork of puppeteer … the API is mostly the same.
Is there a way to monitor an account balance? I have to log into that page and distil isn't seeing any value for the field. I think it is just looking at the publicly accessible version of the page. I need to monitor my privately displayed account balance. Thank you.
Did you try "Browser Steps" he showed it in the video, you can type there too.
Great video. Thanks Tim!!
Great! Thanks very much!
Just ran this up myself, noticed you always cut your video when you do a fresh check - is that because you came across this error "Exception: Target page, context or browser has been closed" and pretty consistently needs to be run a couple of times to get a clean check. Or were your checks all clean and just needed a few moments to run?
I usually cut out long pauses of the video. I did run into that initially when using the web driver + selenium but not since moving to playwright and chrome. You may need to adjust some of the settings but if it only happens on the first initial runs it might be hard to track down unless you look at the logs when it happens.
Great video. Thanks TT
Is there any ethical way to stop Amazon captchas? Even when setting checks to every 1 hour I am still hit with a captcha.
Amazon has the best bot detection on the planet... seriously
Keepa or camel camel camel?
Really cool, thanks
Those explicit xpaths -shudders-
Is it possible to customise the selectors so you don’t end up with horribly flaky browser steps?
yes, that's the reason why it waits for you to click apply so that you can customize the step
Nice, thank you. Is on my kid to try out :)
Thats really good software
what about sites that use MFA?
I think you might be out of luck with that unless you can find a way to automate getting that code and supplying it, unless you only need to supply it once. If that's the case, you can drive the browser to supply it once and then hopefully the cookies are preserved so that the next time it runs it uses those. If not see if there's a way to persist those cookies between runs.
@@TechnoTim I was thinking the way alexa media player integration works in home assistant, it register as a 2fa client... But that means if someone breaks into your ha, has your full Amazon account
First
"Elagto"
“Lagtoe”
13:20 hopefully you removed that webhook already from your channel otherwise you must be getting spammed :)
No. I subscribe to you as I can follow your instructions. The VScode section with adding in the SSH extension was not digestible for my small brain like most of your videos are. You skipped so many necessary pieces there. Im sad.
Hey! Sorry! If you're getting stuck, hop in our Discord!
You should blur your webhook address
No worries, used only for the episode