I once lived in a house where the DSL would go out every day around 5 pm. I created a simple logger using ping to document this for the ISP, because they didn't believe me. When they finally sent a service person at the correct time, we discovered that one of their node boxes next to someone's lawn was being sprayed every day at 5 pm when the lawn sprinklers came on! No wonder it didn't turn up in their diagnostics!!
@@User_1795 If they put something dirt-cheap and simple (and they have to do this) on the "last mile" - it may go unnoticed for years. But how their equipment has survived regular showers? Miracle?
I have to disagree with this video. Isps can also speed up your traffic when connecting to sites like speedtest.com to make it seem like they're giving you what you asked for. We've seen similar things happen with mobile phones were they perform much better when being benchmarked than they would in the real world. There has to be another way to measure the data.
My Grandma has like 900 up and down. With a fiber going all the way to the router. In a distant village. In Ukraine. For ~15 USD. So you're actually having a first world problem, with an expensive slow Internet.
Well, it is about the same here, in Sweden. My mother and father 80+, which live in a rural area with 250+ inhabitants, do have faster Internet then what I have in the middle of a city. :-/
There are 4 available ISPs at my home in Siberia, 3 wired twisted pair etherned and one fiber (that's not counting mobile ones, which are 5 more or something). There's a promo option of 500Mbps symmetric for $8, but regular prices are more like $14 for the same speed. There are no 1gbps speeds (but you could connect to multiple ISPs and do ethernet bonding or something; i used to do that when unlimited internet was like 64kbps per ISP 13 years ago; pay-for-megabyte still was 100Mbps symmetric). One downside of siberian internet is that your latency to the nearest european datacenter is at least 100ms (50ms to Moscow). I was very surprised to see that here in Bay Area your internet is usually DOCSIS cable (a completely nonexistent thing in Russia; even ADSL is very rare, we jumped straight into 100Mbps TP at the very least with fiber coming a few years later) with 40Mbps up being the max that you could get at home for any amount money.
@@provod_ you are absolutely right. I'll tell you more: I've been to several places abroad, and the Bay Area was THE ONLY DAMN PLACE where staying in roaming and using 4G in roaming was actually cheaper than buying anything from a local carier.
US has some serious problem (and that's my guess) with legislation and taxes around infrastructure. Even Brazil, which is tax hell, we have 300Mb down, 150Mb up fiber everywhere, for about 40 USD. You would need to go into the mountains and where there is no paved roads to find places without decent internet.
In Australia with our National Broadband Network (NBN), a lot of ISPs were charging people for higher speeds than actually possible for their connection (e.g. charging 100mbs for a line that could only support 40mbs) this resulted in action by the Australian Consumer Commission. As a result, now, if you can prove you do not get at least greater than the next tier down in speed the ISP has to refund you the difference for all the months it has been charging you the higher rate. This tool could be quite useful for that.
Mine rarely doesn't give me full speed I get it's supposed to be 105mbps and like 12 or 15 up I don't remember but I have QOS set up but last time I turned it off and got like 135 I had not really heard of getting more before but was like cool
hahahahh as a turkish citizen, I just laughed. expensive internet: checked monitored internet: checked nigthmare of CGNAT: checked website blockage: checked
if you have the wan connections anyway then it might be worth using 'OneMarcFifty Virtual Network in Proxmox for MPTCP Test lab' as a starting point to increase the personal upload rate to a cheap remote diy vpn etc. "We build a Virtual Network in Proxmox for the MPTCP or OpenMPTCPRouter Test lab. In this video we do A LOT OF STUFF ! We create Proxmox network virtual networks, we use the tc netem QoS Filter of Linux for shaping of network lines, we add latency to the network in order to simulate a slow network, we install OpenWrt in Proxmox but also OpenMPTCPRouter. We will run a MATE desktop on a Linux machine in an LXC container and run the X2GO Server in the container plus the X2GO client Windows. Furthermore there is a nodejs client that shows ssh remote exexution integration. The three shaper machines are in fact linux LXCE containers which we turn into routers and dhcp servers. The full description with ALL commands, URLs and instructions etc is here (it is longer than 5000 characters)"
oc integrating Webtop in to the mix is also good, although i usually like vm's with 'nomachine' for no fuss single install better real time remote video streaming to a local android/linux/win gui terminal 'Techno Tim Linux desktop, inside of a container, inside of a browser??? Yes. A Webtop.'
Found this channel when looking for Raspberry pi. Had no Pi back then but still loved to watch this valuable content. Now, Thanks to God I got a Pi 4 4GB and I love watching to your videos. Keep up the good work.
This is why they say "Speeds *up to* 1 Gbps". Speed isn't everything either - one time I was getting reasonable speeds but crippling packet loss due to a local fault.
I don't think they all do, I have Verizon Fios and I don't have a cap. However, Xfinity is available in my area also and I believe Xfinity has a data cap.
Thank you Jeff! I have played around with a friends Raspberry Pi several years ago but didn't have any good use for it until now. Went out and bought my first Pi because of this video :)
Thanks Jeff! Xfinity/Comcast have a monopoly in our area. Feels like an abusive relationship. Being able to keep them honest, and know what our devices are doing is super useful. Cheers!
Of course they do. Time Warner won't operate in cities/states they are in. And they don't operate in states TW operates in... It's collusion. They need to be broken up. Wtf is congress even doing?
Yeah... i pay 40 euros for 1gb fiberglass a month. It's also giving me 950+/950+ mbps too, which is reasonable I think. But then again, I don't live in a third world country. It's a good thing that starlink comes to the rescue for those in need.
Oh, believe me I have considered it often. But it's a lot lot LOT more than just the technical problems you have to deal with. I'm not sure if I'm up for the political and social maneuvering that's involved in the local community (which can often lead to a strong division and a lot of foul stares).
@@JeffGeerling well you don't really need to allow that stop you from starting the business. There are always others who can take care of the political and social maneuvering while you take care of the technological aspect of the business.
@@theundertaker5963 North America is literally the worst place to start up an internet service provider fresh from ground zero. You need millions in capital and political leverage to shoo out the 3rd party monopoly providers from your region. It's a financial risk that literally no one will give you money for, unless you have the cooperation of every single seat of power in local government.
@@JosephArata well, dont think so much "start up an internet service provider" more private diy Far bigger pipe than any single isp WAN... without the 3rd party isp hassle/continuous cost starting with how jeff Can make a long distance private diy big pipe to/from his sister ( how many miles line of site) with pi's, old cheap/free 5v-12v wireless routers + pv+ reclaimed batteries, several collectively routed with m.th-cam.com/video/S-Xmcig1ddA/w-d-xo.html 'OneMarcFifty Virtual Network in Proxmox for MPTCP Test lab' at both ends & potentially every pi relay in-between with cheap integrated m.th-cam.com/video/wEtlbNir8KE/w-d-xo.html LoRa LoRaWAN Range World Record Attempt (Re-published No. 120) Andreas Spiess for collective instrumentation & small quick txt, alerts etc ..collect up the sw tools and shell script it to make a working one off on demand clean usb Debian super light 'slax' slax org customize .php deployable from pi-KVM etc
i just wanna say, i set this up today as a linux noob. your tutorial was so easy to follow, but i did have pi knowledge. i had a big fight with errors, but fixed them by giving 'pi' docker perms [thanks issue tracker!] this tool is so nice. i have pi hole just to track queries, as i have two ad blockers already. the internet-tracker is super nice. im gonna gather data on this as i love seeing data. cheers!
Sooo considering speedtest-cli uses 1-1.3 GB every test, every 30 minutes would equal to around 50 GB a day or more. * 30, that's 1.5 TB a month, even with my gigabit internet through Comcast that would exceed the 1.2 TB/month quota and I would end up paying for extra bandwidth even if I never used my internet during those 30 days lol.
Good point-I didn't realize data caps were that low, OUCH! I might change the default to every 60 minutes at most, to help protect people from ISP stupidity. I also plan on making it easier to configure the interval so people could choose once a day or something more for long-term monitoring.
@@JeffGeerling I still really like this project, very impressive, I think I'm going to order a couple Shelly plugs! Thanks again for another awesome video! Long live the Pi generation. :)
I left Xfinity as soon as AT&T Fiber arrived. They actually threatened to BAN me for a year for using over 1 TB (back when 250MB was the limit). I had to move over to Comcast Business and sign a contract to get unlimited data. Then when they lifted the cap on their Xfinity (Residential) side they wouldn't let me move over. And then when the contract expired and AT&T Fiber arrived I said good bye and good riddance.
Thank you for the video Jeff. What about the impact to the other processes in your network while the test is happening? Have you noticed it? Also, the other way around, does the traffic in your network affects the result of the tests?
Verizon FIOS supposedly doesn't have a data cap, but I would not want a speedtest going off while trying to game online. Good video though. Maybe someone could make a much shorter version of speedtest.
THanks for this vid. I finally got it installed (I'm the guy that commented about 15 times over the last day or so in the 'issues' section!). Great package, even though it was a nightmare to get it installed lol. Thank you!
@@realityveil6151 Dude, you cannot put VM into active speaker to be working as LMS client and DAC. And this is only one example - actually 5 of my Pi are LMS clients and DACs. Everything has its purpose
In france, I pay 38€ for a sold _up to_ 8Gb down/1Gb up which is, in fact, a measured 5Gb down/1Gb up connection per month with no data cap. I don't complain with the difference, i was never able to use my connection at its full speed.
Thanks for this. I already have a Pi running PiHole connected to the router, but I've been using a scheduled task on a windows machine to run an hourly speedtest to track my internet speeds. This looks like a much better setup.
It should be noted that if you have internet speed greater than 300 Mb/s, you MUST use a R Pi 4 as the 3B+ only has a 300 Mb/s NIC. Otherwise great video as always!
If you want the automated speed tests, then yes. If you're just using it as a pi-hole, then no. The traffic doesn't flow through the Pi; it just resolves DNS queries. But good point nonetheless.
Almost every ISP shares bandwidth. Unless you are paying for it you are sharing bandwidth with all the customers that are in your area. The analogy I use is a water hose. There is only so much water that can be pushed thru the hose at any given time. Bandwidth is the same. Great video as always!
Nice video as usual 🙂 Well, I think that this problem is relatively extended in every country of the world I guess. Until 20 years ago in Italy there was a single telephone and internet operator but, at the end of this, nothing changed to much because the entire national infrastructure still was owned by that company, so the other "new companies" had to rent part of the network. To be short, this caused a lot of issues as there were more ISPs on the same infrastructure that wasn't upgraded as needed up during the years mainly for the high costs: here if you want to "put down" a new line/fiber you have to broke up streets as the old pipes are too small and full of old cables. Anyway, until 5 years ago I was paying 45€ monthly for a 6 Mbps ADSL, then the fiber (FTTC 100/20) finally came and, as I'm not so far to the cabinet I'm not losing too much signal. I still paying likely 50€ monthly but at least now all it's working as intended. Ah, phone calls are included 😆 Sorry for the poem and thanks for your work here.
Your channel and this video specifically has helped me catch Cox throttling my speeds and get them to give me my actual internet speed back. Went from 200Mbps to ~700-940Mbps with no hardware changes on my end. Thank you!
I am so glad you posted this. I have "suffered" through years of service via charter at only 200mbps down and 11mbps up. I too care only about the upload speed since 200mbps down was totally adequate. I upgraded to gigabit just so I could get the 40mbps up. It's ALMOST adequate now. Symmetrical would make everything right in the world!
So meaning from my home to ISP but not my home to ISP to Google for instance? So the advertised bandwidth means literally nothing if my ISP's connection to google is slow in that example?
So there are peering agreements for ISPs, they have to peer with someone else to get access to the internet. Literal ISPs for the ISPs. These are expensive contracts for multi gigabit connections. Not too mention BGP ASN’s and the constant battle to obtain more v4 addresses from ARIN or other addressing entities. They do have their own battles.
@@DeadlyDragon_ In a past job, I was the one to have to fight those battles - can 100% confirm they're tough. We were able to operate with only one ASN (as many nets are) by using the same routing policy everywhere, and we addressed (pun not intended) the IPv4 shortage problem by going the DS-Lite route (pun still not intended). But all of that was a walk in the park compared to the challenge of purchasing enough transit to meet customer demand. We kept having to readjust our oversell ratios as we updated our traffic engineering practices, and getting a good handle on what "peak demand" looks like is a surprisingly complicated question. We weren't yet big enough to get caching appliances from Akamai, Netflix, and friends... not that I wasn't checking constantly. But we always strove to do right by our customers - small ISPs have to or they don't survive. (So be a customer of a small ISP! Support local businesses that actually give a crap about you!) Glad I wasn't still working there for the COVID pandemic. I would hate to be the one to have to figure out how to handle a bunch of WFH traffic overnight! :)
Finally someone else noticed this problem! Every time I had an internet problem and called my ISP, I couldn't believe how little monitoring they do. ISPs should ENCOURAGE their customers to set up monitoring like this if they're not going to.
Grafana on itself does nothing, you need some data to display in Grafana. Typically this would be in a database (in this video, it's Prometheus, but it could also be MySQL, SQL Server, etc). But then you also need something to put in that database in the first place, which in this video are the speed test results and the website checks.
Im thankful a smaller local ISP started up near me. I'm a few miles outside the ST Louis metro area and I just went from only having satellite to a fiber connection which is faster by 100 fold. Spectrum wanted 20k 6 months before Gateway fiber just ran the line down our street for free if we payed for a subscription. I pay for a gigabit connection. I still see what you get most of the time like 700-900 down depending on the time of day but my uploads are always right around 900+. Still very happy with them, I only spend about 90$ a month on it.
Hi Jeff, it would be great if u did a video about backing up the sd card on a pi to a nas. I am kinda worried it will fail and having to set everything up again.
In Thailand, many ISP can let's customer to set up Internet speed by their owners from the ISP website, for example, customer package is 500/500 Mbps, customer can set speed 500/500 or 600/400 or 800/200 or 900/100 Mbps.
It would be awesome to have a real time traffic monitoring in place. As the pi hole is already recording DNS requests call, it would be pretty nice to see what clients are currently doing on the Internet, since the pihole frontend it’s basically just a list of requests that were resolved. This data may probably be relevant to show such a real time traffic, but it itself doesn’t record the traffic volume for instance.
Hey, you're sitting in the city with your 1 Gbps plan, out here in St. Charles County we get 25/5 and even then AT&T can't deliver it without some spottiness!
Hi, thanks for this! Your tutorial was very helpful. It has a steep learning curve and took me about a day to set it up on two orangepi boards. Now, I can finally monitor my two ISPs.
One missing feature. With everyone monitoring their ISP 7/24 there needs to be an opt in option for users to send anonymized data with ISP, Location down to the city level to a central site on the internet. Then this can be a public service that anyone can search their location/city and see the averages that users monitoring 7/24 in that city are getting from each ISP including outages. This will allow everyone to know before hand which ISP is better to go with and turn this project into the next level.
I'd like to also see a crowd-sourced ISP statistics! This way, no more pesky ISPs claiming "no our internet is not the problem, go restart your router."
This is a really cool idea, but I can't think of a way to stop ISPs from poisoning the data. As the service gets more popular, it's more likely that ISPs would send fake data to it
I really appreciate the work you put in your projects. Yesterday I was wondering if I could use my Pi to check which of all the ISPs would provide the best internet quality and here I have an entire git repo to test! I will be testing 3 providers connected to my Pi via usb eth adapters. Thanks a lot!
You're lucky having 30. Here I can only get 1000/10. It's almost always saturated, and I have to host a lot of stuff on the cloud simply because my connection would get choked as soon as someone remotely starts a download.
Really great videos, and information on new and upcoming hardware, the only problem I see, is your github instructions on this are not the greatest. For someone like myself that is still learning about linux systems by utilizing a raspberry Pi, again for someone like myself hat knows enough to be dangerous, can really get frustrated with install instructions that are lacking, and would make someone want to just throw in the towel all together and stop learning more about raspberry pi's. This is by no means a dig on you, people like you are what made me want to get into buying my 1st raspberry Pi, but again with the github install instructions, its really hard to even tell what exactly it is that I am installing, or how specifically. Am I installing Ansible, or Pi Hole with docker, or do I have to have docker already installed, is it installing ansible? it's really hard to understand what all the instructions are having me do.
Wow! As a "stats junkie" and a Pihole user, I really like that project's results! The bit I would maybe mention/change would be the automatic speedtest. I understand there is no other way to get "real" data for this, but I would fear that the sporadic tests would impact my internet experience. If someone in the house is playing a game for instance, they'll get lag spikes every half hour for "no reason". Other than that, I might try this out on a raspberry pi 3 I have laying around and if it works well, have it replace my current pi4 config. Thanks for the awesome content as usual!
Samknows fixed this issue by monitoring the network activity. If there is any significant Internet usage, the device would do the tests later. I'd love to try this project (I'm on a 1Gbps connection) but with the current prices and stock shortage for Raspberry Pis 4 and 5 it seems it will take a while.
you could also monitor the power grid frequency. This gives indication about the stability of the grid and whether any power provider had issues with their power plants or maybe even an outage
OMG!! This is exactly what I'v been trying to accomplish at my home for months now. Failure after failure I had already giving up. Thank you so much. This shit is the bomb
@@pgp-acc Most connection in HUN are asymetric dl/ul (even business packages other than VDSL), but for 7200HUF/mo (approx 20-22EUR) Telekom offers 2000/1000 on fiber here too (for residential consumers). Not anywhere of course, we have areas where ADSL on crap-old phone copper still cost like 9k HUF (30EUR) and barely reach 10M/500kbps up.
I did something similar but in my laptop and found that my ISP was slowing it down by more than two-thirds at times. When confronted, they doubled down saying that data consumption was "too high" and that it could only mean I was getting the speeds I was promised. Ofcourse, when you calculated the average speed it was a little over this throttled speed. The data consumption was just because of the speedtests running every 2 minutes in the background. I switched my ISP as soon as I found an alternative. Best decision ever.
Symmetrical. You would prefer 150mbps of symmetrical. Asymmetrical means exactly what you already have with the 1gbps down and 40mbps up. Don't sweat it Jeff, I made this mistake too.
They beat me to it, but yeah...I much preferred my 75M synchronous/symmetric FiOS at my old address over the 1G asynchronous/asymmetric Spectrum Business at my current one. Unfortunately I have no other options besides cable here, the phone lines are too old to support more than 128k (per AT&T) and there is no fiber in the area.
@@MarkusOrt It's mostly TCP overhead which if you had a gig would come out to around 949Mbps max. There's just no way to get a 1000Mbps on a 1000Mbps link negotiation.
An OpenVPN GUI install and a DDNS updater on this box and it becomes something I can bill for, as opposed to a rabbit hole of hours spent behind my keyboard. Great work!
I called Xfinity because my Internet had been down from around midnight to 7 AM. X: "No, your Internet connection is currently up." Me: "I'm talking about last night. X: "Yes, it's currently up." ...minutes later... X: "Yes, it was down, and we'll credit you $10 on your next bill." (was never credited, but I gave up) Ever since, I have been running a ping test from a Pi every 5 minutes to track outages. Nice job on your tool and video!👍
None of the European comparisons are apples to apples. The large geography, fragmented regulation etc. of large land mass countries will always allow pockets of monopolies and more expensive operations than small densely populated land areas. You get gigabit or more for less than the price of a pizza in countries like South Korea et al. The better measure is the quality of service and uptime from any ISP.
@@pd8559 Well I pay $7 for 1 gigabit down and 500Mbps up. Real speeds are constant 940Mbps down, 510-520Mbps up. And as for the uptime, I am pretty much always online (me and my servers) and in 10 years I had 1 single time about 2 hours of Internet outage. SO I would say the uptime is good. I live in Romania.
Thank you for a very informative video! I have had a NextCloud test running in my house on a RPi3. It did the job and worked ok, but I now want to upgrade it to a Pi4-8GB with a 6 TB drive and booting from a 512GB NVME in a USB enclosure. What I am wondering about is if I should use docker for the install so I can run the speedtest you mention here as well? I live in Brazil, and the rules here is the the ISP HAS to deliver 40% of the contracted speed at all times. NEVER below that number. But - they also HAVE to deliver a minimum of 80% measured over a month (30 days?). The big reason for upgrading the Pi is to get GB network and USB3. Oh - I almost forgot - I have 450/225 Mb on fiber.
Spectrum is the only non-DSL internet I can get at my address. Even paying for the "highest bandwidth", I get maybe 1/2 of that most of the day, with pretty regular dips and outages. A lot of companies seem to have changed their wording in advertisements using things like "maximum bandwidth, regular service at this speed not guaranteed" or something similar.
I have the same plan same price with Spectrum, no other options. It goes out often and one time it took them 11 months to figure out a packet loss issue smh. I currently have a Pi 3b+ setup to email me ping results every 6 hours via a cron. Unfortunately the Pi 3b+ caps out at ~330Mbps in a speed test due to I believe the USB controller iirc so I am thinking of upgrading it to a 4. It’s also currently my Pihole. Great vid!
Jeff, you should get or add an Ripe Atlas Probe! Contributing to the global data research as well as having a great additional monitoring for your ISP as well as connectivity and routing aka peering.
When I tried to switch to an LTE ISP, I had to talk to the CenturyLink customer retention specialist. They pretty much gave up when I told them "Look, I'll pay you $200/mo if you will give me 100Mbps down." They act like they don't understand how 3Mbps is not sufficient for anything except for email/browsing nowadays. Without ad block, my wife's PC consumes the full 3Mbps down when she opens her forums due to all the auto starting videos/ads. I currently pay $90/mo for two 3Mbps dry DSL circuits. CenturyLink/Frontier/Hughes/Dishnet are going to be in a world of hurt when Starlink gets rolled out in earnest...
DUDE! Just what I was looking for. I am paying for 200mbit but I am pretty sure ISP has me @100 and they are lying... Let you know maybe when I find out wth they are doing :) Great vid!
I work for charter/spectrum in Louisiana and our gig speed is much closer to a gig than what yours is showing. Could be a few things but we had issues with the first docsis 3.1 modems we had that showed similar speeds to what yours is but now that we have the newer version of the 3.1 modem the speed is consistently higher than before. Also what others are doing on the internet in your area doesn't affect your service much of at all, but what can is the plant not being properly balanced and your node just having to many modems on it among other things. Also we're those speeds directly out of the modem or through the router bc spectrum only guarantees 70%+ directly out of the modem.
One thing the continuous testing does not account for is the conditions on the server side. We are essentially assuming server side never experiences slow downs or partial outages(causing you connection to be re-routed). I'm sure most of it is still your ISPs doing, but just maybe a small disclaimer. Also one thing to consider is your ISP charges you for a 1 Gbps connection from your home to their head end. Once your connection reaches that location and is routed to other data centers, it's largely out of your ISPs control. That's maybe something ISPs could be more clear about. The internet and how you accomplish connecting to it is not as cut and dry as we would like to be.
4:50 - YES!! There is little to no reason ISPs couldn't offer a 150 symmetrical service. DOCSIS 3.0 supports 200Mbps upstream, 3.1 is over 1Gbps, and DOCSIS 4.0 is 6Gbps. They should make that the standard just to properly compete with fiber ISPs.
I've been doing something similar for a while, though in my case I have mine set up using node-red, home-assistant and influxdb. Every hour node-red will run a speed test, parse the JSON output and send it to home-assistant... which in turn records it into influxDB for long term storage. I have to wonder, what is the impact of your internet usage on the speed tests / the speed tests on your internet usage. I would assume that speed tests during peak usage periods on your network will have reduced results because other devices are using the available bandwidth too.
That's true, and one of the things you have to discount and/or test further in terms of monitoring at your router as well. I've considered switching to InfluxDB too (Grafana's happy with whatever).
I would love to set something up like this that also can reset the modem when it detects an outage. I live in a poorly connected area and have a DSL connection that drops almost daily and often times power cycling the modem seems to do the trick. I'd love to either use the Shelly Plug or the GPIO pins connected to a relay to reset the power every time the internet is out for more than a couple minutes.
I have no idea what this is going to be about, but Jeff replied to a Stuff Made Here comment on a Smarter Every Day vid, so I checked his channel. Then he has a Princess Bride thumbnail to boot?! Almost subbed sight-unseen.
Great video Jeff, earlier this year I migrated all my home intel servers to raspberry pi’s and some cloud compute. I have a pair of pi 4’s running redundant asterisk pbx services. And a pair of pi 4s running pi hole service, with a lonely pi2 running homebridge, all the pi4’s use Poe hats and the pi 2 running a Poe dongle thing. They are great. Looking at the video would love to setup internet monitoring and maybe some lan monitoring. Probably need to use some snmp traps rather than raw point to point from the pi as running 10Gbe between all the switches and NASs but pie only at 1gbe. Thanks again.
It's running speedtest through the pi's network port? We have 1gb fiber and we have a 2.5gb link to the pfsense box and a 10gb backbone. It woke the network right up!
I'm already using pi hole on a dedicated pi wired to my router so now I'll look into adding the other things you listed but my main goal is to see downtime when I have no internet at all. A report showing date, time, and duration of outages is what I was really wanting until spectrum came out and replaced the cable from my box to my router since it was testing bad. My parents are signed up to get star link since their only choice is hughes net satellite so a pi doing the same stuff as yours will be nice to monitor how good its doing.
So i have recently seen a project where a pi was slotted into a pci e bracket. The project used poe power but im interested to see if a pci card could be made that allows power to be pulled from the slot its self. I would think it has plenty to use. Ethernet could also probably be passed through but i think a second ethernet cable would still be more efficient. There was a suggestion to also use an internal usb to usb hub to power it as well.
I like it. When I get one of those round tuits I intend to set up something like this. It would be fun to have a dashboard to monitor a hydroponic greenhouse (using Shelly of Sonoff plugs).
Where I live, when you have high latency or slower than promised speeds, you call the ISP and it’s fixed in a few hours. ToS go both ways. Greetings from Estonia!
Thanks for this video. This is a project I am going to do. It's really cool to find out you live in St. Louis too! That's very close to where I live and work. :)
@@JeffGeerling I'm running into an error....I've worked through a couple of others but this one has me stumped: fatal: [192.168.xx.xx]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: pi@192.168.xx.xx: Permission denied (publickey,password).", "unreachable": true} Help would be appreciated.
Interesting set-up. I stopped using PiHole a few months back when a TH-cam update broke the block lists and I couldn’t make exemptions for all the weird Google ad domains and ended up not able to watch any YT for a week until I just gave up.
The cat-and-mouse game with TH-cam ads finally convinced me to pay for Premium. Not advertising for it, but since I watch a lot of TH-cam it was worth it for me.
Some ISP in Germany have policies that prohibit to test connection speed too often. Every 45 minutes can be considered as "too often" as it can have very big impact on the ISP network if there are many people doing this.
Care to share which and where they state this as a policy? I've never heard of it and would like to look into it, but couldn't find anything on my preferred search engine.
@@grmasdfII it's stated in the "Allgemeine Geschäftsbedingungen (AGB)". don't know the English term for that. I think 1&1 is/was one of the few who actually enforced that some times. I think my cable TV and internet provider (pyur) has this policy too but i didn't mind to look it up as I'm not planning to do connection speed tests that often. It's mostly formulated very "open". Like "any behavior that can cause unnecessary load to the ISP network".
Home assistant has been monitoring my internet for years. It's actually quite consistent, but the 10mbps upload is a real drag. I'm waiting for AT&T fiber to finally get installed in my neighborhood.
BTW, @JeffGeerling, have you inquired about a "business" cable connection. They often have the same upload and download speeds, and shouldn't be much different than the $150 you're paying.
I once lived in a house where the DSL would go out every day around 5 pm. I created a simple logger using ping to document this for the ISP, because they didn't believe me. When they finally sent a service person at the correct time, we discovered that one of their node boxes next to someone's lawn was being sprayed every day at 5 pm when the lawn sprinklers came on! No wonder it didn't turn up in their diagnostics!!
Omg that had to suck. That's crazy they didn't even know they're shit was shorting.
@@User_1795 If they put something dirt-cheap and simple (and they have to do this) on the "last mile" - it may go unnoticed for years. But how their equipment has survived regular showers? Miracle?
Now that's real-world troubleshooting and diagnostics.
@@IDGinUkraine I'm sure these things are designed to mitigate the damage from water ingress. Probably some good drain holes and passive airflow.
Few years back i was on DSL and everytime somebody was using the phone the internet would stop 😂
This channel is such a goldmine of useful content.
Agree, this is such a great channel.
Everything you could ever want to know about random electronics
I have to disagree with this video. Isps can also speed up your traffic when connecting to sites like speedtest.com to make it seem like they're giving you what you asked for.
We've seen similar things happen with mobile phones were they perform much better when being benchmarked than they would in the real world.
There has to be another way to measure the data.
My Grandma has like 900 up and down. With a fiber going all the way to the router. In a distant village. In Ukraine. For ~15 USD.
So you're actually having a first world problem, with an expensive slow Internet.
Well, it is about the same here, in Sweden. My mother and father 80+, which live in a rural area with 250+ inhabitants, do have faster Internet then what I have in the middle of a city. :-/
There are 4 available ISPs at my home in Siberia, 3 wired twisted pair etherned and one fiber (that's not counting mobile ones, which are 5 more or something). There's a promo option of 500Mbps symmetric for $8, but regular prices are more like $14 for the same speed. There are no 1gbps speeds (but you could connect to multiple ISPs and do ethernet bonding or something; i used to do that when unlimited internet was like 64kbps per ISP 13 years ago; pay-for-megabyte still was 100Mbps symmetric). One downside of siberian internet is that your latency to the nearest european datacenter is at least 100ms (50ms to Moscow).
I was very surprised to see that here in Bay Area your internet is usually DOCSIS cable (a completely nonexistent thing in Russia; even ADSL is very rare, we jumped straight into 100Mbps TP at the very least with fiber coming a few years later) with 40Mbps up being the max that you could get at home for any amount money.
@@provod_ you are absolutely right. I'll tell you more: I've been to several places abroad, and the Bay Area was THE ONLY DAMN PLACE where staying in roaming and using 4G in roaming was actually cheaper than buying anything from a local carier.
Same in svk
US has some serious problem (and that's my guess) with legislation and taxes around infrastructure. Even Brazil, which is tax hell, we have 300Mb down, 150Mb up fiber everywhere, for about 40 USD. You would need to go into the mountains and where there is no paved roads to find places without decent internet.
When your ISP promises "UP TO", and you get 3/4 of that promise, consider yourself lucky!
"Count your blessings"
@@JeffGeerling "Pray we don't alter the deal further" lol
@@mewtwo064 "HERE IS A UNICYCLE"
In Australia with our National Broadband Network (NBN), a lot of ISPs were charging people for higher speeds than actually possible for their connection (e.g. charging 100mbs for a line that could only support 40mbs) this resulted in action by the Australian Consumer Commission. As a result, now, if you can prove you do not get at least greater than the next tier down in speed the ISP has to refund you the difference for all the months it has been charging you the higher rate. This tool could be quite useful for that.
Mine rarely doesn't give me full speed I get it's supposed to be 105mbps and like 12 or 15 up I don't remember but I have QOS set up but last time I turned it off and got like 135 I had not really heard of getting more before but was like cool
"My cable internet connection which is supposed to be 1Gbps, is only about 700Mbps on average..."
Me: **cries in 10Mbps ADSL**
hahahahh as a turkish citizen, I just laughed.
expensive internet: checked
monitored internet: checked
nigthmare of CGNAT: checked
website blockage: checked
At least I get a stable IP address!
Let's not forget "fair usage policy" and lack of ipv6 connection of any sort.
if you have the wan connections anyway then it might be worth using
'OneMarcFifty Virtual Network in Proxmox for MPTCP Test lab'
as a starting point to increase the personal upload rate to a cheap remote diy vpn etc.
"We build a Virtual Network in Proxmox for the MPTCP or OpenMPTCPRouter Test lab. In this video we do A LOT OF STUFF ! We create Proxmox network virtual networks, we use the tc netem QoS Filter of Linux for shaping of network lines, we add latency to the network in order to simulate a slow network, we install OpenWrt in Proxmox but also OpenMPTCPRouter. We will run a MATE desktop on a Linux machine in an LXC container and run the X2GO Server in the container plus the X2GO client Windows. Furthermore there is a nodejs client that shows ssh remote exexution integration. The three shaper machines are in fact linux LXCE containers which we turn into routers and dhcp servers.
The full description with ALL commands, URLs and instructions etc is here (it is longer than 5000 characters)"
oc integrating Webtop in to the mix is also good, although i usually like vm's with 'nomachine' for no fuss single install better real time remote video streaming to a local android/linux/win gui terminal
'Techno Tim Linux desktop, inside of a container, inside of a browser??? Yes. A Webtop.'
Website blockages when the government's involved for censorship reasons then that's great and is a reality for some countries
Found this channel when looking for Raspberry pi.
Had no Pi back then but still loved to watch this valuable content.
Now, Thanks to God I got a Pi 4 4GB and I love watching to your videos.
Keep up the good work.
This is why they say "Speeds *up to* 1 Gbps". Speed isn't everything either - one time I was getting reasonable speeds but crippling packet loss due to a local fault.
ISPs in the US sell gigabit plans with a 1TB/mo data cap, which you could literally burn through in a few hours 😂
I don't think they all do, I have Verizon Fios and I don't have a cap. However, Xfinity is available in my area also and I believe Xfinity has a data cap.
I'm lucky to not have a data cap. I can't imagine living under those conditions.
@@abousono1 It’s mostly cable providers that do this, that’s true.
Yup comcast tried to pull this shit in the middle of the pandemic. They quickly pulled back and “graciously” decided to wait on fully implementing it.
I have Suddenlink with 400 down and 40 upload with unlimited data and for our 4-30-21 billing period my house used a whopping 3.2tb of data
Thank you Jeff! I have played around with a friends Raspberry Pi several years ago but didn't have any good use for it until now. Went out and bought my first Pi because of this video :)
Have fun with it!
Thanks Jeff! Xfinity/Comcast have a monopoly in our area. Feels like an abusive relationship. Being able to keep them honest, and know what our devices are doing is super useful. Cheers!
same.
Of course they do. Time Warner won't operate in cities/states they are in. And they don't operate in states TW operates in... It's collusion. They need to be broken up. Wtf is congress even doing?
Thanks!
hearing $150 a month for internet is giving me a heart attack
Yeah... i pay 40 euros for 1gb fiberglass a month. It's also giving me 950+/950+ mbps too, which is reasonable I think. But then again, I don't live in a third world country.
It's a good thing that starlink comes to the rescue for those in need.
@@MrMediator24 Very true in the USA, but i think/hope Starlink is going to change that.
go look at business internet, you have a stroke and heart attack at the same
Workers get overpaid there so it evens out, I guess.
for like $80 we get 50mbps in australia, and the average cost for a 1gbps is $321.50 all in aud
Super video! I applauded for $2.00 👏
I would like to see Jeff Gerling setting up his own ISP and show how to be an internet rebell.
Oh, believe me I have considered it often. But it's a lot lot LOT more than just the technical problems you have to deal with. I'm not sure if I'm up for the political and social maneuvering that's involved in the local community (which can often lead to a strong division and a lot of foul stares).
@@JeffGeerling well you don't really need to allow that stop you from starting the business. There are always others who can take care of the political and social maneuvering while you take care of the technological aspect of the business.
@@theundertaker5963 North America is literally the worst place to start up an internet service provider fresh from ground zero. You need millions in capital and political leverage to shoo out the 3rd party monopoly providers from your region. It's a financial risk that literally no one will give you money for, unless you have the cooperation of every single seat of power in local government.
@@JosephArata well, dont think so much "start up an internet service provider" more private diy Far bigger pipe than any single isp WAN... without the 3rd party isp hassle/continuous cost
starting with how jeff Can make a long distance private diy big pipe to/from his sister ( how many miles line of site)
with pi's, old cheap/free 5v-12v wireless routers + pv+ reclaimed batteries, several collectively routed with m.th-cam.com/video/S-Xmcig1ddA/w-d-xo.html
'OneMarcFifty Virtual Network in Proxmox for MPTCP Test lab' at both ends & potentially every pi relay in-between
with cheap integrated m.th-cam.com/video/wEtlbNir8KE/w-d-xo.html LoRa LoRaWAN Range World Record Attempt (Re-published No. 120) Andreas Spiess for collective instrumentation & small quick txt, alerts etc
..collect up the sw tools and shell script it to make a working one off on demand clean usb Debian super light 'slax' slax org customize .php deployable from pi-KVM etc
i just wanna say, i set this up today as a linux noob. your tutorial was so easy to follow, but i did have pi knowledge. i had a big fight with errors, but fixed them by giving 'pi' docker perms [thanks issue tracker!]
this tool is so nice. i have pi hole just to track queries, as i have two ad blockers already. the internet-tracker is super nice. im gonna gather data on this as i love seeing data.
cheers!
Sooo considering speedtest-cli uses 1-1.3 GB every test, every 30 minutes would equal to around 50 GB a day or more. * 30, that's 1.5 TB a month, even with my gigabit internet through Comcast that would exceed the 1.2 TB/month quota and I would end up paying for extra bandwidth even if I never used my internet during those 30 days lol.
Good point-I didn't realize data caps were that low, OUCH!
I might change the default to every 60 minutes at most, to help protect people from ISP stupidity. I also plan on making it easier to configure the interval so people could choose once a day or something more for long-term monitoring.
@@JeffGeerling I still really like this project, very impressive, I think I'm going to order a couple Shelly plugs! Thanks again for another awesome video! Long live the Pi generation. :)
I left Xfinity as soon as AT&T Fiber arrived. They actually threatened to BAN me for a year for using over 1 TB (back when 250MB was the limit). I had to move over to Comcast Business and sign a contract to get unlimited data. Then when they lifted the cap on their Xfinity (Residential) side they wouldn't let me move over. And then when the contract expired and AT&T Fiber arrived I said good bye and good riddance.
Thank you for the video Jeff. What about the impact to the other processes in your network while the test is happening? Have you noticed it? Also, the other way around, does the traffic in your network affects the result of the tests?
Verizon FIOS supposedly doesn't have a data cap, but I would not want a speedtest going off while trying to game online. Good video though. Maybe someone could make a much shorter version of speedtest.
THanks for this vid. I finally got it installed (I'm the guy that commented about 15 times over the last day or so in the 'issues' section!). Great package, even though it was a nightmare to get it installed lol. Thank you!
so I started with 1 Pi as my DAC - now I am at 12 (doing many things - not music only), and now I need a dashboard to my Pis ;-)
A monitor to monitor all your Pis!
One monitor to monitor them all
Dude, get a hypervisor and run VMs, lol. 12 tiny physical boxes replaced with one big physical box running infinity tiny boxes
@@realityveil6151 Dude, you cannot put VM into active speaker to be working as LMS client and DAC. And this is only one example - actually 5 of my Pi are LMS clients and DACs. Everything has its purpose
I did run your docker compose and within minutes it works flawlessly ! THX
0:11 Ah, for 1 Gb/s down and 400 Mb/s up it's 20€/month in France.
In Mumbai we have FTTH , 1gb up and down with 3tb cap at around 56€/mo plus voip and a bunch of ott services .
I live in a rural area of Northern California, USA. $130/month for cellular based service… 10 down/3 up. Still wait for my Starlink service.
@@x5Deadmeat5x hope you get it soon👍
gbit down and 50 mbit up is 50€ in germany
In france, I pay 38€ for a sold _up to_ 8Gb down/1Gb up which is, in fact, a measured 5Gb down/1Gb up connection per month with no data cap.
I don't complain with the difference, i was never able to use my connection at its full speed.
Thanks for this. I already have a Pi running PiHole connected to the router, but I've been using a scheduled task on a windows machine to run an hourly speedtest to track my internet speeds. This looks like a much better setup.
It should be noted that if you have internet speed greater than 300 Mb/s, you MUST use a R Pi 4 as the 3B+ only has a 300 Mb/s NIC. Otherwise great video as always!
If you want the automated speed tests, then yes. If you're just using it as a pi-hole, then no. The traffic doesn't flow through the Pi; it just resolves DNS queries. But good point nonetheless.
Almost every ISP shares bandwidth. Unless you are paying for it you are sharing bandwidth with all the customers that are in your area. The analogy I use is a water hose. There is only so much water that can be pushed thru the hose at any given time. Bandwidth is the same. Great video as always!
Nice video as usual 🙂
Well, I think that this problem is relatively extended in every country of the world I guess.
Until 20 years ago in Italy there was a single telephone and internet operator but, at the end of this, nothing changed to much because the entire national infrastructure still was owned by that company, so the other "new companies" had to rent part of the network.
To be short, this caused a lot of issues as there were more ISPs on the same infrastructure that wasn't upgraded as needed up during the years mainly for the high costs: here if you want to "put down" a new line/fiber you have to broke up streets as the old pipes are too small and full of old cables.
Anyway, until 5 years ago I was paying 45€ monthly for a 6 Mbps ADSL, then the fiber (FTTC 100/20) finally came and, as I'm not so far to the cabinet I'm not losing too much signal.
I still paying likely 50€ monthly but at least now all it's working as intended.
Ah, phone calls are included 😆
Sorry for the poem and thanks for your work here.
Your channel and this video specifically has helped me catch Cox throttling my speeds and get them to give me my actual internet speed back. Went from 200Mbps to ~700-940Mbps with no hardware changes on my end. Thank you!
Thanks Jeff!
Once again you provide an excellent set of tools for us. I'll be using a Pi 4B in my cluster for a dedicated instance of this project.
Can I run openmediavault and PIHole on the same raspberry pi at once?
I am so glad you posted this. I have "suffered" through years of service via charter at only 200mbps down and 11mbps up. I too care only about the upload speed since 200mbps down was totally adequate. I upgraded to gigabit just so I could get the 40mbps up. It's ALMOST adequate now. Symmetrical would make everything right in the world!
Even if the asymmetric were more like 500/100, that would be leagues better than it is today.
I found out years ago that the offered bandwidth is only guaranteed from the demarc point to your nearest ISP first hop.
So meaning from my home to ISP but not my home to ISP to Google for instance? So the advertised bandwidth means literally nothing if my ISP's connection to google is slow in that example?
@@spacescout87 exactly. But hen again, they have no control of the internet between their point and Google (or whatever you connect to).
So there are peering agreements for ISPs, they have to peer with someone else to get access to the internet. Literal ISPs for the ISPs. These are expensive contracts for multi gigabit connections. Not too mention BGP ASN’s and the constant battle to obtain more v4 addresses from ARIN or other addressing entities. They do have their own battles.
@@DeadlyDragon_ In a past job, I was the one to have to fight those battles - can 100% confirm they're tough. We were able to operate with only one ASN (as many nets are) by using the same routing policy everywhere, and we addressed (pun not intended) the IPv4 shortage problem by going the DS-Lite route (pun still not intended).
But all of that was a walk in the park compared to the challenge of purchasing enough transit to meet customer demand. We kept having to readjust our oversell ratios as we updated our traffic engineering practices, and getting a good handle on what "peak demand" looks like is a surprisingly complicated question. We weren't yet big enough to get caching appliances from Akamai, Netflix, and friends... not that I wasn't checking constantly. But we always strove to do right by our customers - small ISPs have to or they don't survive. (So be a customer of a small ISP! Support local businesses that actually give a crap about you!)
Glad I wasn't still working there for the COVID pandemic. I would hate to be the one to have to figure out how to handle a bunch of WFH traffic overnight! :)
@@CFSworks
Oh, if only I lived somewhere a small isp existed.
Finally someone else noticed this problem! Every time I had an internet problem and called my ISP, I couldn't believe how little monitoring they do. ISPs should ENCOURAGE their customers to set up monitoring like this if they're not going to.
Grafana is the only service that I wanted to have for a long time, but still very confused about how to start…. Video series Jeff ? ;)
Grafana on itself does nothing, you need some data to display in Grafana. Typically this would be in a database (in this video, it's Prometheus, but it could also be MySQL, SQL Server, etc). But then you also need something to put in that database in the first place, which in this video are the speed test results and the website checks.
Im thankful a smaller local ISP started up near me. I'm a few miles outside the ST Louis metro area and I just went from only having satellite to a fiber connection which is faster by 100 fold. Spectrum wanted 20k 6 months before Gateway fiber just ran the line down our street for free if we payed for a subscription. I pay for a gigabit connection. I still see what you get most of the time like 700-900 down depending on the time of day but my uploads are always right around 900+. Still very happy with them, I only spend about 90$ a month on it.
Hi Jeff, it would be great if u did a video about backing up the sd card on a pi to a nas. I am kinda worried it will fail and having to set everything up again.
In Thailand, many ISP can let's customer to set up Internet speed by their owners from the ISP website, for example, customer package is 500/500 Mbps, customer can set speed 500/500 or 600/400 or 800/200 or 900/100 Mbps.
It would be awesome to have a real time traffic monitoring in place. As the pi hole is already recording DNS requests call, it would be pretty nice to see what clients are currently doing on the Internet, since the pihole frontend it’s basically just a list of requests that were resolved. This data may probably be relevant to show such a real time traffic, but it itself doesn’t record the traffic volume for instance.
Really appreciated your explanation, from which I learned that the telecommunication infrastructure of the U.S. is a bit outdated...
Hey, you're sitting in the city with your 1 Gbps plan, out here in St. Charles County we get 25/5 and even then AT&T can't deliver it without some spottiness!
Hi, thanks for this! Your tutorial was very helpful. It has a steep learning curve and took me about a day to set it up on two orangepi boards. Now, I can finally monitor my two ISPs.
One missing feature. With everyone monitoring their ISP 7/24 there needs to be an opt in option for users to send anonymized data with ISP, Location down to the city level to a central site on the internet. Then this can be a public service that anyone can search their location/city and see the averages that users monitoring 7/24 in that city are getting from each ISP including outages. This will allow everyone to know before hand which ISP is better to go with and turn this project into the next level.
I'd like to also see a crowd-sourced ISP statistics! This way, no more pesky ISPs claiming "no our internet is not the problem, go restart your router."
This is a really cool idea, but I can't think of a way to stop ISPs from poisoning the data. As the service gets more popular, it's more likely that ISPs would send fake data to it
Speedtest dot net already collects this kind of info, not 24/7 but it does save the speeds and providers for analytics.
Samknows
I really appreciate the work you put in your projects. Yesterday I was wondering if I could use my Pi to check which of all the ISPs would provide the best internet quality and here I have an entire git repo to test! I will be testing 3 providers connected to my Pi via usb eth adapters. Thanks a lot!
Now I'm wondering how can I run multiple Pi-hole and the Grafana dashboards for each network interface.
I feel you with cable upload speeds. Here in Ontario I can get 1000/30, which definitely hurts to pay for when all I really care about is upload!
You know the pain!!!
You're lucky having 30. Here I can only get 1000/10. It's almost always saturated, and I have to host a lot of stuff on the cloud simply because my connection would get choked as soon as someone remotely starts a download.
Cry’s in Australia internet getting 40/10 which is concerned good for my area
I wish I had the bandwidth or data cap to leave a device plugged in running speedtests every so often.
Yours is truly a first world problem Jeff. xD
CBI:- Lets track em
Jeff:- U know na I got y'all
Timely as always. I was just this afternoon looking for a home network bandwidth monitoring solution! Cheers Jeff.
Really great videos, and information on new and upcoming hardware, the only problem I see, is your github instructions on this are not the greatest. For someone like myself that is still learning about linux systems by utilizing a raspberry Pi, again for someone like myself hat knows enough to be dangerous, can really get frustrated with install instructions that are lacking, and would make someone want to just throw in the towel all together and stop learning more about raspberry pi's. This is by no means a dig on you, people like you are what made me want to get into buying my 1st raspberry Pi, but again with the github install instructions, its really hard to even tell what exactly it is that I am installing, or how specifically. Am I installing Ansible, or Pi Hole with docker, or do I have to have docker already installed, is it installing ansible? it's really hard to understand what all the instructions are having me do.
Wow! As a "stats junkie" and a Pihole user, I really like that project's results!
The bit I would maybe mention/change would be the automatic speedtest. I understand there is no other way to get "real" data for this, but I would fear that the sporadic tests would impact my internet experience. If someone in the house is playing a game for instance, they'll get lag spikes every half hour for "no reason".
Other than that, I might try this out on a raspberry pi 3 I have laying around and if it works well, have it replace my current pi4 config.
Thanks for the awesome content as usual!
Samknows fixed this issue by monitoring the network activity. If there is any significant Internet usage, the device would do the tests later. I'd love to try this project (I'm on a 1Gbps connection) but with the current prices and stock shortage for Raspberry Pis 4 and 5 it seems it will take a while.
Man, this channel is a gem for the open source community. Love your content, keep it up 👍
Thanks for this awesome project. I have already installed it and it's working great.
Cool Project !
looks like my pi 3b+ is getting some more work to do than just host a discord bot and a local chat server :)
@The Techie Scientist what kind of chat server do you host? I also want to try that out but I cant decide what application I should choose^^
@@paraniodify nodejs socket.io and express
@@TheTechieScientist thx mate :)
you could also monitor the power grid frequency. This gives indication about the stability of the grid and whether any power provider had issues with their power plants or maybe even an outage
In India internet is pretty cheap it's about 54 dollars for 1Gbps connection with no data cap and 10 dollars for 100Mbps connection with no data cap
In Italy we pay about 25/30 dollars for a stable gigabit connection. BUT, we pay the same for 100/50/20 or 7 Mbps ADSL!
Are you talking about the fiber broadband? There is a data cap of 3.3 TB. They just say it is unlimited.
@@UtkarshAmitabhSrivastava ya you are right but 3.3tb more than enough 1 tb is enough because generally we don't reach that limit
@@pavan13 yes that is enough for most people, but I think the gigabit plan should have higher cap. You could use all of it in just a few hours.
@@pavan13 If that's cheap, what about 1Gbps down and 500Mbps up, unlimited data, for $7?
OMG!! This is exactly what I'v been trying to accomplish at my home for months now. Failure after failure I had already giving up. Thank you so much. This shit is the bomb
Quick question..does this work on a raspberry pi running ubuntu server??
Friend in Hungary pays €10 a month and gets 600Mb/s down and 200Mb/s up...
I'm not impressed!
Yup, DIGI FTTH is 1Gbps/300Mbps GPON for 3100HUF (approx 8-10EUR)
@@84GDi I'll have to tell my friend she was stitched up! 🤣
@@84GDi In Spain Digi offers 1Gbps/1Gbps for 20€. In Spain, there are only symmetric connections.
@@pgp-acc for 30€ you could get a 50/5 over cable here in germany. Or 100/10 for 35€
@@pgp-acc Most connection in HUN are asymetric dl/ul (even business packages other than VDSL), but for 7200HUF/mo (approx 20-22EUR) Telekom offers 2000/1000 on fiber here too (for residential consumers). Not anywhere of course, we have areas where ADSL on crap-old phone copper still cost like 9k HUF (30EUR) and barely reach 10M/500kbps up.
I did something similar but in my laptop and found that my ISP was slowing it down by more than two-thirds at times. When confronted, they doubled down saying that data consumption was "too high" and that it could only mean I was getting the speeds I was promised. Ofcourse, when you calculated the average speed it was a little over this throttled speed. The data consumption was just because of the speedtests running every 2 minutes in the background.
I switched my ISP as soon as I found an alternative. Best decision ever.
Symmetrical. You would prefer 150mbps of symmetrical.
Asymmetrical means exactly what you already have with the 1gbps down and 40mbps up.
Don't sweat it Jeff, I made this mistake too.
Yes, oops!
If you download at 1gbps your upload is almost maxed out sending ACKs which is probably part of the reason you „only“ get 930 Mbit/s
They beat me to it, but yeah...I much preferred my 75M synchronous/symmetric FiOS at my old address over the 1G asynchronous/asymmetric Spectrum Business at my current one. Unfortunately I have no other options besides cable here, the phone lines are too old to support more than 128k (per AT&T) and there is no fiber in the area.
@@MarkusOrt It's mostly TCP overhead which if you had a gig would come out to around 949Mbps max. There's just no way to get a 1000Mbps on a 1000Mbps link negotiation.
An OpenVPN GUI install and a DDNS updater on this box and it becomes something I can bill for, as opposed to a rabbit hole of hours spent behind my keyboard. Great work!
I wonder when you will move to Home Assistant ...slowly :D
Heh, it may happen, we'll see. I think there's already a drop-in plugin for the Shelly Plug there.
@@JeffGeerling Yea there is an integration for Shellies among many others :)
I called Xfinity because my Internet had been down from around midnight to 7 AM. X: "No, your Internet connection is currently up." Me: "I'm talking about last night. X: "Yes, it's currently up." ...minutes later... X: "Yes, it was down, and we'll credit you $10 on your next bill." (was never credited, but I gave up) Ever since, I have been running a ping test from a Pi every 5 minutes to track outages. Nice job on your tool and video!👍
I get 990Mbps max min 950Mbps on my gigabit fiber ping under 5ms and 530Mbps upload for 31 euro
/me is jealous!
None of the European comparisons are apples to apples. The large geography, fragmented regulation etc. of large land mass countries will always allow pockets of monopolies and more expensive operations than small densely populated land areas. You get gigabit or more for less than the price of a pizza in countries like South Korea et al. The better measure is the quality of service and uptime from any ISP.
@@pd8559 Well I pay $7 for 1 gigabit down and 500Mbps up. Real speeds are constant 940Mbps down, 510-520Mbps up. And as for the uptime, I am pretty much always online (me and my servers) and in 10 years I had 1 single time about 2 hours of Internet outage. SO I would say the uptime is good. I live in Romania.
In Chattanooga we get municipal gigabit fiber (1000 up & 1000 down) for $70/month. Pretty sweet.
Thanks for the video! 👌
Well they clearly state "up to" 1Gb per second.
This makes it right? No.
I like this cow shot at the end.
*Laughs in European*
Thank you for a very informative video!
I have had a NextCloud test running in my house on a RPi3. It did the job and worked ok, but I now want to upgrade it to a Pi4-8GB with a 6 TB drive and booting from a 512GB NVME in a USB enclosure. What I am wondering about is if I should use docker for the install so I can run the speedtest you mention here as well?
I live in Brazil, and the rules here is the the ISP HAS to deliver 40% of the contracted speed at all times. NEVER below that number. But - they also HAVE to deliver a minimum of 80% measured over a month (30 days?). The big reason for upgrading the Pi is to get GB network and USB3.
Oh - I almost forgot - I have 450/225 Mb on fiber.
I just set it up and submitted my first pull request for it, thanks!
Spectrum is the only non-DSL internet I can get at my address. Even paying for the "highest bandwidth", I get maybe 1/2 of that most of the day, with pretty regular dips and outages.
A lot of companies seem to have changed their wording in advertisements using things like "maximum bandwidth, regular service at this speed not guaranteed" or something similar.
100% gonna add this next week to my pihole settup. Our ISP rarely ever gives me my full upload speed and it’s horrible because upload is all I want.
I have the same plan same price with Spectrum, no other options. It goes out often and one time it took them 11 months to figure out a packet loss issue smh. I currently have a Pi 3b+ setup to email me ping results every 6 hours via a cron. Unfortunately the Pi 3b+ caps out at ~330Mbps in a speed test due to I believe the USB controller iirc so I am thinking of upgrading it to a 4. It’s also currently my Pihole. Great vid!
You know the pain!
Jeff, you should get or add an Ripe Atlas Probe! Contributing to the global data research as well as having a great additional monitoring for your ISP as well as connectivity and routing aka peering.
When I tried to switch to an LTE ISP, I had to talk to the CenturyLink customer retention specialist. They pretty much gave up when I told them "Look, I'll pay you $200/mo if you will give me 100Mbps down." They act like they don't understand how 3Mbps is not sufficient for anything except for email/browsing nowadays. Without ad block, my wife's PC consumes the full 3Mbps down when she opens her forums due to all the auto starting videos/ads. I currently pay $90/mo for two 3Mbps dry DSL circuits.
CenturyLink/Frontier/Hughes/Dishnet are going to be in a world of hurt when Starlink gets rolled out in earnest...
DUDE! Just what I was looking for. I am paying for 200mbit but I am pretty sure ISP has me @100 and they are lying... Let you know maybe when I find out wth they are doing :) Great vid!
I watched to the very end and I appreciate the artistry of the final cow mooing. It worked, dude. And yeah, Pi-Holes kick ass.
I work for charter/spectrum in Louisiana and our gig speed is much closer to a gig than what yours is showing. Could be a few things but we had issues with the first docsis 3.1 modems we had that showed similar speeds to what yours is but now that we have the newer version of the 3.1 modem the speed is consistently higher than before. Also what others are doing on the internet in your area doesn't affect your service much of at all, but what can is the plant not being properly balanced and your node just having to many modems on it among other things. Also we're those speeds directly out of the modem or through the router bc spectrum only guarantees 70%+ directly out of the modem.
Can't wait to get home from work and get this set up on my Pi, amazing!
One thing the continuous testing does not account for is the conditions on the server side. We are essentially assuming server side never experiences slow downs or partial outages(causing you connection to be re-routed). I'm sure most of it is still your ISPs doing, but just maybe a small disclaimer. Also one thing to consider is your ISP charges you for a 1 Gbps connection from your home to their head end. Once your connection reaches that location and is routed to other data centers, it's largely out of your ISPs control. That's maybe something ISPs could be more clear about. The internet and how you accomplish connecting to it is not as cut and dry as we would like to be.
4:50 - YES!! There is little to no reason ISPs couldn't offer a 150 symmetrical service. DOCSIS 3.0 supports 200Mbps upstream, 3.1 is over 1Gbps, and DOCSIS 4.0 is 6Gbps. They should make that the standard just to properly compete with fiber ISPs.
I've been doing something similar for a while, though in my case I have mine set up using node-red, home-assistant and influxdb.
Every hour node-red will run a speed test, parse the JSON output and send it to home-assistant... which in turn records it into influxDB for long term storage.
I have to wonder, what is the impact of your internet usage on the speed tests / the speed tests on your internet usage.
I would assume that speed tests during peak usage periods on your network will have reduced results because other devices are using the available bandwidth too.
That's true, and one of the things you have to discount and/or test further in terms of monitoring at your router as well. I've considered switching to InfluxDB too (Grafana's happy with whatever).
I would love to set something up like this that also can reset the modem when it detects an outage. I live in a poorly connected area and have a DSL connection that drops almost daily and often times power cycling the modem seems to do the trick. I'd love to either use the Shelly Plug or the GPIO pins connected to a relay to reset the power every time the internet is out for more than a couple minutes.
I have no idea what this is going to be about, but Jeff replied to a Stuff Made Here comment on a Smarter Every Day vid, so I checked his channel. Then he has a Princess Bride thumbnail to boot?! Almost subbed sight-unseen.
2.5 min in - Holy crap this is useful stuff! Subbed
One day this man will land a rover on Mars using only Raspberry Pi's
Great video Jeff, earlier this year I migrated all my home intel servers to raspberry pi’s and some cloud compute. I have a pair of pi 4’s running redundant asterisk pbx services. And a pair of pi 4s running pi hole service, with a lonely pi2 running homebridge, all the pi4’s use Poe hats and the pi 2 running a Poe dongle thing. They are great. Looking at the video would love to setup internet monitoring and maybe some lan monitoring. Probably need to use some snmp traps rather than raw point to point from the pi as running 10Gbe between all the switches and NASs but pie only at 1gbe. Thanks again.
I hope next-gen Pi SoCs can handle 10 GbE (not built-in, but via PCI Express), would make for some fun new projects!
It's running speedtest through the pi's network port? We have 1gb fiber and we have a 2.5gb link to the pfsense box and a 10gb backbone. It woke the network right up!
I'm already using pi hole on a dedicated pi wired to my router so now I'll look into adding the other things you listed but my main goal is to see downtime when I have no internet at all. A report showing date, time, and duration of outages is what I was really wanting until spectrum came out and replaced the cable from my box to my router since it was testing bad. My parents are signed up to get star link since their only choice is hughes net satellite so a pi doing the same stuff as yours will be nice to monitor how good its doing.
So i have recently seen a project where a pi was slotted into a pci e bracket. The project used poe power but im interested to see if a pci card could be made that allows power to be pulled from the slot its self. I would think it has plenty to use. Ethernet could also probably be passed through but i think a second ethernet cable would still be more efficient. There was a suggestion to also use an internal usb to usb hub to power it as well.
For some reason (on two PI's, one brand new Grafana will only bind to TCP6
I like it. When I get one of those round tuits I intend to set up something like this. It would be fun to have a dashboard to monitor a hydroponic greenhouse (using Shelly of Sonoff plugs).
Your content is always interesting. Thanks.
Where I live, when you have high latency or slower than promised speeds, you call the ISP and it’s fixed in a few hours. ToS go both ways.
Greetings from Estonia!
Jeff this was awesome and was able to get it up and running with minimal fuss. thank you sir, pivpn and nas are next!
Thanks for this video. This is a project I am going to do. It's really cool to find out you live in St. Louis too! That's very close to where I live and work. :)
Hello neighbor!
@@JeffGeerling I'm running into an error....I've worked through a couple of others but this one has me stumped: fatal: [192.168.xx.xx]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: pi@192.168.xx.xx: Permission denied (publickey,password).", "unreachable": true} Help would be appreciated.
Interesting set-up. I stopped using PiHole a few months back when a TH-cam update broke the block lists and I couldn’t make exemptions for all the weird Google ad domains and ended up not able to watch any YT for a week until I just gave up.
The cat-and-mouse game with TH-cam ads finally convinced me to pay for Premium. Not advertising for it, but since I watch a lot of TH-cam it was worth it for me.
Great project 👍
Thanks for your experience with all of us 👍😀
Thank you Jeff. I'm gonna clone your exact build. It'll be interesting to see how much my ISP shafts me in South Africa.
You have *_TWO_* options for internet?! In the United States?! You're a god among men.
This guy is awesome, I really enjoy his videos.
It seems half way through this video I started printing the Pi-Rack :D Thanks
Ha, well it'll take a while!
Some ISP in Germany have policies that prohibit to test connection speed too often. Every 45 minutes can be considered as "too often" as it can have very big impact on the ISP network if there are many people doing this.
Care to share which and where they state this as a policy?
I've never heard of it and would like to look into it, but couldn't find anything on my preferred search engine.
@@grmasdfII it's stated in the "Allgemeine Geschäftsbedingungen (AGB)". don't know the English term for that. I think 1&1 is/was one of the few who actually enforced that some times. I think my cable TV and internet provider (pyur) has this policy too but i didn't mind to look it up as I'm not planning to do connection speed tests that often. It's mostly formulated very "open". Like "any behavior that can cause unnecessary load to the ISP network".
@@kane587mad Ah, the "don't be a dick" clause. I doubt a check once every 30 minutes will be a problem.
Home assistant has been monitoring my internet for years. It's actually quite consistent, but the 10mbps upload is a real drag. I'm waiting for AT&T fiber to finally get installed in my neighborhood.
BTW, @JeffGeerling, have you inquired about a "business" cable connection. They often have the same upload and download speeds, and shouldn't be much different than the $150 you're paying.
great video. I think you should mantion that some cables cannot handle 1gb speed. example: 1gb requires cat6 cable