small correction at 8:03 MQTT is not a.k.a. mosquitto, the former is the name of the protocol and the latter is just a free and open-source implementation.
Just some additional info here: Depending on the broker you use, you can allow clients to only subscribe, publish or do both on specific topics. This whole thing can be easily mitigated, by only allowing administrators (or the cloud service itself) to actually publish new messages on topics, which the devices are listening to. As I understood this vulnerability, you authenticate "as the device" to control the device - you don't authenticate "as an admin" to control it.
@@Apokalyptika91 Or sign your messages. The server should be signing all it's messages. Even more bulletproof is only allow a specific list of commands with typed arguments.
Working with MQTT in my daily work as a software engineer. From Wikipedia: "Historically, the "MQ" in "MQTT" came from the IBM MQ (then "MQSeries") product line, where it stands for "Message Queue". However, the protocol provides publish-and-subscribe messaging (no queues, in spite of the name). In the specification opened by IBM, as version 3.1, the protocol was referred to as "MQ Telemetry Transport". Subsequent versions released by OASIS strictly refer to the protocol as just "MQTT", although the technical committee itself is named "OASIS Message Queuing Telemetry Transport Technical Committee". Since 2013, "MQTT" does not stand for anything."
At least they reversed the serial number before hashing it as password. Security by obscurity at its best. Next time they might add a c13 rotation (Caesar 'encryption'). 😂
My thought exactly. If it's discovered later after some state actor has already been using it, they can claim "whoopsie" even if it was actually intentional. Kind of like the Juniper backdoor from a while back, or any number of "oops, I suck at programming and nations have been using it to their advantage" incidents.
@@ytmadpoo Are we soon looking to incarceration due to "stupidity" ? I guess negligence is already a criminalizing thing. It baffles me how these things escape audits that should be mandatory for routers!
Not being a security expert by any stretch, my question is: why should we suspect that deliberate malice is more plausible than not here?? (ie. Is this sequence of bugs so egregious that it should never have emerged due to incompetence alone??)
idk man these things are usually not developed by one person. the guy doing the authentication probably didnt know the serial number was being broadcast, and vice versa
@@smb1397 but whats up with the guy who did authentication using the serial number in the first place lmao, bro thought reversing it would be good enough
The scary thing is - I've seen people ("developers") who genuinely thought that because they didn't know better. I'm sure you agree that good developers are more expensive than regular, fresh from the diploma-mint-factory ones. And developers that are good in auth & crypto in general? Even more expensive (for a good reason).
Yes, but in this case, stupidity alone doesn't explain the unlikely convenience of these bugs. Stupidity can explain some of the design choices, but the fact that all these design choices + specific bugs led to be able to get all the required informations needed to hack the devices with almost no effort is indeed rather suspicious.
I highly doubt Hanlon’s razor. The idea of using your serial number for encryption has been a known problem for YEARS. Add that to purposefully broadcasting what is essentially the key to your shell, and it makes no sense.
There was a similar exploit with Vodafone EasyBox routers a decade ago that I used to get internet access before I got my own. They were a bit worse because the default Wi-Fi password was derived from the MAC address.
oooo man, can't wait for Monday, I am going to rub this in so good. This video on big screen with big boss present and everyone who did not listen to me when I told them not to buy this overpriced, untested junk. It's going to be fun 🎉.
You overestimate the ability of management to understand or even care about security. I dont know where you work, but i honestly would just assume nobody apart from any programmers you might have even knows what RCE means or how bad it really is
@maxave7448that just means he needs to explain it in management terms. Hours spent patching/fixing vs liability lawsuit when their device is responsible for someone getting hacked at home. Home security systems, the panic of ioT baby cameras with a speaker, ‘Personal’ photos, and “just think of the kids” If the C suite isn’t puckering so hard they suctioned their chairs, they have less imagination than a rock.
so there are two kinds of management: 1. You were right, we messed up, have to take actions ASAP. 2. It's entirely your fault, you are so incompetent you cannot secure an access pint, you are fired
Once you get past dumping the firmware this seems so straightforward that it's asking to be found - not good for an obscured backdoor. Equally, if it is incompetence then it seems like no-one with a tiny bit of knowledge really even thought about security which also seems unlikely. So it's incompetent implementation both ways, either of device security or a backdoor😅
I don't know much about this company but I don't think it is a one-man operation. That means that this should have been approved by multiple people before releasing to the public, which makes it way worse.
I so badly want to buy the courses at Lowlevel academy but since i already have tons to learn already from other sites which I have purchased, i am controlling myself. I am sure i will come back after i finish those. Your videos are just splendid and mean a lot so much so that I skip a heartbeat when i see a new video from you. Keep doing these videos and thank you so much from all of us.
That kind of developers ignorance to product quality is the reason that I don’t have any "smart" device in my home, unless it runs the OS that I personally ser myself (OpenWrt on network devices, OpenIpc surveillance cameras) or strictly cut out off the network access by being caged in a dedicated VLAN.
Let me stir up a little paranoia :) How many packages do you have installed on your openwrt devices? How many of them expect to run as root, or they won't work at all? Do you think anybody audited them all (each single version) and also made sure that they play together nicely? If so you have my full admiration - I'd like to be happy like this, too! :) From what I could tell openwrt is better than the regular consumer grade router manufacturer in that they can at least fix the vulns quicker after they are found, but only nat/iptables/nft sit between the big bad internet and your multiple untested packages running with uid 0
Idk.. this sounds kinda like incompetence. The person responsible for writing the networking/MQTT part probably wasn't aware of the lower-level details. And they were prolly like, "all this password needs to do is _feel_ random. So long as no one knows the method/algorithm I use, they won't be able to tell how to break it." Definitely not me haha 😅😅
I agree, also if they wanted a backdoor they could still have had unique passwords (that could be generated from a private known seed) and it wouldn't look suspicious
I find it funny we as a community came up with the mantra of "never do your own security scheme", for, like, the last 3 decades, and people still do this stupid mistake... Like, we have a good solution, and still a company makes millions while hiring the kind of developer who does this sort of thing... 🤔🤦🤷
Your channel's great. The best on your subject, imho. Keep it up? I hope you're making enough to keep at it, it's truly making the internet a better place.
A lot of people are talking about programming, but there are two groups at play here. In order for the vendor to identify the HW, you need a root of trust, a shared secret between the vendor and the HW. Ideally, it should be burned into one of the chips. If the HW group never gives a root of trust, the SW group has to make do with what they do have. So this could be pathalogical cost savings. (Edit) after reading more, I don't think the company employs any security-minded people. Most egregious is the lack of ACLs in the messaging service, thereby allowing every device to impersonate the company.
For many devices, the serial number is explicitly not sensitive/private information, as it's written on the product's retail box (for network devices, its often written next to or near the MAC address).
Great content! I’m not a security expert but been in tech for 20 years and always felt like when it comes to security if you are a target there is nothing you can do, someone, or a team skillful enough will find a way in, so I sleep good at night not being in cyber security 😆😆
True to some degree, but everyone has something of value. Most people have a bank account, for instance. Everyone is a target, it's just that some targets are more juicy than others. (more valuable/less defended)
Never attribute to incompetence that which can be explained by malice. Also, just because you're paranoid, doesn't mean they aren't out to get you. 🙂 I also live by these creeds. Have served me well.
4 วันที่ผ่านมา +3
It's nice of them to broadcast the credentials so you don't have to deal with all that pesky hacking
10:54 @Low Level, it says in the doc that for the username, it takes the serial+identifier from local configuration. Doesn't that mean that in order to obtain the username you have to get SSH access to the device first?
15:55 The saying is: "Never attribute to malice what can be reasonably explained by incompetence". The key word being "reasonably". I personally don't find it reasonable to implement a Remote Procedure Call system and not even whitelist the allowed commands. But then again, I've been developing web applications for government entities and corporations for over 6 years and I've seen stuff like that too many times. The only time everyone was rushing to patch a vulnerability was for the Log4j RCE, while having mission critical software that to this day still runs with Java 7...
I would guess almost every o day is intentional. Just like cups and ipv6 it could not be more obvious. A random but can cause security issues but that is obviously by design and random bugs tend to give security issues not directly executing code. Yet this is more and more what we see, very suspicious bugs.
I wish there was a way to "legislate away" this level of incompetence, like "if your software is executing a shell command line at any point, it is illegal to ship it with consumer devices, without a secury audit from a certified computer security auditor. Every individual update of such software must also be re-audited"... But I already see the problem with that approach... We'd end up in a CrowdStrike like situation, where devs have the audited core be too generic, so as to avoid re-certification, and move the bulk of the logic in upper layers, where upper layers being upgraded cause it to be briked.
The whole point of being a good programmer is learning how to read documentation and build a model and be able to iterate on it. Courses are good if you don’t know anything. Besides that, just read the documentation, write your own code, and look up what you don’t know/understand. That’s what I did, what’s what all these TH-camrs who sell courses did as well.
Excellent job on "pièce de résistance" with lovely american accent (mosly piece isn't with an e sound but a è wich sound a bit like ea in bear and the final e needs to be said), but you nailed résistance and that's the most important word! (native french speaker)
So this is basically the inverse of "security by obfuscation"! Something tells me that the guy that came approved this idea got a big fat bonus when he shipped the product in 3 months and never ever bothered to actually think about it 😆😆
That device impersonation vulnerability is really an oversight in the standard as MQTT does not define any topic based permission controls. This is left to the broker where it can or can't be implemented by configuration or plug-ins, depending on the vendor. I've seen it unfold myself in a similar way, many devs see mTLS and shut their brains off right at the gate. I only advise for MQTT for truly public and uncritical sensor data. If you want to use it for anything remotely critical or confidential, control and configure the broker if possible or sign / encrypt the messages (which defeats the purpose of pub/sub IMHO).
Yeah, I've used mqtt for a truly public sort of messages, and it's fine for that. Also, importantly, the video mentions a very important note about not relying on a serial number. The team who implemented this was just a tiny bit too clever with popular pieces, and overestimated what they could do.
Ruijie Network is a Chinese company which has connections with PLA. It is in Fujian province where Xi Jinping used to be major. China empire worried local people who have close connections with Taiwan, thus they heavily restricted investment in Fujian province and high tech vendors like Ruijie all connections to PLA or intelligence. Not sure why US government hasn’t boycotted it.
the part where they just broadcast their serials in management frames is so silly but at the same time so fricking cool, cuz that would allow adversaries to perform some watch dogs style shit
5:20 Entropy being very high might instead also mean it's compressed. However, almost all practically used compression algorithms have a way of being recognized as such by deeper inspections. However, deep inspections are more costly & more time consuming, so your mileage may vary.
We need more open source firmware. Should be legally mandated that if a company doesn't want to maintain something anymore to fix vulnerabilities, it becomes open source
and if you add a "Y" to MQTT, you get MQTTY. And that's what it is. Sending "direct-to-shell" commands from the cloud to execute directly? That's a no-no. Imo, devices should be dumb and do just one thing. And that thing very good.
Using the Serial number as the credential isn't the issue in this case IMO. The real issue is that they absolutely failed at setting up the ACLs on their broker, and thus leaving it open to this kind of abuse. If say, the broker only allowed devices to ONLY publish to the topics meant for sending to the cloud, and only allowed their cloud to publish to the topics the devices subscribed to, and only allow each device to subscribe to it's own SN topic... This could have been prevented entirely.
In that case, even if someone guessed the SN for a device, at most they could do is subscribe and receive commands from the cloud (which, granted, could still give away confidential information like wifi credentials or something), but at least they wouldn't be able to use the broker to send arbitrary messages to a device. Even if they used mTLS for MQTT authentication, the lack of any ACLs on their broker would still likely have allowed compromise anyway, assuming their MQTT broker doesn't have ANY ACLs enforced at all.
Stay out of my moms office ED!! Side note, I got in LLA at the right time, glad to see business is doing well. For those on the fence. Ed WILL teach you well. I recommend it hands down.
A Chinese company. They often have bad security as it is a requirement in China that the government can access anything. There were some hire scooters in my locality a few years ago. I could access those even when someone was already using it, and could read out it's current speed, etc. Most fun was had by telling it to reset while it was in use though (a walking pace chase :-) ). I think there is a guy called Matt Brown that finds many similar flaws in Chinese devices.
I think this is just "the firmware team talked to the network team but the product manager didn't understand it" bug. Or worse, "I followed all the tutorial to meet my deadline" bug.
broadcasting serial sounds as a good idea generally speaking. And using serial as a password "seed" is also an elegant solution - for a POC or something :D
For such cloud devices, they really need to use a private certificate embedded into hardware that will do the validation, like that of a security key or a sim card
Never attribute to malice what can adequately be ascribed to incompetence? Maybe. However a consistent pattern of "mistakes" and deliberate actions by a party and their known affiliates that repeatedly and consistently favors that party's interests should not be considered accidental. Especially not if you're in security. Ruijin is one of China's leading providers of network infrastructure technology. I wonder if they run the same firmware inside China as they do in the rest of the world?
Never assume that just because a large community has scrutinized a technology that it works properly. Several times I've found bugs in techs used all over the world. Everyone thought it worked. The world was wrong.
What would be interesting to know is if the same person or persons wrote both parts of the code. If they did then can ask some questions to determine if purposeful or not.
I don't believe it could be a back-door 3 "innocent" bugs in a row, that give you acted to the root shell? Its just too complicated that anyone could come with such idea
If this chain constituted an intentional back door, they went to a lot of trouble to feign incompetence. Why not just use per dev certs and sell CA access?
You are, indeed, a paranoid freak! 😃 Possibly because you know how this should be done correctly. I suppose the team who developed the firmware was chosen based on cost :) So they either didn't know what you know to begin with, or they "forgot" it to cut costs and be "competitive". The same rationale probably happened with the security auditing and testing - IF there even was such a thing. Just guessing here, based on what I've seen for many years. BTW, your French for the "pièce de résistance" was very good 😊 And you chose a good expression to describe it, I wasn't expecting them to broadcast those serial numbers to everyone 😂
The article glosses over “an identifier it reads from a chip” combined with serial number. This sounds like it would require more than just a serial number to exploit then?
That sentence is a bit poorly written. It just says that the serial number is read from ROM and used as the username when the MQTT service is first set up.
Serial numbers are often sequential… next_serial = serial + 1. Sometimes shared into batches or production sites, but still - very commonly sequential neighbors. You could mass exploit knowing just one or a few serial numbers, unless they have done the unusual step of true random serial numbers…
These might be the Access Points uses at my university... Gotta try and have a close look. I often wonder how you turn a crash into an exploit. Like I get someone to crash and it unexpectedly printed all my user data(untrusted user code) to stderr and hence my shell... surely I need to have some exploit in that shell to turn a print(stdout or stderr) into an exploit. Or some downstream app that relies on this pipe to provide useful data like a "valid"/"not valid" response.
As a programmer i am being asked to do things quickly rather than correctly which means i don’t have time to learn proper security implementation. I am certain i would have made such bug.
Your first pronunciation of Meraki (muh raw key) (sometimes mur awe key) was correct. The company uses greek pronunciation (or at least as best as English speakers can produce it). The Google provided "American pronunciation" is just wrong.
Not really. You can connect it to a remotely/cloud hosted controller, but it is neither required nor an unavoidable default. Put the IP of your local controller into your DHCP server, and the devices will not even try to talk to the cloud when they cannot find a local controller. Also, with Unifi the pairing of devices to the controller looks solid and there seems to be no way for any device to send control commands anywhere, only the controller can do that.
16:14 I have to feel like this is just incompetence. The devs were probably told to make it work, not make it secure. Sounds like big structural problems to me.. The biggest issue has to be the authentication to the broker, to me this part really feels like it was written without the necessary knowledge of security.
Been watching your videos for a few months now and I love how you break down the topics and explain the "hacking" steps. That said, the "is this incompetency or malice?!" feels way out of left field and very sensationalized. An awfully slippery slope from "bug" to "conspiracy". But I am not an offensive security specialist, maybe you see it differently and that just didn't come across in the video
legitimately if this doesn't surprise you, you haven't worked around enough corpo garbage, honestly the users are lucky they even bothered to reverse the serial number for the key
The problem with basing Your "homeCooked password scheme" on the devices serial number is not only that users don't treat their serial-numbers as "sensitive information". Or that "hackers" can hack any devices who's serial number they know. Rather I would argue that the bigger problem is that it's trivial to automate a bruteforce attack on ANY device of that "class", simply by knowing the "general format" of the serial number for that type of device.....
If it were malace, all they would need to do is hash the serial number with a salt known only to the company to generate the password. That's so obvious and simple that there is no chance of this being a deliberate 'backdoor'. Nope, this will be a bit of code written by a developer under pressure that probably has a comment in the source along the lines of _# this is for testing only. Bill needs to decide how we want to do this securely before production deploy._ Never put down to malice that which can be explained by incompetence, but do not rush to blame incompetence for that which can be readily be explained by laziness and/or forgetfulness.
14:34 And here I was betting on easily enumerated serial numbers, then asking the AP's where they are to create a list of targets you can break at any time. But no this is just as stupid.
Just locking down the MQTT broker so that connections with one username would kick each other, and not be able to publish to /:SN, just to the reporting topics (which should ofc be namespaced with :SN too) would nullify most of this. But I guess they are allowing arbitrary client IDs that dont match the username? What for?
Serial Numbers (12:00) - I wouldn't be surprised if you could interrogate them to find out their SN like via NBT or Wi-Fi. Fing Called it (14:52) - So obviously I wrote the above before watching the entire video. It's almost like I've heard this kind of story before. Malice or Incompetence (16:30) - Assuming I understand the infrastructure correctly, I'm going with incompetence here. With this kind of infrastructure, the smarter malicious play would be to "backdoor" (a.k.a. provide a user with r/w acls to every topic) the cloud server where it's harder to investigate. So even if it was malicious, it was kind of incompetent at it.
there are no backdoors at lowlevel.academy where you can learn the fundamentals of programming (i hope so)
Hey brother, can you add the links you're sharing to the video description?
shit mb
thats a really strong statement, does the building have only one door?
u sure? Seems like Vuejs to me. Running behind Nginx (no Nginx Plus).
I like what you're doing. Maybe you could examine D-Link routers sometime soon. I would even mail one to you, on me.
I need a sign in my house that says "Days since last zero-day vulnerability" and then the counter I will just leave at zero indefinitely
Until someone hacks the sign.
It should display a signed int overflow
I would legit buy that as a bumper sticker
With an rss ticker listing latest zero-days
Only if the sign is WiFi connected.
small correction at 8:03 MQTT is not a.k.a. mosquitto, the former is the name of the protocol and the latter is just a free and open-source implementation.
Just some additional info here: Depending on the broker you use, you can allow clients to only subscribe, publish or do both on specific topics. This whole thing can be easily mitigated, by only allowing administrators (or the cloud service itself) to actually publish new messages on topics, which the devices are listening to.
As I understood this vulnerability, you authenticate "as the device" to control the device - you don't authenticate "as an admin" to control it.
I'm willing to bet money they're now busy RTFM-ing to learn how to put ACLs in place!
@@Apokalyptika91 Or sign your messages. The server should be signing all it's messages. Even more bulletproof is only allow a specific list of commands with typed arguments.
aka Message Queuing Telemetry Transport
Working with MQTT in my daily work as a software engineer. From Wikipedia:
"Historically, the "MQ" in "MQTT" came from the IBM MQ (then "MQSeries") product line, where it stands for "Message Queue". However, the protocol provides publish-and-subscribe messaging (no queues, in spite of the name). In the specification opened by IBM, as version 3.1, the protocol was referred to as "MQ Telemetry Transport". Subsequent versions released by OASIS strictly refer to the protocol as just "MQTT", although the technical committee itself is named "OASIS Message Queuing Telemetry Transport Technical Committee". Since 2013, "MQTT" does not stand for anything."
I audibly gasped at the "THEY BROADCAST THE F.. SERIAL NUMBER" part
At least they reversed the serial number before hashing it as password.
Security by obscurity at its best. Next time they might add a c13 rotation (Caesar 'encryption'). 😂
@@ralfbauerfeind8236 Rot26 anyone? Twice as safe!
We all did! (gasp)
Somebody thought I don’t want to get a ladder to go look at the box to read the serial every time.
Lol, and I almost sent my face to the back of my head with the facepalm I did when I heard you can just execute cli commands using mqtt messages!
Zero days already? Starting the year off strong!
Yup let's goooo
Competing with 2024
Yeah but it’s the 9th so we are late 🤓
zero days since no zero days have been detected
Nice. Even if serial was not broadcast, it's extremely low entropy and easy to guess/brute force...
Perfect thing to distract me from my vulnerability meeting right now.
nononono plz pay attention
@@LowLevelTV na, crappy SIEM false positives. Rapid 7 can go straight to the dumpster.
sounds like an emotional support group
@@Elesario It is, we are all sick of this shit. If it weren't for Xcode I'd likely have gone to mobile development years ago.
@@Elesario "hi, I'm Eric, I've seen three zero-days this week and I'm terrified about the future"
(everyone unanimously): "Hi Eric"
Weaponized incompetence is the latest APT. Plausible deniability provides great cover.
yeeeeeeeeeeeeeeeeeeeeeep
I am so done with this free pass for incompetence, any publication calling these attacks now, is getting on the black list. They are bought shills.
My thought exactly. If it's discovered later after some state actor has already been using it, they can claim "whoopsie" even if it was actually intentional. Kind of like the Juniper backdoor from a while back, or any number of "oops, I suck at programming and nations have been using it to their advantage" incidents.
@@ytmadpoo Are we soon looking to incarceration due to "stupidity" ? I guess negligence is already a criminalizing thing. It baffles me how these things escape audits that should be mandatory for routers!
Not being a security expert by any stretch, my question is: why should we suspect that deliberate malice is more plausible than not here?? (ie. Is this sequence of bugs so egregious that it should never have emerged due to incompetence alone??)
Deciding to use the serial num as authentication and then send it clear text...that doesnt feel like an accident 😆
Precisely my thought. That seems like a fully intentional backdoor.
idk man these things are usually not developed by one person. the guy doing the authentication probably didnt know the serial number was being broadcast, and vice versa
@@smb1397 but whats up with the guy who did authentication using the serial number in the first place lmao, bro thought reversing it would be good enough
@@hamburgerfatso Sadly happens very often. Some people think hiding the generation process is security.
The scary thing is - I've seen people ("developers") who genuinely thought that because they didn't know better. I'm sure you agree that good developers are more expensive than regular, fresh from the diploma-mint-factory ones. And developers that are good in auth & crypto in general? Even more expensive (for a good reason).
16:00 It’s Hanlon’s razor. “Never attribute to malice that which can be adequately explained by stupidity.”
hanlons razor is a psyop
Good rule of thumb, but sometimes way too many strange things align in way too many ways.
Yes, but in this case, stupidity alone doesn't explain the unlikely convenience of these bugs. Stupidity can explain some of the design choices, but the fact that all these design choices + specific bugs led to be able to get all the required informations needed to hack the devices with almost no effort is indeed rather suspicious.
I highly doubt Hanlon’s razor. The idea of using your serial number for encryption has been a known problem for YEARS. Add that to purposefully broadcasting what is essentially the key to your shell, and it makes no sense.
The corollary is of course "...but never _rule out_ malice, either." Weaponized incompetence _is_ malice, functionally speaking.
> password is a mangle of the serial number
Oh you've got to be kidding me
> broadcasts serial number publicly
Oh you've *got* to be kidding me
There was a similar exploit with Vodafone EasyBox routers a decade ago that I used to get internet access before I got my own. They were a bit worse because the default Wi-Fi password was derived from the MAC address.
Who said security through obscurity is not a thing?
oooo man, can't wait for Monday, I am going to rub this in so good. This video on big screen with big boss present and everyone who did not listen to me when I told them not to buy this overpriced, untested junk. It's going to be fun 🎉.
You overestimate the ability of management to understand or even care about security. I dont know where you work, but i honestly would just assume nobody apart from any programmers you might have even knows what RCE means or how bad it really is
@maxave7448that just means he needs to explain it in management terms. Hours spent patching/fixing vs liability lawsuit when their device is responsible for someone getting hacked at home. Home security systems, the panic of ioT baby cameras with a speaker, ‘Personal’ photos, and “just think of the kids”
If the C suite isn’t puckering so hard they suctioned their chairs, they have less imagination than a rock.
Don't just tell them about it as a security risk, explain to them what can happen, and how it affects the bottom line.
so there are two kinds of management: 1. You were right, we messed up, have to take actions ASAP. 2. It's entirely your fault, you are so incompetent you cannot secure an access pint, you are fired
Weird and sinister behaviour.
This takes "free internet for everyone" to a new level of meaning.
Once you get past dumping the firmware this seems so straightforward that it's asking to be found - not good for an obscured backdoor. Equally, if it is incompetence then it seems like no-one with a tiny bit of knowledge really even thought about security which also seems unlikely.
So it's incompetent implementation both ways, either of device security or a backdoor😅
can we call it a front door then?
It's the dog and pony show to distract us from the real back doors/bot net contained in better coding.
I don't know much about this company but I don't think it is a one-man operation. That means that this should have been approved by multiple people before releasing to the public, which makes it way worse.
Thank you very much for video, we appreciate it as always. Best regards from Germany
That’s very kind thank you so much
Thank you for donating he really deserves it!!
I so badly want to buy the courses at Lowlevel academy but since i already have tons to learn already from other sites which I have purchased, i am controlling myself. I am sure i will come back after i finish those. Your videos are just splendid and mean a lot so much so that I skip a heartbeat when i see a new video from you. Keep doing these videos and thank you so much from all of us.
That kind of developers ignorance to product quality is the reason that I don’t have any "smart" device in my home, unless it runs the OS that I personally ser myself (OpenWrt on network devices, OpenIpc surveillance cameras) or strictly cut out off the network access by being caged in a dedicated VLAN.
Let me stir up a little paranoia :) How many packages do you have installed on your openwrt devices? How many of them expect to run as root, or they won't work at all? Do you think anybody audited them all (each single version) and also made sure that they play together nicely? If so you have my full admiration - I'd like to be happy like this, too! :) From what I could tell openwrt is better than the regular consumer grade router manufacturer in that they can at least fix the vulns quicker after they are found, but only nat/iptables/nft sit between the big bad internet and your multiple untested packages running with uid 0
Idk.. this sounds kinda like incompetence. The person responsible for writing the networking/MQTT part probably wasn't aware of the lower-level details. And they were prolly like, "all this password needs to do is _feel_ random. So long as no one knows the method/algorithm I use, they won't be able to tell how to break it." Definitely not me haha 😅😅
I agree, also if they wanted a backdoor they could still have had unique passwords (that could be generated from a private known seed) and it wouldn't look suspicious
I find it funny we as a community came up with the mantra of "never do your own security scheme", for, like, the last 3 decades, and people still do this stupid mistake... Like, we have a good solution, and still a company makes millions while hiring the kind of developer who does this sort of thing... 🤔🤦🤷
Then you are trusting someone else to make the mistakes for you.
Your channel's great. The best on your subject, imho. Keep it up? I hope you're making enough to keep at it, it's truly making the internet a better place.
The S in IoT stands for 'security' and the P for 'privacy'...
A lot of people are talking about programming, but there are two groups at play here. In order for the vendor to identify the HW, you need a root of trust, a shared secret between the vendor and the HW. Ideally, it should be burned into one of the chips. If the HW group never gives a root of trust, the SW group has to make do with what they do have. So this could be pathalogical cost savings.
(Edit) after reading more, I don't think the company employs any security-minded people. Most egregious is the lack of ACLs in the messaging service, thereby allowing every device to impersonate the company.
For many devices, the serial number is explicitly not sensitive/private information, as it's written on the product's retail box (for network devices, its often written next to or near the MAC address).
As someone that works in the CCTV industry, oh god 💀
New Low Level video, W, always so informative and entertaining!
hey I seriously appreciate that.
Ty, not bug this is feature..
My years of sw development tell me it was sheer stupidity. Most people don't care
Agree.
Great content! I’m not a security expert but been in tech for 20 years and always felt like when it comes to security if you are a target there is nothing you can do, someone, or a team skillful enough will find a way in, so I sleep good at night not being in cyber security 😆😆
True to some degree, but everyone has something of value. Most people have a bank account, for instance. Everyone is a target, it's just that some targets are more juicy than others. (more valuable/less defended)
The source code for devices like these should be required to be open source by law. We can't trust these companies to handle our security.
Never attribute to incompetence that which can be explained by malice. Also, just because you're paranoid, doesn't mean they aren't out to get you. 🙂 I also live by these creeds. Have served me well.
It's nice of them to broadcast the credentials so you don't have to deal with all that pesky hacking
10:54 @Low Level, it says in the doc that for the username, it takes the serial+identifier from local configuration. Doesn't that mean that in order to obtain the username you have to get SSH access to the device first?
15:55 The saying is: "Never attribute to malice what can be reasonably explained by incompetence". The key word being "reasonably". I personally don't find it reasonable to implement a Remote Procedure Call system and not even whitelist the allowed commands. But then again, I've been developing web applications for government entities and corporations for over 6 years and I've seen stuff like that too many times. The only time everyone was rushing to patch a vulnerability was for the Log4j RCE, while having mission critical software that to this day still runs with Java 7...
I would guess almost every o day is intentional. Just like cups and ipv6 it could not be more obvious. A random but can cause security issues but that is obviously by design and random bugs tend to give security issues not directly executing code. Yet this is more and more what we see, very suspicious bugs.
I wish there was a way to "legislate away" this level of incompetence, like "if your software is executing a shell command line at any point, it is illegal to ship it with consumer devices, without a secury audit from a certified computer security auditor. Every individual update of such software must also be re-audited"...
But I already see the problem with that approach... We'd end up in a CrowdStrike like situation, where devs have the audited core be too generic, so as to avoid re-certification, and move the bulk of the logic in upper layers, where upper layers being upgraded cause it to be briked.
The whole point of being a good programmer is learning how to read documentation and build a model and be able to iterate on it. Courses are good if you don’t know anything. Besides that, just read the documentation, write your own code, and look up what you don’t know/understand. That’s what I did, what’s what all these TH-camrs who sell courses did as well.
That's some good advice!
Excellent job on "pièce de résistance" with lovely american accent (mosly piece isn't with an e sound but a è wich sound a bit like ea in bear and the final e needs to be said), but you nailed résistance and that's the most important word! (native french speaker)
So this is basically the inverse of "security by obfuscation"! Something tells me that the guy that came approved this idea got a big fat bonus when he shipped the product in 3 months and never ever bothered to actually think about it 😆😆
That device impersonation vulnerability is really an oversight in the standard as MQTT does not define any topic based permission controls. This is left to the broker where it can or can't be implemented by configuration or plug-ins, depending on the vendor. I've seen it unfold myself in a similar way, many devs see mTLS and shut their brains off right at the gate.
I only advise for MQTT for truly public and uncritical sensor data. If you want to use it for anything remotely critical or confidential, control and configure the broker if possible or sign / encrypt the messages (which defeats the purpose of pub/sub IMHO).
Yeah, I've used mqtt for a truly public sort of messages, and it's fine for that. Also, importantly, the video mentions a very important note about not relying on a serial number. The team who implemented this was just a tiny bit too clever with popular pieces, and overestimated what they could do.
Why should mqtt do that? It is a transmission protocol. What to do with the message once it has been transferred is outside its scope.
nice job, but to be honest, I preferr to see the old format, where there is actual code being written and explained. I can read a CVE myself :(
Some of us bozos dont have the attention span to find and read CVEs :(
But to be fair I also miss when he actually wrote code
Most mind-blowing: why are clients allowed to post to the topic where commands are issued to the clients?
Ruijie Network is a Chinese company which has connections with PLA. It is in Fujian province where Xi Jinping used to be major. China empire worried local people who have close connections with Taiwan, thus they heavily restricted investment in Fujian province and high tech vendors like Ruijie all connections to PLA or intelligence. Not sure why US government hasn’t boycotted it.
These things are all over Japan, that much I know.
My personal takeaway: It could be in your home office, or your mom's office. Cardinal Biggles, we may be here a hot minute.
the part where they just broadcast their serials in management frames is so silly but at the same time so fricking cool, cuz that would allow adversaries to perform some watch dogs style shit
Oddly enough it would be more forgivable if they didn't sha256 the serial, then it would look like a mistake. But they tried to make it look random.
"Never ascribe to malice that which can be explained by incompetence".
Hanlon's Razor was what you were looking for.
Finally signed up for Low Level Academy. WE GETTIN OUT OF TUTORIAL HELL WITH THIS ONE!! 🔥🔥🔥
5:20 Entropy being very high might instead also mean it's compressed. However, almost all practically used compression algorithms have a way of being recognized as such by deeper inspections. However, deep inspections are more costly & more time consuming, so your mileage may vary.
A compressed file usually has some sort of header. zip files do at least and it's how I have found them in malware for manual extraction.
We need more open source firmware. Should be legally mandated that if a company doesn't want to maintain something anymore to fix vulnerabilities, it becomes open source
and if you add a "Y" to MQTT, you get MQTTY. And that's what it is.
Sending "direct-to-shell" commands from the cloud to execute directly? That's a no-no.
Imo, devices should be dumb and do just one thing. And that thing very good.
Using the Serial number as the credential isn't the issue in this case IMO.
The real issue is that they absolutely failed at setting up the ACLs on their broker, and thus leaving it open to this kind of abuse.
If say, the broker only allowed devices to ONLY publish to the topics meant for sending to the cloud, and only allowed their cloud to publish to the topics the devices subscribed to, and only allow each device to subscribe to it's own SN topic...
This could have been prevented entirely.
In that case, even if someone guessed the SN for a device, at most they could do is subscribe and receive commands from the cloud (which, granted, could still give away confidential information like wifi credentials or something), but at least they wouldn't be able to use the broker to send arbitrary messages to a device.
Even if they used mTLS for MQTT authentication, the lack of any ACLs on their broker would still likely have allowed compromise anyway, assuming their MQTT broker doesn't have ANY ACLs enforced at all.
When you said "cloud connected" it took me back to listening to In Flames as a teenager
Stay out of my moms office ED!! Side note, I got in LLA at the right time, glad to see business is doing well. For those on the fence. Ed WILL teach you well. I recommend it hands down.
A Chinese company. They often have bad security as it is a requirement in China that the government can access anything. There were some hire scooters in my locality a few years ago. I could access those even when someone was already using it, and could read out it's current speed, etc. Most fun was had by telling it to reset while it was in use though (a walking pace chase :-) ). I think there is a guy called Matt Brown that finds many similar flaws in Chinese devices.
That password generation system has to be a placeholder that accidentally made it to production
Some poor intern is getting absolutely booted 💀
if they're dumb enough to try and roll out their own password generation mechanism, they're dumb enough to implement it like that
@maxave7448 or they left after their internship 🤣
These are systemic issues, no one person for such a large company should be ever in charge of this.
At 0:52 , I believe he meant to say "raises the question" - it's confusing to try to figure out what he meant with the phrase otherwise.
I think this is just "the firmware team talked to the network team but the product manager didn't understand it" bug.
Or worse, "I followed all the tutorial to meet my deadline" bug.
broadcasting serial sounds as a good idea generally speaking. And using serial as a password "seed" is also an elegant solution - for a POC or something :D
0:54 oh no conspiracy eddy is at it again!
What did we learn class made me laugh. Awesome video as always
For such cloud devices, they really need to use a private certificate embedded into hardware that will do the validation, like that of a security key or a sim card
Never attribute to malice what can adequately be ascribed to incompetence? Maybe. However a consistent pattern of "mistakes" and deliberate actions by a party and their known affiliates that repeatedly and consistently favors that party's interests should not be considered accidental. Especially not if you're in security. Ruijin is one of China's leading providers of network infrastructure technology. I wonder if they run the same firmware inside China as they do in the rest of the world?
For me, it's typically a team bug!
Each developer on the team works on tasks without knowing the entire project, which can cause this kind of issue.
Never assume that just because a large community has scrutinized a technology that it works properly. Several times I've found bugs in techs used all over the world. Everyone thought it worked. The world was wrong.
An easy way to check for malice would be to see if the same access points installed in China, have the same set of vulnerabilities.
IoT is the security gift that just keeps on giving.
Never attribute to malice that which can be more easily explained by incompetence.
This smacks of different people/departments being responsible for different parts of the system, and no-one joining the dots
What would be interesting to know is if the same person or persons wrote both parts of the code. If they did then can ask some questions to determine if purposeful or not.
Wait, the office next to me uses Ruijie devices.
The expression you were looking for is Hanlon's razor: Never attribute to malice that which is adequately explained by stupidity.
I don't believe it could be a back-door
3 "innocent" bugs in a row, that give you acted to the root shell?
Its just too complicated that anyone could come with such idea
"Never chalk up to malice what can be adequately explained by incompetence," say the malicious.
If this chain constituted an intentional back door, they went to a lot of trouble to feign incompetence. Why not just use per dev certs and sell CA access?
Because then, it is intentional.
You are, indeed, a paranoid freak! 😃 Possibly because you know how this should be done correctly. I suppose the team who developed the firmware was chosen based on cost :) So they either didn't know what you know to begin with, or they "forgot" it to cut costs and be "competitive". The same rationale probably happened with the security auditing and testing - IF there even was such a thing. Just guessing here, based on what I've seen for many years.
BTW, your French for the "pièce de résistance" was very good 😊 And you chose a good expression to describe it, I wasn't expecting them to broadcast those serial numbers to everyone 😂
The article glosses over “an identifier it reads from a chip” combined with serial number.
This sounds like it would require more than just a serial number to exploit then?
That sentence is a bit poorly written. It just says that the serial number is read from ROM and used as the username when the MQTT service is first set up.
Serial numbers are often sequential… next_serial = serial + 1. Sometimes shared into batches or production sites, but still - very commonly sequential neighbors. You could mass exploit knowing just one or a few serial numbers, unless they have done the unusual step of true random serial numbers…
Very good point, understand the serial number scheme, own them all.
These might be the Access Points uses at my university... Gotta try and have a close look.
I often wonder how you turn a crash into an exploit. Like I get someone to crash and it unexpectedly printed all my user data(untrusted user code) to stderr and hence my shell... surely I need to have some exploit in that shell to turn a print(stdout or stderr) into an exploit. Or some downstream app that relies on this pipe to provide useful data like a "valid"/"not valid" response.
As a programmer i am being asked to do things quickly rather than correctly which means i don’t have time to learn proper security implementation. I am certain i would have made such bug.
Your first pronunciation of Meraki (muh raw key) (sometimes mur awe key) was correct. The company uses greek pronunciation (or at least as best as English speakers can produce it). The Google provided "American pronunciation" is just wrong.
Unifi is another cloud based network stack
👀
They will f up a release and make them all public.
Not really. You can connect it to a remotely/cloud hosted controller, but it is neither required nor an unavoidable default. Put the IP of your local controller into your DHCP server, and the devices will not even try to talk to the cloud when they cannot find a local controller.
Also, with Unifi the pairing of devices to the controller looks solid and there seems to be no way for any device to send control commands anywhere, only the controller can do that.
14:11 Better than GoodTimesWithScar
1:08 Occam’s razor
14:15 we say _plat de résistance_ (not piece, and the "t" is silent)
Mikrotik, fortunately, does not have direct control from the cloud, it only has cloud DDNS and config cloud storage.
16:14 I have to feel like this is just incompetence. The devs were probably told to make it work, not make it secure. Sounds like big structural problems to me..
The biggest issue has to be the authentication to the broker, to me this part really feels like it was written without the necessary knowledge of security.
Been watching your videos for a few months now and I love how you break down the topics and explain the "hacking" steps. That said, the "is this incompetency or malice?!" feels way out of left field and very sensationalized. An awfully slippery slope from "bug" to "conspiracy". But I am not an offensive security specialist, maybe you see it differently and that just didn't come across in the video
These are not “bugs” they are design decisions.
its not a bad way to do it actually its so you can contact support tell them the seial and then they know the credentails to push an update!
Yes 👀 ever heard of ECDH?
If it's not intentional this is criminal stupidity
Me when the company tells us we can't run raspberry pi's inhouse, so they buy proprietary hardware that does the same thing that is "safer"
legitimately if this doesn't surprise you, you haven't worked around enough corpo garbage, honestly the users are lucky they even bothered to reverse the serial number for the key
I had the same reaction as in 15:42 😂
The problem with basing Your "homeCooked password scheme" on the devices serial number is not only that users don't treat their serial-numbers as "sensitive information". Or that "hackers" can hack any devices who's serial number they know. Rather I would argue that the bigger problem is that it's trivial to automate a bruteforce attack on ANY device of that "class", simply by knowing the "general format" of the serial number for that type of device.....
If it were malace, all they would need to do is hash the serial number with a salt known only to the company to generate the password. That's so obvious and simple that there is no chance of this being a deliberate 'backdoor'.
Nope, this will be a bit of code written by a developer under pressure that probably has a comment in the source along the lines of
_# this is for testing only. Bill needs to decide how we want to do this securely before production deploy._
Never put down to malice that which can be explained by incompetence, but do not rush to blame incompetence for that which can be readily be explained by laziness and/or forgetfulness.
cloud connected and router should never be uttered in the same phrase
14:34 And here I was betting on easily enumerated serial numbers, then asking the AP's where they are to create a list of targets you can break at any time. But no this is just as stupid.
Using a serial number as a password is like having the same user name and password. The serial number can be considered as a user name.
Just locking down the MQTT broker so that connections with one username would kick each other, and not be able to publish to /:SN, just to the reporting topics (which should ofc be namespaced with :SN too) would nullify most of this. But I guess they are allowing arbitrary client IDs that dont match the username? What for?
Serial Numbers (12:00) - I wouldn't be surprised if you could interrogate them to find out their SN like via NBT or Wi-Fi.
Fing Called it (14:52) - So obviously I wrote the above before watching the entire video. It's almost like I've heard this kind of story before.
Malice or Incompetence (16:30) - Assuming I understand the infrastructure correctly, I'm going with incompetence here. With this kind of infrastructure, the smarter malicious play would be to "backdoor" (a.k.a. provide a user with r/w acls to every topic) the cloud server where it's harder to investigate. So even if it was malicious, it was kind of incompetent at it.