Cool, I’m not the only one doing this 😊. Interesting to see a different take on having a house full of robots. I skipped ROS too, because I wanted to learn the theory myself, and make everything modular along different lines.
I feel like the idea of ROS is great. A plug and play robot system, in theory it should greatly reduce development time. but it's just so big , the learning curve and limitations it necessarily imposes that you have to work around kind of work against that goal.
Randomly stumbled upon this video earlier. I thought it was interesting and decide to watch the whole series. I love the series so far and can't wait to see where your project goes. I would love to do something similar one day. :)
Very cool. Ive always daydreamed about building robots that work in domestic environments. Super interesting to see the challenges you face and overcome. Good luck in your project
Great video. Doing everything from scratch is a great way to learn but keep in mind that you could still use ROS at any point in your project if you like to. This might also be interesting for others who like to learn robotics. Lets say you find a nice ROS-library that does something you like to use. You just need to write a simple ROS node that connects somehow (in your case probably via socket) and forwards the data as ros topic or that receives ros service calls, handles them and send that back to your software stack. It would also later be interesting to see how your own solution benchmarks against a default ROS 2 nav 2 stack. You can also run ROS outside of the robot, in a previous IoT project we were running ROS in the cloud to run plansys 2 for high-level planning while all connected devices that provide data were connected with their own protocols (it was quite the zoo with websockets, kafka and mqtt).
I feel you, I'm with you. Thanks for inspiring. Seeing somebody else doing their thing on the same topic gives you ideas and reminders (oh, pose estimation, dang, of course I must from the get go align that data to embodied/proprioceptive self-understanding of the robot). Thanks. We all have a common problem and goal here. 🙂 Subscribed.
Hey joe, very interesting series 👍 can't wait for the update. Uhm, so I wanted to ask why are your videos in reverse order in the Playlist? Is it sth I could fix from my account or do you do it on purpose for some reason 👀
HaHa!! I don't move my fridge too often either!! Awesome progress with GORT! Great solution on the motor rev to stop the robot from falling. I need to create a similar solution in my robot as well. Did I miss what sort of fall or tilt sensor you are using for that?
I'm using some AS5600 "Magnetic Potentiometers". They're really convenient, just stick a magnet to the end of the shaft, and they read it's orientation with 12-bit resolution. Highly recommended!
I am enjoying your videos. I am planning on making a similar project at some point. Maybe when I get going we can compare notes! Will you be adding arms or an arm to the robot at some point? Also will you make the robot more modular so it can swap parts for certain tasks? These questions maybe beyond where you are at this time. Just curious but still good work and I hope you achieve your goals!
That's cool! I am planning on having arms... technically it only needs one, but but I plan on having two. I don't plan on having any attachments... unless the robot can change them by itself. One of my goals is to make it as independent as possible.
@@joethieman9400 Yes making it independent is definitely a great goal. Its one that I feel needs to be met to make robots more useful in general. I look forward to seeing your design expand and I hope my project can be as fruitful as yours is turning out!
Hi there, great project you are working on! The videos are well done. I currently work on a project using an Jetson Orin Nano Dev Kit just like you do. It an assistive robotic dog, made for helping visually disabled people navigate to an adress and many more. I'm currently at the stage of your Project video 1. I try to learn OpenCV with CUDA for my Jetson Orin Nano but cant find good online resources to teach me how to do it. Do you or any other have some recommendations where to start or get help from? Regards from Germany
I wish I had more to offer in terms of help... but for now I'm just using Jetson Inference packages for my machine vision. It's based on Tensorflow, instead of OpenCV. ...And I haven't even tried learning CUDA yet.. 😂 Here's what I have found Helpful: Dusty-NV github.com/dusty-nv www.youtube.com/@NVIDIADeveloper And JetsonHacks www.youtube.com/@JetsonHacks Hope this helps!
I must admit, I newer knew about Robot Operating System existing. Thanks for sharing. Anything that its meant to run on is probably out of reach for me at the moment. :D But I 100% agree with you that sometimes it is worth "reinventing the wheel" every now and then. My electronics is far better than my programming - Borland Pascal years ago and simple C programs on AVR micro controllers directly or on the Arduino; need to go deeper in embedded programming. You can learn so much by recreating even a bit older technologies yourself - 8 bit computers, game engines, some AI and Machine Learning stuff, if you have time, highly recommend for all new programmers and engineers.
ROS is said to run on a Raspberry Pi... so that's not too bad. :) Yeah, I started with electronics too, but then I got into microcontrollers and the software just went hand-in-hand with my projects ever since. I highly recommend learning Python... I was a hard-core C guy for years, but once I learned Python I realized how quick it was to prototype in, and how portable it is from one operating system to another. It's a handy tool in the toolbox.
@joethieman9400 Oh raspberry is ok then. I was expecting something more expensive like Nvidia Jetson or full fledge x86-64 computer. Just found an interesting project, may be interesting for some m.th-cam.com/video/QgqkCPKWOdw/w-d-xo.html A homemade airsoft sentry turret inspired by the game "Portal" The guy... doesn't talk much. But he has gave detailed lists of parts and instructions. Maybe visual aiming system of that robot will give ideas to someone willing to do their own build of...home assistant. Trough might be harder to follow for westerners (side topic, but culture affects technology. A lot of Japanese websites look convoluted for westerners).
@@joethieman9400 python isnt very performant. On top of that it can never do real multithreading due to an inherent design flaw called the Global Interpreter Lock, the GIL. The power it gets for ML all comes from callouts to compiled C and C++ libs like pytorch.
That's an interesting idea... My biggest worries though would be battery life and the noise running the flywheel all the time. Makes for some interesting possibilities though!
I think people get the wrong idea with ROS(2). Like they say it is to avoid reinventing the wheel, but while most just assume "oh so use other people's stuff..." It is more about not reinventing your own stuff. In the end it is just a fancy alternative to using stuff like Sockets that offers you a basic messaging framework for Publish-Subscribe and Reply-Request style messaging between Python and C programs/devices. To make it easier to copy/re-use/update stuff. to dismiss it cause "I WANT to reinvent the wheel!!" is a bit silly if you ask me as you aren't obligated to use other people's stuff. Now if only it was "friendlier" to a wider variety of linux distros. Being native to only Ubuntu makes it really annoying to use with Debian based systems like The Raspberry Pi and its own native OS. Like you gotta use Docker which if not familiar with can make it an nightmare for beginners that suddenly are looking at having to learn BOTH at the same time.
Oh yes! I have to be careful with my memory usage: the Jetson Orin Nano only has 8GB that has to be shared between the vision and LLM, but for now Phi 3b seems to work just fine. :)
I second @erniea5843.. ROS is a powerhouse, especially ros2 control. When you get the basics right, you can reuse code in any project out there, just that it has a learning curve that discourages beginners before they even start 😒
If you are doing segmentation and object recognition one way to avoid the steep price tag of the Jetsons is use an SBC that has integrated NPU like the Rockchip RK3588. It can far outperform the GPU on any SBC for these tasks. They use quantized versions of standard models like Inception and MobileNet and the excellent YOLO. As an inference optimized ASIC some can pump out hundreds of frames per second.
I'll keep that in mind... I've never used an SBC with an NPU. I'm pretty happy with how the Jetson performs, but I sure struggle with getting things to function on it properly.... probably user error, but still, it sure slows me down when I'm struggling just to get a camera working even before writing any code!
@@joethieman9400 I have lots of those SBCs. I found the Jetson Orin to be pretty good but Nvidia software is always gigabytes and always a hassle to set up. I havent managed to get more than 5 or 10 FPS out of it running python segmentation. On top of that its a power hungry little beast.
@@joethieman9400 It is not just you, working with/developing for the jetson platform is TRICKY. I started by downloading the jetson ai lab docker images and following those tutorials, then dove into the docker images themselves to see what they were doing in the backend. Using those as a template I was then able to develop my own tools on my desktop machine and package them for use on the jetson. Much better experience than developing on-device. The jetson has so many weird little undocumented dependencies and particularities when it comes to packages that you'll drive yourself mad starting from zero
I found ROS to be kind of a bloated pile myself, and I use Java for my robotic ecosystem. There used to be a ROSJava implementation but support was dropped years ago so I forked and rewrote it to be highly optimized. It now runs in a JAR less than 500k with near realtime performance on small SBC's. Instead of bloated XML/RPC I used native Java serialization. Unlike python, Java can do real multithreading. You get functional programming and pretty universal platform support. ROS is a great publish/subscribe bus that allows you to have fault-tolerant modular point to point comms across the whole robotic platform using generic networking gear and SBC's. Java supports callouts to native libraries for most low level functions like GPIO, video, etc. Maybe not right for you, but it can be done and thats how I do it with some vastly different physical robots that all share the same underlying software infrastructure.
Sounds like a nice setup. Right now I'm liking the way my modules are all independent and communicating through sockets. I can write any module in any language, and even run them all on different computers. It also isolates errors nicely. :)
@@joethieman9400 ROS and my ROSJavaLite also use sockets at their core. The master node initiates the publisher and subscriber nodes that then handshake up persistent socket communications point to point. They can be running on any node the network can get to. The master merely initiates the handshake and setup, then the complex pub/sub is all point to point traffic. Java has callouts to C and C++ so you get those languages if you like, The nodes attempt reconnect if something drops in the point to point comms.
@@vikaloseva6996 Very Badly... 😆 Right now updates usually break my code... so I need a solution to that. I've heard Docker is an answer to this problem... but I don't know anything about it yet. If you have suggestions, let me know! I really need it.. lol
Legendary youtube recommendations pull
The combination of the engineering with the self-balancing, and the ML stuff on the sensors and mapping is amazing.
Cool, I’m not the only one doing this 😊. Interesting to see a different take on having a house full of robots. I skipped ROS too, because I wanted to learn the theory myself, and make everything modular along different lines.
this is exactly how i wanna live. you sir got yourself a subscriber
this is the best channel on youtube. i wanna build a robot now.
The wheel may be the most re-invented invention of human kind. Nice videos btw
I feel like the idea of ROS is great. A plug and play robot system, in theory it should greatly reduce development time. but it's just so big , the learning curve and limitations it necessarily imposes that you have to work around kind of work against that goal.
hey man nice work , keep it going
This is amazing! I enjoy the process and I am learning tons to make my own very soon. Can't wait to see how this turns out.
Great Work!!!
Randomly stumbled upon this video earlier. I thought it was interesting and decide to watch the whole series. I love the series so far and can't wait to see where your project goes. I would love to do something similar one day. :)
Very cool. Ive always daydreamed about building robots that work in domestic environments. Super interesting to see the challenges you face and overcome. Good luck in your project
highly inspired and wait for next working process
awesome project!
Great video. Doing everything from scratch is a great way to learn but keep in mind that you could still use ROS at any point in your project if you like to.
This might also be interesting for others who like to learn robotics.
Lets say you find a nice ROS-library that does something you like to use.
You just need to write a simple ROS node that connects somehow (in your case probably via socket) and forwards the data as ros topic or that receives ros service calls, handles them and send that back to your software stack.
It would also later be interesting to see how your own solution benchmarks against a default ROS 2 nav 2 stack.
You can also run ROS outside of the robot, in a previous IoT project we were running ROS in the cloud to run plansys 2 for high-level planning while all connected devices that provide data were connected with their own protocols (it was quite the zoo with websockets, kafka and mqtt).
I feel you, I'm with you. Thanks for inspiring. Seeing somebody else doing their thing on the same topic gives you ideas and reminders (oh, pose estimation, dang, of course I must from the get go align that data to embodied/proprioceptive self-understanding of the robot). Thanks. We all have a common problem and goal here. 🙂 Subscribed.
"Gorp is back here with me. He's uh lost his face, but that will be fixed." :')
Instant subscribe 👏👏
Very Interesting
TH-cam algorithm on point with this recommendation
I am so happy that you are not using ROS
Why?
Very cool! Hope to see GORP conduct an introduction someday :)
That would be awesome! Gort can't talk yet, but it's in my plan!
Hey joe, very interesting series 👍 can't wait for the update.
Uhm, so I wanted to ask why are your videos in reverse order in the Playlist? Is it sth I could fix from my account or do you do it on purpose for some reason 👀
I had no idea... 😆 Thank you for letting me know!
Jetsons can run Nvidia Isaac ROS that will do a much better job of mapping the environment with a Realsense or ZED camera.
HaHa!! I don't move my fridge too often either!! Awesome progress with GORT! Great solution on the motor rev to stop the robot from falling. I need to create a similar solution in my robot as well. Did I miss what sort of fall or tilt sensor you are using for that?
I'm just using an MPU6050. it's cheap, but seems to do the job well.
speed is essential custom rules
My robot use the same motors. What encoders do you use?
I'm using some AS5600 "Magnetic Potentiometers". They're really convenient, just stick a magnet to the end of the shaft, and they read it's orientation with 12-bit resolution. Highly recommended!
wow. Do you use Nvidia ominverse for robotic issac sim.
Not yet, though it looks like a powerful system.
I am enjoying your videos. I am planning on making a similar project at some point. Maybe when I get going we can compare notes! Will you be adding arms or an arm to the robot at some point? Also will you make the robot more modular so it can swap parts for certain tasks? These questions maybe beyond where you are at this time. Just curious but still good work and I hope you achieve your goals!
That's cool! I am planning on having arms... technically it only needs one, but but I plan on having two. I don't plan on having any attachments... unless the robot can change them by itself. One of my goals is to make it as independent as possible.
@@joethieman9400 Yes making it independent is definitely a great goal. Its one that I feel needs to be met to make robots more useful in general. I look forward to seeing your design expand and I hope my project can be as fruitful as yours is turning out!
Hi there, great project you are working on! The videos are well done. I currently work on a project using an Jetson Orin Nano Dev Kit just like you do. It an assistive robotic dog, made for helping visually disabled people navigate to an adress and many more. I'm currently at the stage of your Project video 1. I try to learn OpenCV with CUDA for my Jetson Orin Nano but cant find good online resources to teach me how to do it. Do you or any other have some recommendations where to start or get help from? Regards from Germany
I wish I had more to offer in terms of help... but for now I'm just using Jetson Inference packages for my machine vision. It's based on Tensorflow, instead of OpenCV. ...And I haven't even tried learning CUDA yet.. 😂
Here's what I have found Helpful:
Dusty-NV
github.com/dusty-nv
www.youtube.com/@NVIDIADeveloper
And JetsonHacks
www.youtube.com/@JetsonHacks
Hope this helps!
I must admit, I newer knew about Robot Operating System existing. Thanks for sharing.
Anything that its meant to run on is probably out of reach for me at the moment. :D
But I 100% agree with you that sometimes it is worth "reinventing the wheel" every now and then.
My electronics is far better than my programming - Borland Pascal years ago and simple C programs on AVR micro controllers directly or on the Arduino; need to go deeper in embedded programming.
You can learn so much by recreating even a bit older technologies yourself - 8 bit computers, game engines, some AI and Machine Learning stuff, if you have time, highly recommend for all new programmers and engineers.
ROS is said to run on a Raspberry Pi... so that's not too bad. :)
Yeah, I started with electronics too, but then I got into microcontrollers and the software just went hand-in-hand with my projects ever since. I highly recommend learning Python... I was a hard-core C guy for years, but once I learned Python I realized how quick it was to prototype in, and how portable it is from one operating system to another. It's a handy tool in the toolbox.
@joethieman9400 Oh raspberry is ok then. I was expecting something more expensive like Nvidia Jetson or full fledge x86-64 computer.
Just found an interesting project, may be interesting for some
m.th-cam.com/video/QgqkCPKWOdw/w-d-xo.html
A homemade airsoft sentry turret inspired by the game "Portal"
The guy... doesn't talk much. But he has gave detailed lists of parts and instructions.
Maybe visual aiming system of that robot will give ideas to someone willing to do their own build of...home assistant.
Trough might be harder to follow for westerners (side topic, but culture affects technology. A lot of Japanese websites look convoluted for westerners).
@@joethieman9400 python isnt very performant. On top of that it can never do real multithreading due to an inherent design flaw called the Global Interpreter Lock, the GIL. The power it gets for ML all comes from callouts to compiled C and C++ libs like pytorch.
@@Sven_Dongle True that python can be slow to run, but it can be a lot easier and quicker to write for a simple prototype.
@@amazingmation97 Depends on how good you are, when youre good, anything is quick and easy, when youre not, wel...
2:40 why don't animals have wheels? Because they don't have streets 😊
How about using a gyro motor for balance
That's an interesting idea... My biggest worries though would be battery life and the noise running the flywheel all the time. Makes for some interesting possibilities though!
@@joethieman9400 Oh yh, makes sense
I think people get the wrong idea with ROS(2). Like they say it is to avoid reinventing the wheel, but while most just assume "oh so use other people's stuff..." It is more about not reinventing your own stuff. In the end it is just a fancy alternative to using stuff like Sockets that offers you a basic messaging framework for Publish-Subscribe and Reply-Request style messaging between Python and C programs/devices. To make it easier to copy/re-use/update stuff. to dismiss it cause "I WANT to reinvent the wheel!!" is a bit silly if you ask me as you aren't obligated to use other people's stuff.
Now if only it was "friendlier" to a wider variety of linux distros. Being native to only Ubuntu makes it really annoying to use with Debian based systems like The Raspberry Pi and its own native OS. Like you gotta use Docker which if not familiar with can make it an nightmare for beginners that suddenly are looking at having to learn BOTH at the same time.
do you teach???
Not usually, but I do have a "How To Robot" series, and I hope to cover machine learning in it soon.
@ you should dude. I already love the work you’ve done.
how difficult is ROS.
I actually don't know... 😆 I've never used ROS, but I do plan to experiment with it in the future. Just not on this robot. It's a personal challenge.
@ gotcha I’ve been trying to start up my Unitree go2 dog and have it do a handstand and or boxing pose 😂. Now that will be cool
Add a small LLM
Oh yes! I have to be careful with my memory usage: the Jetson Orin Nano only has 8GB that has to be shared between the vision and LLM, but for now Phi 3b seems to work just fine. :)
ROS is a hole few escape from. Best to avoid it and do your own development as you are doing.
I disagree, ROS is nice when it works… however it rarely works 😅
I second @erniea5843.. ROS is a powerhouse, especially ros2 control. When you get the basics right, you can reuse code in any project out there, just that it has a learning curve that discourages beginners before they even start 😒
Noo Don't use python!!!
😆
If you are doing segmentation and object recognition one way to avoid the steep price tag of the Jetsons is use an SBC that has integrated NPU like the Rockchip RK3588. It can far outperform the GPU on any SBC for these tasks. They use quantized versions of standard models like Inception and MobileNet and the excellent YOLO. As an inference optimized ASIC some can pump out hundreds of frames per second.
I'll keep that in mind... I've never used an SBC with an NPU. I'm pretty happy with how the Jetson performs, but I sure struggle with getting things to function on it properly.... probably user error, but still, it sure slows me down when I'm struggling just to get a camera working even before writing any code!
@@joethieman9400 I have lots of those SBCs. I found the Jetson Orin to be pretty good but Nvidia software is always gigabytes and always a hassle to set up. I havent managed to get more than 5 or 10 FPS out of it running python segmentation. On top of that its a power hungry little beast.
@@joethieman9400 It is not just you, working with/developing for the jetson platform is TRICKY. I started by downloading the jetson ai lab docker images and following those tutorials, then dove into the docker images themselves to see what they were doing in the backend. Using those as a template I was then able to develop my own tools on my desktop machine and package them for use on the jetson. Much better experience than developing on-device. The jetson has so many weird little undocumented dependencies and particularities when it comes to packages that you'll drive yourself mad starting from zero
I found ROS to be kind of a bloated pile myself, and I use Java for my robotic ecosystem. There used to be a ROSJava implementation but support was dropped years ago so I forked and rewrote it to be highly optimized. It now runs in a JAR less than 500k with near realtime performance on small SBC's. Instead of bloated XML/RPC I used native Java serialization. Unlike python, Java can do real multithreading. You get functional programming and pretty universal platform support. ROS is a great publish/subscribe bus that allows you to have fault-tolerant modular point to point comms across the whole robotic platform using generic networking gear and SBC's. Java supports callouts to native libraries for most low level functions like GPIO, video, etc. Maybe not right for you, but it can be done and thats how I do it with some vastly different physical robots that all share the same underlying software infrastructure.
Sounds like a nice setup. Right now I'm liking the way my modules are all independent and communicating through sockets. I can write any module in any language, and even run them all on different computers. It also isolates errors nicely. :)
@@joethieman9400 ROS and my ROSJavaLite also use sockets at their core. The master node initiates the publisher and subscriber nodes that then handshake up persistent socket communications point to point. They can be running on any node the network can get to. The master merely initiates the handshake and setup, then the complex pub/sub is all point to point traffic. Java has callouts to C and C++ so you get those languages if you like, The nodes attempt reconnect if something drops in the point to point comms.
@@joethieman9400 Ros uses sockets too, master node sets up point to point with publishers and subscribers.
How do you handle package support for hardware?
@@vikaloseva6996 Very Badly... 😆 Right now updates usually break my code... so I need a solution to that. I've heard Docker is an answer to this problem... but I don't know anything about it yet. If you have suggestions, let me know! I really need it.. lol