I have been waiting for something like this since I fiddled around with simulating robots in Unity a few years back. Make. A. Series. Seriously. I love this.
I really love this format where you walk us through the whole process of you actually struggling and powering through. Your enthusiasm and positive attitude is also very contagious! I could watch your videos all day just for that reason. And PS to anyone reading this: the book is frikn awesome. I've done some courses on ML and I can tell you it's absolutely brilliant, both for newcomers and intermediates.
Thanks! I enjoyed making this style of video and it's definitely one of my personal favorites in a while. Glad to hear you also enjoyed the book, thank you for sharing your thoughts!
I was always fascinated about training a project in a simulation and export its results to world with brutal and breaking physics. This seems a good choice. "I feel future".
This how they do control algorithms for rockets and aircraft! Build a simulation, then go fly. Alot of times there even using a kalman filter against the simulation results and real life!
This is awesome, Thank you for this! I have been working on a project that I have known that eventually, I will need to have a simulation for some of the machine learning, but I had no idea where to start. This is a great start!
Your videos always feel so grounded. Most other videos on such topics make me feel so insignificant. Also, I have a friend, Zeke, from Florida, and you two are soo uncannily similar!!!!
Wow this is an awesome video! Really cool to see your thought process and how you figure out all these new systems. Excited for the next part in the series
This is EXCATLY what i would have needed back in early 2021. I did my Bachelor Thesis in simulating and modeling a robotic manipulator and generating a path using genetic algorithms. I digged a little bit into physics engines (or something like ROS) but ended up using simple 3D Plots which took a shit load of time to calculate and animate... Really looking forward for your series!
I think that ROS would be ideal for communicating between your controller script and the script running the sim. You would just have a topic to which you would listen to in the sim and publish to it from the controller. BTW. Awesome work
@@sentdex I am currently enrolled in an excellent course by theconstructsim. It is the ROS in 5 days and honestly ROS is super easy even for me not so experienced python dev. What you did in your video is basically what ros topics are xD. Also I find the official ROS docs not so easy to use :/ with all the different versions and all xD
Fantastic video - I've previously only had experience animating rigged characters in Maya or Blender, but have really wanted to get into robotics. Definitely excited to get started!
🤫 I’m thinking: Calibrating what the simulator can do/"is allowed to do" based on what the robot dog can do (possibly for real) (extent of movement, acceleration, refresh rate etc.). Simulation runs only seem to make sense within the limits of the robot dog, not the simulator. Love this exploratory format!
A few of the problems you ran into screamed "smart developers forgetting that the end user doesn't have the same context as them, and throwing friendly, descriptive names out of the window"...
Yes, great series Harrsion! Keep demoi'ing different types of robot dogs.. (eventually working your way up to.... SPOT!) (or do a collab with michael reeves lmao)
this is the sort of thing I want to see. Train a robot in a in a simulator for a million virtual years and then bring the boy out into the real world. Chef's kiss.
sweet! You re awesome! I had been fiddling with IsaacGym for a while and was a bit frustrated navigating through, but now I have newfound confidence! Any thoughts on getting interactive input to this Isaac Sim simulation? maybe connecting to Unreal - Unity? Do you have any first feelings on how to proceed!
How about using ROS with Omniverse. As usual great content. I look forward to watching this series. Thank u ✌🏻 PS 2nd try to post a comment. Apparently TH-cam rejected the previous one coz I added the link to the documentation for the ROS brigde.
Not sure where your original went, I checked spam too. Yeah there are some definite features in sim that I've seen that look insanely fancy. Unfortunately I am a total ROS noob. I know what it is and that's about it. I keep controlling my robots serially with pyserial :P If you know of a great intro to ROS that ramps up to something like this, let me know. It's a subject I've been wanting to dive into eventually. Also try replying with the docs maybe again.
Thanks great video. What is the workstation that you are using for Isaac Sim and Gym (graphics card)? Also I guess you're running Ubuntu 20.04? Thanks!
I really appreciate your work. I learned Machine Learning 2 years ago with your videos and others. I always liked how you present your work and that you have no problem to admit when you can't do something 100% yet. You have a clear voice, and I never had any issue to understand what you say. I am from Germany. By the way, Panda it the name of the Robot from FRANKA EMIKA 😉 On this video I sadly had some issues to rebuild it. I really want to accomplish this and build my own solution for the Reinforcement Learning for the robot. What I don’t understand is If I need the simplified Script you created at 7:04? How do I create a scene as .usd like Bittle.usd? How should I structure the file locations for everything I need for manual control? I hope so much you find the time to answer 😊
7:14 was gym, so nope. USD is what omniverse app saves, so I just manually build and adjust in Isaac sim, then save, that makes a USD and is default stored in the Isaac directory, which can then be loaded. Id be happy to answer more and share the files, I can probably slap them into github or something. Feel free to email me harrison@pythonprogramming.net
Great video, however after putting together Bittle, (as mentioned in its manual) small things like putting on the rubber "socks", reversing direction of the battery to change the centre of gravity, all these will significantly impact gait of Bittle, which I highly doubt can be simulated in virtual environment (without very detailed modelling of the say dimension and shape of battery) Bittle's gait is basically simple half sine curve, which "supposingly" should allow it to walk basically but in reality, i find it wobble left and right each time Bittle raises a leg, which again i doubt can be modelled virtually without great level of detail. For quadruped-type of robots, shouldn't we be heading down the ROS route instead? Personally i have not used Omniverse before so i can be totally wrong
I presume since it's Nvidia it's supposed to integrate easily with their machine learning tools. I'm not sure how well something like Gazebo would be used to train an AI, it might be just as easy, I don't know
It seems Omniverse is analigous to Gazebo as opposed to ROS. Gazebo is a simulation environment that can be directly integrated with ROS to setup controls of the robot in the simulation.
Hi sentdex! Im dealing with an similar problem in RL with Isaac-Sim, where I’m trying to teach a mobile robot to park. I have a lot of problems with the simulation program, because I don’t really know what are the limits of the joints of my robot, so that I can’t set my observation and action space correctly. Did you figured out how to set the limits of each joint in the simulation, so that I can use them to set the obs and action space ? I would appreciate if we can chat a little bit more detailed on about dealing with the Isaac sim simulator and it’s dynamic library? Kind rewards!
Oh gosh, you started with the msot difficult simulator. When I used Isaac sim.... I felt your pain. The documentation is horrible. There's other sims that are easier to work with but are definitely worse in terms of limitations (ROS gazebo, pybullet, VREP)
Oh my god! how do you learn this stuff? I think simulations could save a lot of time for robotics developers so they don't have to deal with the electronics and complicated background earlier in the development.
if fullVideoWatched == true: print("This is probably the best TH-cam video ending that I ever saw") print("Yeah I know the true check is useless it's just for reading purpose") return thumb_up = 1
I went to follow along with this before discovering that omniverse only runs if you have an RTX card. Any way around that? In the future, can you do things like this with technologies everyone can access?
Consider a situation where I have var_name with 600 unique values, bins, woe values and event rate values. Now I am having to create a dashboard with all the above features as input such that when I select a particular variable name from a dropmenu and a woe or event rate feature from the other dropmenu, I can visual a bar plot of the woe or event rate values against the bins of that particular selected var_name value. Can someone please help me with that code?
how do you deal with the overwhelming anxiety when you start out with something completely new? I am working on carla project using ros and I feel anxious with how much I don't know yet
Start by using « the code you wish you had » from the highest level. Then break it down and implement it. It will help you figure out what you’ll be needing in order to achieve the goal. In other words, start by moving away from the stress of having to figure out thousands of details. Act as if someone else will do it later. You first focus on what you’re trying to do and let the _how_ emerge. This will also help you avoid coding useless low-level things that look important at first but then prove unneeded for the current focus. HTH.
@@sentdex do you recommend Ubuntu as a daily driver or would you say Windows or Mac would be someone’s best bet. I am a PhD student in Robotics Engineering and considering the switch but don’t know if it is a good idea or not and looking for additional opinions
I’ve always wanted to play around with gait algorithms but felt that unless I have a simulation to test it on first it’s too much money to spend on a test robot first
I have been waiting for something like this since I fiddled around with simulating robots in Unity a few years back. Make. A. Series. Seriously. I love this.
Definitely looking to continue with this project, have already made some good progress for the next part too :)
I've also been thinking of doing this. 3D printing robots, then make them learn how to move with machine learning. Will be awesome!
lmao , i was also trying to simulate robots in unity like 2.5 years back
I really love this format where you walk us through the whole process of you actually struggling and powering through. Your enthusiasm and positive attitude is also very contagious! I could watch your videos all day just for that reason. And PS to anyone reading this: the book is frikn awesome. I've done some courses on ML and I can tell you it's absolutely brilliant, both for newcomers and intermediates.
Thanks! I enjoyed making this style of video and it's definitely one of my personal favorites in a while.
Glad to hear you also enjoyed the book, thank you for sharing your thoughts!
Yeah half of robotics and machinelearning especially nvidia specific stuff is going through tedious documentation.
I was always fascinated about training a project in a simulation and export its results to world with brutal and breaking physics. This seems a good choice. "I feel future".
This how they do control algorithms for rockets and aircraft! Build a simulation, then go fly. Alot of times there even using a kalman filter against the simulation results and real life!
This is awesome, Thank you for this! I have been working on a project that I have known that eventually, I will need to have a simulation for some of the machine learning, but I had no idea where to start. This is a great start!
Awesome, good luck with the project!
I love your sense of humor in the beginning, because that's mostly what simulators are used for, shooting things and blowing them up.
Your videos always feel so grounded. Most other videos on such topics make me feel so insignificant. Also, I have a friend, Zeke, from Florida, and you two are soo uncannily similar!!!!
Awesome! Eager to see what's gonna be a result!
Wow this is an awesome video! Really cool to see your thought process and how you figure out all these new systems. Excited for the next part in the series
Your problem solving impressed me more than the sim.
Heh, tbh it's really just a bunch of random walks til something works out ;)
I have to say my random walks look nothing like your random, mine usually end in tears and dark corners...hahaha
Fantastic! I’ve been looking for a beginners guide to robotics simulation and found this helpful
This is EXCATLY what i would have needed back in early 2021. I did my Bachelor Thesis in simulating and modeling a robotic manipulator and generating a path using genetic algorithms. I digged a little bit into physics engines (or something like ROS) but ended up using simple 3D Plots which took a shit load of time to calculate and animate... Really looking forward for your series!
I think that ROS would be ideal for communicating between your controller script and the script running the sim. You would just have a topic to which you would listen to in the sim and publish to it from the controller.
BTW. Awesome work
If you have any documentation about this that you think might be helpful, please share!
@@sentdex I am currently enrolled in an excellent course by theconstructsim. It is the ROS in 5 days and honestly ROS is super easy even for me not so experienced python dev. What you did in your video is basically what ros topics are xD. Also I find the official ROS docs not so easy to use :/ with all the different versions and all xD
Fantastic video - I've previously only had experience animating rigged characters in Maya or Blender, but have really wanted to get into robotics. Definitely excited to get started!
Awesome! Can’t wait to watch the rest of the series!
🤫 I’m thinking: Calibrating what the simulator can do/"is allowed to do" based on what the robot dog can do (possibly for real) (extent of movement, acceleration, refresh rate etc.). Simulation runs only seem to make sense within the limits of the robot dog, not the simulator.
Love this exploratory format!
I don't understand much, but it's very interesting!
Really like that you describe your whole way of trial and error
wow this is great, Please do make a playlist on this
Cool video. I’d like to see more of this “simulating robots” topic
I did not know that you are building a quad!! Great clip I hope you'll make more content on the topic!
Btw Unity is supporting URDF and ROS now... 😎
You're awesome. Love your videos.
A few of the problems you ran into screamed "smart developers forgetting that the end user doesn't have the same context as them, and throwing friendly, descriptive names out of the window"...
Brilliant work man, please make video on Augmented reality in which Computer vision models have been used. Thank you.
Lets goooo! This is what I've been waiting for!
interesting video and demo, isaac gym is worth exploring , ROS & gazebo are famous for this kind of work
Thank you for a very interesting video
The rhought process is amazing, makes me wonder how can I think like an engineer. Thx.
Yes, great series Harrsion! Keep demoi'ing different types of robot dogs.. (eventually working your way up to.... SPOT!) (or do a collab with michael reeves lmao)
Reloading the sim to an accidental working jump was pretty funny.
top notch content creator
I've been waiting for this... Roooooobot
Yeah, This is awesome. Can you make more on robot simulation and Reinforcement learning pls
this is the sort of thing I want to see. Train a robot in a in a simulator for a million virtual years and then bring the boy out into the real world. Chef's kiss.
Totally new to robotics, but would something like ROS work better for this case? Seems to be a common platform for robotics.
sweet! You re awesome!
I had been fiddling with IsaacGym for a while and was a bit frustrated navigating through, but now I have newfound confidence!
Any thoughts on getting interactive input to this Isaac Sim simulation? maybe connecting to Unreal - Unity? Do you have any first feelings on how to proceed!
Its pretty obvious the sliding is due to joint oscillation which does occur even with servos with a bit of force applied to them.
0:47 Leave doggo alone!
A partnership with James Bruton is absolutely necessary!
Is he still working on his dog and solving problems by putting in more powerful boards instead of learning programming?
That guy is super narcissistic
@@maxhouseman3129 He knows programming? What are you on about?
How about using ROS with Omniverse.
As usual great content. I look forward to watching this series.
Thank u ✌🏻
PS 2nd try to post a comment.
Apparently TH-cam rejected the previous one coz I added the link to the documentation for the ROS brigde.
Not sure where your original went, I checked spam too.
Yeah there are some definite features in sim that I've seen that look insanely fancy. Unfortunately I am a total ROS noob. I know what it is and that's about it. I keep controlling my robots serially with pyserial :P
If you know of a great intro to ROS that ramps up to something like this, let me know. It's a subject I've been wanting to dive into eventually.
Also try replying with the docs maybe again.
I second the ros decision.
@@sentdex there are great ros courses on youtube by ETH Zurich
Thanks great video. What is the workstation that you are using for Isaac Sim and Gym (graphics card)? Also I guess you're running Ubuntu 20.04? Thanks!
يا اخي ماشاء الله
شكراالشرح مفيد ولذيذ في الالقاء
Very cool project 😀
Your video's never fail to entertain :)
this is awesome
fkin love the vid, 10/10 content
Ty
Hello mr. Robot 🤖
Hahaha. The guy who wrote the documentation must be crying watching this video 😁
Nice video
RIP the Robot who fell into Abyss
Some say that it is still falling.
I really appreciate your work. I learned Machine Learning 2 years ago with your videos and others. I always liked how you present your work and that you have no problem to admit when you can't do something 100% yet. You have a clear voice, and I never had any issue to understand what you say. I am from Germany. By the way, Panda it the name of the Robot from FRANKA EMIKA 😉
On this video I sadly had some issues to rebuild it. I really want to accomplish this and build my own solution for the Reinforcement Learning for the robot. What I don’t understand is If I need the simplified Script you created at 7:04? How do I create a scene as .usd like Bittle.usd? How should I structure the file locations for everything I need for manual control?
I hope so much you find the time to answer 😊
7:14 was gym, so nope.
USD is what omniverse app saves, so I just manually build and adjust in Isaac sim, then save, that makes a USD and is default stored in the Isaac directory, which can then be loaded.
Id be happy to answer more and share the files, I can probably slap them into github or something. Feel free to email me harrison@pythonprogramming.net
@@sentdex Your are the Best😃 Thanks for your help. That is a lot more than I expectet in the best case. I will do my best and contact you if needed.😉
Great video, however after putting together Bittle, (as mentioned in its manual) small things like putting on the rubber "socks", reversing direction of the battery to change the centre of gravity, all these will significantly impact gait of Bittle, which I highly doubt can be simulated in virtual environment (without very detailed modelling of the say dimension and shape of battery) Bittle's gait is basically simple half sine curve, which "supposingly" should allow it to walk basically but in reality, i find it wobble left and right each time Bittle raises a leg, which again i doubt can be modelled virtually without great level of detail.
For quadruped-type of robots, shouldn't we be heading down the ROS route instead? Personally i have not used Omniverse before so i can be totally wrong
I presume since it's Nvidia it's supposed to integrate easily with their machine learning tools. I'm not sure how well something like Gazebo would be used to train an AI, it might be just as easy, I don't know
It seems Omniverse is analigous to Gazebo as opposed to ROS. Gazebo is a simulation environment that can be directly integrated with ROS to setup controls of the robot in the simulation.
these things should be taught in schools
Finally!
Thanks -
sentdex is life
You should team up with James Bruton to make the ultimate AI dog!
Will the Neural Networls from scratch in python series continue ? When the next video in that series will come.
Please clarify this matter !
There are too many robot dogs now!!
Need more
Hi sentdex! Im dealing with an similar problem in RL with Isaac-Sim, where I’m trying to teach a mobile robot to park. I have a lot of problems with the simulation program, because I don’t really know what are the limits of the joints of my robot, so that I can’t set my observation and action space correctly. Did you figured out how to set the limits of each joint in the simulation, so that I can use them to set the obs and action space ? I would appreciate if we can chat a little bit more detailed on about dealing with the Isaac sim simulator and it’s dynamic library? Kind rewards!
nvidia software - really powerful stuff that's nearly impossible to figure out
Oh gosh, you started with the msot difficult simulator. When I used Isaac sim.... I felt your pain. The documentation is horrible. There's other sims that are easier to work with but are definitely worse in terms of limitations (ROS gazebo, pybullet, VREP)
Oh my god! how do you learn this stuff? I think simulations could save a lot of time for robotics developers so they don't have to deal with the electronics and complicated background earlier in the development.
can I do that witht he boston dynamics spot URDF?
Brave
if fullVideoWatched == true:
print("This is probably the best TH-cam video ending that I ever saw")
print("Yeah I know the true check is useless it's just for reading purpose")
return thumb_up = 1
Gonna get an error since no cap on that true :p
"Who's Isaac?"
You don't know? Isaac Newton? The one with the groundwork for everyday physics?
I went to follow along with this before discovering that omniverse only runs if you have an RTX card. Any way around that? In the future, can you do things like this with technologies everyone can access?
Are there somebody knows what kind of IDE sentdex uses?
Boston Dynamics...
Let's get to brass tacks here, How much until we can make them shot?
Consider a situation where I have var_name with 600 unique values, bins, woe values and event rate values. Now I am having to create a dashboard with all the above features as input such that when I select a particular variable name from a dropmenu and a woe or event rate feature from the other dropmenu, I can visual a bar plot of the woe or event rate values against the bins of that particular selected var_name value. Can someone please help me with that code?
how do you deal with the overwhelming anxiety when you start out with something completely new? I am working on carla project using ros and I feel anxious with how much I don't know yet
Start by using « the code you wish you had » from the highest level. Then break it down and implement it. It will help you figure out what you’ll be needing in order to achieve the goal. In other words, start by moving away from the stress of having to figure out thousands of details. Act as if someone else will do it later. You first focus on what you’re trying to do and let the _how_ emerge. This will also help you avoid coding useless low-level things that look important at first but then prove unneeded for the current focus. HTH.
@@ArnaudMEURET that's great advice. Thanks!
Which Linux distribution are you using here? It doesn't seem to be Ubuntu but I could be mistaken?
Yep Ubuntu
@@sentdex do you recommend Ubuntu as a daily driver or would you say Windows or Mac would be someone’s best bet.
I am a PhD student in Robotics Engineering and considering the switch but don’t know if it is a good idea or not and looking for additional opinions
what simulator is the best for RL python? pls tell me
I’ve always wanted to play around with gait algorithms but felt that unless I have a simulation to test it on first it’s too much money to spend on a test robot first
which programming language do use to program the robot?
why not use ROS2 rviz or gazebo
Have you tried the Webots simulator?
I know this is a 2yo video, but for the life of me I can't find the sample files anymore. Any ideas?
you have a lot of trust in random code from the internet lol
Where can buy this robot ?
Can we represent in a digital world a piece of machinery made to digitally navigate the real world?
I think this is naming like nicola and tesla thing. But this time it is perhaps refering to Isaac Newton?
I think webots simpler :)
which platform is it?
Who's Isaac? Well it's either Hayes or Newton
Newton would make the most sense I think
Newton lives in Python
What's your pc spec?
Maybe Isaac is Isaac Newton?
Robotics and AI are even more of a jungle compared to web stack
I think it's because there's no set standard, and essentially everyone thinks they can come up with a better way, so we wind up with 9000 ways :D
@@sentdex Lol yeah, I have lost a lot of hair and have possibly developed OCD after installing tensorflow and keras a couple of times