Reinforcement Learning - My Algorithm vs State of the Art

แชร์
ฝัง
  • เผยแพร่เมื่อ 22 พ.ย. 2024

ความคิดเห็น • 279

  • @chris-graham
    @chris-graham 11 วันที่ผ่านมา +405

    I think you would be interested in network pruning. This is something that's typically done periodically during training to thin networks. If you examine the weights in your PPO-optimized network, you'll find that many are very small, while others are larger. If some near-zero weights are set to zero, networks will often become more stable after fine-tuning. You'll find that the connections in the network begin to look sparse and very similar to networks generated via. Evolutionary methods. PPO is just an optimizer and will work with whatever network configuration you want. The evolutionary networks shown in the video are all differentiable, so PPO would be able to optimize. That would be an interesting comparison if you'd want to pursue that!

    • @nodrance
      @nodrance 11 วันที่ผ่านมา +20

      i smell a part 4

    • @w花b
      @w花b 10 วันที่ผ่านมา +7

      ​@@nodrance I smell you smelling something

    • @Firestorm-tq7fy
      @Firestorm-tq7fy 10 วันที่ผ่านมา +4

      They only sparsen if you use regulation methods like L1

    • @chris-graham
      @chris-graham 10 วันที่ผ่านมา +5

      @@nodrance Do you smell it? That smell. The kind of smelly smell. The kind of smelly smell that smells... smelly.

    • @bitblit
      @bitblit 10 วันที่ผ่านมา +1

      @@chris-graham Right you are, Mr. Krabs.

  •  11 วันที่ผ่านมา +362

    triple pendulum next?

    • @dongyulee2095
      @dongyulee2095 8 วันที่ผ่านมา +2

      Impossible...

    • @sumitbiswas164
      @sumitbiswas164 8 วันที่ผ่านมา +1

      How to get the solution for dynamic (n) chain of pendulums? Is it possible now?

    • @alxklgn364
      @alxklgn364 7 วันที่ผ่านมา +5

      I think that I've read a paper explaining why triple pendulum is a total chaos and impossible to solve. But I would also like to see an attempt.

    • @elie_
      @elie_ 7 วันที่ผ่านมา +1

      @@dongyulee2095 "Source: lol"...
      th-cam.com/video/cyN-CRNrb3E/w-d-xo.html
      And all possible instable equilibrium states th-cam.com/video/I5GvwWKkBmg/w-d-xo.html

    • @elie_
      @elie_ 7 วันที่ผ่านมา +1

      @@dongyulee2095 "Source: lol"
      th-cam.com/video/cyN-CRNrb3E/w-d-xo.html (13 years ago)
      th-cam.com/video/meMWfva-Jio/w-d-xo.html
      th-cam.com/video/I5GvwWKkBmg/w-d-xo.html (even more impressive)

  • @Waffle_6
    @Waffle_6 11 วันที่ผ่านมา +87

    getting that sort of aid from NVIDIA is super nice. super cool, my school just got an ai accelerator, " AGX Orin" very cool piece of computing and fantastic of AI training and research. also, as someone who is more hardware orientated, it has a super fascinating architecture(shared cpu and gpu global memory!)

    • @meronamsamho
      @meronamsamho 10 วันที่ผ่านมา +4

      security be damned I want faster training!

    • @conorstewart2214
      @conorstewart2214 7 วันที่ผ่านมา

      They definitely are cool but I would not class the AGX orin as an AI accelerator, not in the same way GPUs are. Or at least not just an AI accelerator. The AGX Orin and the whole Jetson lineup is meant for embedding in things, like robots, cars, etc. It is a full system, CPU, RAM, GPU.
      It is also not very powerful for the cost, at least in terms of raw compute performance. Even a 4060 gets 242 TOPS whereas the AGX Orin only gets 275 TOPS. If you don't need the portability and embeddability of a Jetson system then you are far better just buying GPUs. I can get a 4060 for £250 (yes this is without a CPU and only has 8 GB VRAM) but the AGX orin costs £1992, so just going with desktop PC hardware your money goes much further. For the price of an AGX Orin you could likely build a 4080 or possibly 4090 PC and get much more performance. If RAM is that much of an issue then you should probably look at enterprise or data centre level systems.

    • @jibcot8541
      @jibcot8541 11 ชั่วโมงที่ผ่านมา

      When you are the most valuable company to have ever existed, I guess you can give a bit of money away to teachers and researches, still nice of them I guess.

  • @fluffsquirrel
    @fluffsquirrel 11 วันที่ผ่านมา +36

    Thank you so much for this demonstration and adding the links! I didn't know of Isaac Lab and was wondering how it was possible to control the mechanics. Great video!

  • @imanuelbaca2468
    @imanuelbaca2468 10 วันที่ผ่านมา +5

    The quality and education of these videos is unmatched please keeping making stuff like this!

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา

      Thank you :)

  • @max_me_is
    @max_me_is 11 วันที่ผ่านมา +345

    We got Pezzza's work X Nvidea collab before GTA VI 😭

  • @kubstoff1418
    @kubstoff1418 11 วันที่ผ่านมา +8

    I've been looking for a subject for my engineering degree and this video might be exactly it! Thank you for the inspiration, your videos are always a blast!

  • @briandeanullery
    @briandeanullery 10 วันที่ผ่านมา +1

    This is just brilliant. I verbally gasped at those numbers. I am so grateful to be living in a world with this sort of stuff, it's truly amazing!

  • @PatrickHoodDaniel
    @PatrickHoodDaniel 11 วันที่ผ่านมา +3

    Oh my god, a video from Pezzza!! I'm so excited!!

  • @poketopa1234
    @poketopa1234 11 วันที่ผ่านมา +3

    PPO and gradient-based policy learning in general is amazing. I will still say that your struggle to get an evolutionary algorithm to learn this problem led to some really creative and impressive curriculum learning ideas which also apply to PPO :)

  • @conorstewart2214
    @conorstewart2214 7 วันที่ผ่านมา +1

    This is very impressive and makes me want to look into RL for robotics again.
    I really don't think you can make much comparison about network size though if you only tried one network configuration that you chose randomly. A followup video seeing how small you can make it would be very interesting. It would also be interesting to see you try and take it from simulation to real life.

  • @_nemo
    @_nemo 11 วันที่ผ่านมา +58

    17:06 That's so similar to what the timescales of evolution in nature, and a human learning a skill are like. That's kinda crazy. Really makes it look like the algorithms successfully mimic real counterparts.

    • @TheRealZitroX
      @TheRealZitroX 11 วันที่ผ่านมา

      And still, some Human doesn't learn at all.

    • @raspberryjam
      @raspberryjam 11 วันที่ผ่านมา +4

      @@TheRealZitroX mean

    • @0osk
      @0osk 8 วันที่ผ่านมา

      @@TheRealZitroX *some humans don't learn
      :)

  • @realzakariax
    @realzakariax 9 วันที่ผ่านมา +1

    7:50 I love how the base also returns to the middle of the field, so fascinating!

  • @drhxa
    @drhxa 11 วันที่ผ่านมา +13

    Have you considered adding physical parameters from motor torque and motor weight? This would help you get much more realistic sim and difficulty level. Also, realistic response times (based on inference speed + connection latency). Also, you can either have a motor at the base and one at the middle joint or both at the base.
    You may also consider adding a battery's weight, so you have the voltage required to power those two motors for some period (say 5 min). This will be an awesome challenge and help you connect simulation to reality much more closely, which sounds super exciting. Looking forward to see if you end up working on it!

    • @lorem9587
      @lorem9587 8 วันที่ผ่านมา +2

      I like these suggestions. Where are the two motors, though? I thought there was only one, the one driving the carriage.

    • @drhxa
      @drhxa 7 วันที่ผ่านมา

      @@lorem9587 oops, haha, you're absolutely right!

    • @drhxa
      @drhxa 7 วันที่ผ่านมา

      The hinges have to be free, that's the whole point of the control problem! My bad haha

  • @requestfx5585
    @requestfx5585 9 วันที่ผ่านมา +2

    Thanks for this high quality video and comparison of those algorithms, very nice. Keep it up

  • @nodrance
    @nodrance 11 วันที่ผ่านมา +2

    I'd love it if you spent more time playing with this. Smaller network, triple pendulum, add random forces to the sim to increase stability, maybe make it target alternate configurations (for example first arm up second arm down or vice versa) and make it chooseable, make it not able to exert as much force. Really push it to the limits and see what it can accomplish

  • @FoXMaSteR001
    @FoXMaSteR001 11 วันที่ผ่านมา +3

    Aweome :D Try to apply to same method to penspinning learning, the fact the brain can coordinate all fingers to use the momentum of the pen in complexe figures is amazing, the duration you need to learn the tricks is probably linked to the touch sensation instead of the view of the figures, as a pro can perform tricks without watching his hand. With time the brain can adjust the position of the hand and fingers depending of the rotation of the pen to save the figure or to trigger a new one to smoothly, that's very automatic at some point. Using a different pen with another balance lead to faster adaptation once the person is pro, the only way to learn it is to try and error, which looks like this video. The movement tends to optimized with time, once you manage to make a trick with the pen it seems your brain remember what happened, which can help to do it again, when this happen it's like bicycling, you can spend a week to try with no success and suddently reach very high sucess rate in a few hours once you made it once. That's a very weird feeling.

  • @FIT7Y
    @FIT7Y 11 วันที่ผ่านมา +1

    I would love to see you tackle other kinds of equilibrium positions. Where one of the pendulums is up while the other is down. And maybe even efficiently switching between the different equilibriums.
    Something like Embedded Control Lab's videos about switching between the different equilibriums for a triple pendulum.

  • @optozorax
    @optozorax 11 วันที่ผ่านมา +16

    I'm solving similiar task: I'm trying to learn AI car to drive, with realistic physics. And I was struggling with learning as you do in previous video, I was inspired by your solution and tried another approach: I started from simple physics (no inertia, no wheels, just rotations + offsets), then gradually interpolated between this simple physics and hard physics. And my NN was able to learn how to drive perfectly. But then I tried energy-based model, basically it's an NN that receives current state, desired action and outputs just a single number - energy. You need to find best action that outputs minimum energy. I iterated over 9 possible actions, and that NN was able to learn how to drive in complex physics without any hacks and very fast.
    So, what do I think: first try CMA-ES, as a superior zero-order optimization method. I think that NEAT is a trash, and one day I will test it out. Then you should try energy-based model. Then it will be someway fair comparison. Now it's not fair absolutely, and I slightly disappointed with this video.

    • @vastabyss6496
      @vastabyss6496 10 วันที่ผ่านมา

      what's the difference between energy and the loss? Also, your method sounds a lot like a DQN if I understand you correctly, and vanilla DQNs are much worse than PPO

    • @optozorax
      @optozorax 10 วันที่ผ่านมา

      @@vastabyss6496 energy is minimized during inference (to find best action for an agent), loss is being minimized during traiting. So, to train an energy based model you need to minimize energy in every step of a simulation, while minimizing overall loss. Many minimizations inside a big one.

  • @marcserraortega8772
    @marcserraortega8772 9 วันที่ผ่านมา

    Thenks a lot for the high quality video! I would love to see more videos related to RL in the future. Keep it up!

  • @rcnhsuailsnyfiue2
    @rcnhsuailsnyfiue2 10 วันที่ผ่านมา +20

    Please consider a side quest to balance a double pendulum IRL?! 😱 You could (relatively) easily build a device for this with a single stepper motor, drive belt, and an arduino. Look at X/Y plotters like Axidraw, enthusiasts regularly build these things themselves with off-the-shelf parts. Hook the stepper motor up to your model, and you’ve got a scientific viral video just waiting to happen…

    • @firedeveloper
      @firedeveloper 10 วันที่ผ่านมา +4

      I would love to see it IRL but that's a serious task.
      1. There is a huge gap between model and real hardware.
      2. IRL you can't have x,y,z positions without camera. The most viable way would be with accelerometers and definitely rings with contacts for data transfer.
      Imagine how hard are some projects with a simple PID, this is 100x more difficult.

    • @rcnhsuailsnyfiue2
      @rcnhsuailsnyfiue2 10 วันที่ผ่านมา +3

      @@firedeveloper fair point, maybe not “easy”. I just think for a motivated novice it would certainly be achievable. If it were me I would use a rotational angle sensor on each pivot point, they’re very cheap and can be frictionless too. Mount it all on a sliding steel rail, pulled continuously along the long axis by a computer-controlled stepper motor. Then it’s simply a motion control system running in a feedback loop. Because the stepper motor is quantized, you can know the entire state of the system from just the 3 angle sensors.

    • @conorstewart2214
      @conorstewart2214 7 วันที่ผ่านมา +1

      ​@@firedeveloper it would need some way of sensing position but if they can make the simulated model as accurate as possible to the real one, including the sensor data the model is fed, they it should be possible for it to work in real life.

    • @rsflipflopsn
      @rsflipflopsn 6 วันที่ผ่านมา

      @@conorstewart2214 same thought. maybe balancing a double pendulum (so I mean a pendulum with two moveable axis, could be a triple pendulum? sorry I am not that familiar with the nomenclature of these in the field of physics) is possible if you have two really precise sensors at both axis which respond with their positions accordingly and really fast. the bigger challenge (if you do something like that with ML) could even be the response time of the model plus the call to the actuator (?).
      nevertheless I really like your thought!

    • @rcnhsuailsnyfiue2
      @rcnhsuailsnyfiue2 3 วันที่ผ่านมา

      @@conorstewart2214 there’s no need to sense position, only the angle of the pendulum. The position is inherently measured by simply knowing the history of commands given to the stepper motor. The same technique is used by 3D printers, as long as they start from the “home corner”, their position will be known to the computer by simple addition/subtraction.

  • @sutsuj6437
    @sutsuj6437 11 วันที่ผ่านมา +5

    Do note that Evolutionary algorithms are usually better than pure RL agents for problems with very sparse rewards (Which is not the case here). For these problems, a hybrid approach might work best.

  • @florianvanleeuwen6683
    @florianvanleeuwen6683 6 วันที่ผ่านมา

    Randomly seeing my physics lecture building on youtube, nice video :)

  • @smokeydude3
    @smokeydude3 11 วันที่ผ่านมา +54

    Why not try testing a more compact PPO network?

    • @miran248
      @miran248 11 วันที่ผ่านมา +13

      Silently hoping for a part 4 and a triple pendulum :)

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +38

      I tried but I couldn't manage to find a good solution (they were very unstable)

    • @stephaneduhamel7706
      @stephaneduhamel7706 11 วันที่ผ่านมา +8

      @@PezzzasWork Maybe you could try distilling the working network and see how small you can make it before it breaks?

    • @cagedgandalf3472
      @cagedgandalf3472 11 วันที่ผ่านมา +2

      @@PezzzasWork Try compacting only the actor network (and also lowering the learning rate) and keeping the critic network to default. That is what I did, although I use TD3 with auxiliary networks.

    • @vincentverbergy9816
      @vincentverbergy9816 11 วันที่ผ่านมา

      ​@@cagedgandalf3472 PPO is not an actor critic network? In general with RL size comes at the cost of computing time and risk for over fitting is not necessarily that big meaning that bigger network size isn't really a drawback given enough compute.

  • @waity5856
    @waity5856 9 วันที่ผ่านมา +2

    It's amazing to see it temorarily give up on balancing when it gets too close to the edge of the rail, so it can try again later in a more favorable position

  • @nexttonic6459
    @nexttonic6459 10 วันที่ผ่านมา +6

    Now you have to add flex to the materials, a small gap to the rollers and the beam. Then add a slack in the bearings...

    • @rcnhsuailsnyfiue2
      @rcnhsuailsnyfiue2 10 วันที่ผ่านมา +4

      Don’t forget to account for the acoustic energy of a squeaky pendulum hinge… And a gentle breeze from a robot farting nearby…

    • @nexttonic6459
      @nexttonic6459 10 วันที่ผ่านมา +4

      @@rcnhsuailsnyfiue2 Agreed. Nvidia talks about real physical world, yet farts and acoustic energy probably not accounted for.. though that is a simulation thing I don't think the video maker can effect that.

  • @BananaDude508
    @BananaDude508 11 วันที่ผ่านมา +2

    just did a school based research paper on machine learning and pendulums using your other videos as reference, this video wouldve been perfect if it was 2 months earlier lol
    Either way thanks!

  • @R.B.
    @R.B. 11 วันที่ผ่านมา +2

    The next task is transitioning between states, of which there are four positions, both arms down, both arms up, and two positions with one arm up and the other down. After that you can move to three arms, where there are 8 states. At three arms you have a chaotic system, but this has been solved already with physical systems, so it would be interesting for a simulated system.

    • @jaiveersingh5538
      @jaiveersingh5538 9 วันที่ผ่านมา +1

      Isn't it already a chaotic system with just 2 pendulum arms?

  • @StevenJAckerman
    @StevenJAckerman 11 วันที่ผ่านมา +2

    Very nice work. Thank you for sharing.

  • @harriehausenman8623
    @harriehausenman8623 11 วันที่ผ่านมา +1

    Beautiful and informative video! 🤗 So satisfying animations. thx 🙏

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +1

      Thanks :)

    • @harriehausenman8623
      @harriehausenman8623 11 วันที่ผ่านมา

      @@PezzzasWork Wow! that was quick 😄

    • @harriehausenman8623
      @harriehausenman8623 11 วันที่ผ่านมา

      @@PezzzasWork I only heard of Isaac before, but wasn't aware it's *that* powerful! 😲 I wouldn't mind a follow-up video where you show the things addressed in this comment section. Like how the smaller layer sizes failed (blooper-time!!) and stuff like that.

  • @alejandromartinez-vp4sx
    @alejandromartinez-vp4sx 8 วันที่ผ่านมา

    Beautiful as usual.

  • @Deniil2000
    @Deniil2000 10 วันที่ผ่านมา +1

    15:58 i really like how it knows not to chase the pendulum into the end of the rail, and makes a flip instead

  • @dottedboxguy
    @dottedboxguy 11 วันที่ผ่านมา +43

    well, sure it's only a few minutes of training, but just how much computational power (or just electricity) was used during these few minutes ? i think it's much much more than your simpler approach. it's cool, but it would be interesting to do a test with power usage normalization to do a fair efficiency comparison

    • @sirynka
      @sirynka 11 วันที่ผ่านมา +5

      Still, 8h of cpu time, even in single core mode, would consume around 40w, so 320wh total. A fully utilized 4090 - 450w * 5m = 40wh.
      Units were edited according to @somedudewillson.
      Thanks for the explanation.

    • @dottedboxguy
      @dottedboxguy 11 วันที่ผ่านมา +2

      @@sirynka what tells you it's a 4090 though ? as it stands, it seems more like a cloud GPU compute approach within a large GPU bay, which consume a tremendous amount of power, though only pezzza could confirm that

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +39

      I didn’t specify it in the video as I thought the difference in time was large enough. My algorithm consumes around 120wh for around 5 hours when the 4090 consumes around 150wh for 3 minutes. I agree that it would have been a nice addition in the comparison.

    • @dottedboxguy
      @dottedboxguy 11 วันที่ผ่านมา +2

      @@PezzzasWork thanks for the precision ! this is indeed good to know, and does change things around a little as to which solution is better, especially considering the resulting NN depths

    • @somdudewillson
      @somdudewillson 11 วันที่ผ่านมา +3

      @@sirynka A Watt is a rate of energy transfer - specifically a Joule per second. A 40W CPU does not consume 40 Joules per second per hour, on account of how that doesn't make sense in this context (If the rate of energy usage was changing it _would_ make sense as a unit, however).

  • @EricSundquistKC
    @EricSundquistKC 9 วันที่ผ่านมา

    That is seriously impressive!

  • @Jiorgos3D
    @Jiorgos3D 11 วันที่ผ่านมา +3

    Yay! New Video

  • @thomas_c
    @thomas_c 11 วันที่ผ่านมา +2

    Amazing job ! I'm in love with PPO now :)
    What hardware did you need to train your ai ?

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +3

      Thank you :) I added the PC spec in the description.

  • @thor9000
    @thor9000 9 วันที่ผ่านมา

    Super nice video and explanation! Question, how much did you need to tune the reward, and how essential are the rewards with the low weights?

  • @louisdupont2126
    @louisdupont2126 6 วันที่ผ่านมา +1

    Great video man ! Is it possible to share your code you really motivated me to dive deeper into isaac lab !

  • @michael_pio
    @michael_pio 11 วันที่ผ่านมา +1

    Great informative video

  • @devsquaaa
    @devsquaaa 10 วันที่ผ่านมา

    Love the content. Please keep it up.

  • @Blooper1980
    @Blooper1980 10 วันที่ผ่านมา

    This was so interesting! 🎉🎉❤❤

  • @gpjedy7379
    @gpjedy7379 10 วันที่ผ่านมา

    Sick! Will you do videos on training multi-agent tasks?

  • @rafa_br34
    @rafa_br34 11 วันที่ผ่านมา +21

    Well done. However, I feel like the video was a bit rushed. Primarily because you didn't test other network sizes, which would have made it more fair for the evolutionary algorithm. It also makes me wonder if the network really "learned" how to balance the pendulum or if it just memorized how to do it in the weights.

    • @NaifAlqahtani
      @NaifAlqahtani 11 วันที่ผ่านมา +9

      Agreed. This video contained no real information. Just an ad and a benchmark of an algorithm on dissimilar hardware

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +22

      I didn't specify other architectures because I couln't manage to have a satisfactory solution with smaller networks. Since I am not an expert with PPO I prefered to only mentioned that it is probably possible in the video. Regarding the learning I think the fact that the solution was able to recover from any perturbation means that there is no overfitting here.

  • @lMINERl
    @lMINERl 11 วันที่ผ่านมา +2

    Love your work im a big fan XD

  • @Gabonidaz
    @Gabonidaz 9 วันที่ผ่านมา +4

    1:08 what are this dashboard? How did you builded? I need to try ...

  • @goatknight777
    @goatknight777 11 วันที่ผ่านมา +1

    PPO really is incredible in all ways

  • @tyler2854
    @tyler2854 11 วันที่ผ่านมา

    Amazing work

  • @Xotexgaming
    @Xotexgaming 9 วันที่ผ่านมา

    Beaucoup trop fort !

  • @glizzygoat
    @glizzygoat 11 วันที่ผ่านมา

    the legend is back

  • @nikbivation
    @nikbivation 11 วันที่ผ่านมา

    Love this!

  • @fenokineza6259
    @fenokineza6259 9 วันที่ผ่านมา

    I just watched a 20 min ad, and im happy

  • @IsaiahSugar
    @IsaiahSugar 10 วันที่ผ่านมา

    would love to see you implement ppo yourself! i think that as a viewer i would learn a lot more from that

  • @seanoconnor1984
    @seanoconnor1984 22 ชั่วโมงที่ผ่านมา

    Continuous Gray Code optimization is very good.
    SwitchNet, the one using a fast transform, is a neural network that uses a low number of parameters.

  • @GelloMello-j9z
    @GelloMello-j9z 11 วันที่ผ่านมา

    woah....the graphical interface is so gooood

  • @luke.perkin.inventor
    @luke.perkin.inventor 10 วันที่ผ่านมา

    Great video, can you do evolutionary distillation or pruning of the ~65536 parameter ppo model?

  • @Rekklessss
    @Rekklessss 9 วันที่ผ่านมา +3

    How did you manage to create such a sleek looking dashboard for the model in the beginning of the video? 1:19

  • @Thk10188965
    @Thk10188965 8 วันที่ผ่านมา +1

    I wonder if you can use PPO to get a solution fast, then evolution to slim it down (by adding some cost per node/connection I assume)

  • @AureliusAurelius
    @AureliusAurelius 11 วันที่ผ่านมา

    Awesome! Next, can you build a real-world robot (or partner with a hardware TH-camr) and test the algorithms on it?

  • @narpwa
    @narpwa 10 วันที่ผ่านมา

    WTF IT LOOKS SOOO CLEAN HOW DID YOU DO VISUALS LIKE THAT ???

  • @UonBoat
    @UonBoat 10 วันที่ผ่านมา

    Such a smooth live chart system in the initial part of the video. Does this came from certain library or you wrote it yourself? Thanks.

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา

      Thank you :) It is a tool I wrote myself, I plan on doing a tutorial on the subject.

    • @UonBoat
      @UonBoat 10 วันที่ผ่านมา

      @@PezzzasWork Sounds cool! I know it's a bit off topic, but I’m looking forward to it whenever it’s out.

  • @felixconrad9248
    @felixconrad9248 11 วันที่ผ่านมา +1

    great video as always, i am not excited to see a video by a lot of youtubers but you are surely one of them

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +1

      Thank you :)

  • @Nothingguy562
    @Nothingguy562 10 วันที่ผ่านมา

    Hey
    I am very starstruck by your work
    I would be very grateful if u could tell me about how you learnt all of this.What would you recommend to a total beginner.
    Thanks

  • @r.g.thesecond
    @r.g.thesecond 9 วันที่ผ่านมา +1

    11:20
    I'm a bit surprised. Is it not possible to use constraints or IK in Blender to also describe the joints, and export them as well?

    • @PezzzasWork
      @PezzzasWork  9 วันที่ผ่านมา +1

      It is certainly possible but I am not very familiar with all these tools, for my use it was simpler to rig the model directly into Isaac Sim

  • @marcelob.5300
    @marcelob.5300 11 วันที่ผ่านมา +3

    Would it be possible to include in the description the hardware specs, please?

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +8

      I added them in the description

    • @marcelob.5300
      @marcelob.5300 11 วันที่ผ่านมา +5

      @@PezzzasWork thanks a lot!

  • @sucim
    @sucim 8 วันที่ผ่านมา

    You might be interested in looking into RLtools / the "Learning to Fly in Seconds" paper!

  • @marcelob.5300
    @marcelob.5300 11 วันที่ผ่านมา +1

    You're great!

  • @Aeorthian
    @Aeorthian 11 วันที่ผ่านมา +1

    In this simulation you mention you need both the position and the velocity of each joint. Your model does not appear to have any rotary encoders modeled on it unless you have a point mass added to represent it that we can't see? You would have to retrain this if you actually wanted to use this in the real world as it would require rotary encoders to measure the angular velocity/position, no? Also, does your bottom motor have a rotary encoder built into it or does it also lack a rotary encoder? Still a great job with the proof of concept even if it's not actually usable in real life.

  • @n.lu.x
    @n.lu.x 10 วันที่ผ่านมา

    did you by any chance try the OpenAI-ES algorithm from their 2017 paper? its quite simple yet powerful for (larger) neural networks. + you could also run it on the gpu in parallel

  • @Build_the_Future
    @Build_the_Future 11 วันที่ผ่านมา +1

    Can You do more with Issac Lab I always run into problems when using it.

  • @MarimeGui
    @MarimeGui 11 วันที่ผ่านมา +2

    Did this simulation include limits on acceleration to try to match real motors ?

  • @phrozenwun
    @phrozenwun 11 วันที่ผ่านมา +1

    For the single pendulum, is it possible to move the inverted "upper" node to any horizontal position as fast as the driven node can move?

  • @rishiniranjan1746
    @rishiniranjan1746 5 วันที่ผ่านมา

    this is crazy Man

  • @Bluelightzero
    @Bluelightzero 11 วันที่ผ่านมา +3

    Is it possible to analyse what these neurons are doing?

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา

      Probably, I don't know how though :D

  • @steve_gatsis
    @steve_gatsis 10 วันที่ผ่านมา +1

    Us there a comparison on how "demanding" each method was in terms of computational resources and memory?
    What i mean is; after training, how much does your pc "struggle" to obtain the result it trained upon
    Do you think something like that matters in the end?

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา

      This is a tough question. On the one hand, PPO uses a much larger network than the evolutionary approach, but inference is performed on specialized hardware that is far more efficient for mass computation.

  • @lefm_
    @lefm_ 11 วันที่ผ่านมา

    Yeah i came from home building a small evo AI class in c# using maybe 6 nodes, stumbled upon ml-agents where solutions involves 256, often 512 nodes. It looks like a need for PPO.

  • @left_eyebr0w
    @left_eyebr0w 3 วันที่ผ่านมา

    You talk about things you like and I like that
    So am I 🧐

  • @biobuu4118
    @biobuu4118 10 วันที่ผ่านมา

    Very impressive but what would happen if the task was not to balance the poles but to aim let's say 90° angle at their joint ?

  • @expired___milk
    @expired___milk 11 วันที่ผ่านมา

    Could you use a big network using PPO and then make it smaller using the evolutionary algorithm?

  • @kahwigulum
    @kahwigulum 11 วันที่ผ่านมา

    this is how the borg will learn to modulate their shields against phaser fire

  • @cloudzero2049
    @cloudzero2049 4 วันที่ผ่านมา

    Any possibility of comparing TD3 (Twin Delayed Deep Deterministic Policy Gradient) to PPO for this? I'm curious because I am working with TD3. It's a little more complex than PPO from what I understand, and maybe overkill for this project if that holds true, but I was just curious.

  • @lucao9059
    @lucao9059 9 วันที่ผ่านมา

    how can i like this vid more than once?

  • @tofonofo4606
    @tofonofo4606 11 วันที่ผ่านมา

    So cool.

  • @FlashTheMusik
    @FlashTheMusik 10 วันที่ผ่านมา

    How do you make these awesome dashboards for your visualization?

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา +1

      I am using a tool I wrote myself

  • @EliasMheart
    @EliasMheart 9 วันที่ผ่านมา +1

    I enjoyed your showcase! There was something nagging at me throughout, though...
    I feel like you are comparing apples and oranges here.
    Evo and RL are different paradigms. And comparing them on the same task with different amounts of resources... I guess I'm just not sure what you are trying to say.

    • @PezzzasWork
      @PezzzasWork  9 วันที่ผ่านมา +2

      The point of this video is just to compare my homemade approach vs modern reinforcement algorithm. These are different algorithms but in the end what I want is a balanced double pendulum. This video only shows what to expect from the two options regarding training and results.

    • @EliasMheart
      @EliasMheart 9 วันที่ผ่านมา +2

      @@PezzzasWork Ahh, right! So exploring different approaches, and the comparison is not of the approach in general, but specifically for this task. Gotcha, thanks for the explanation (:

  • @yannsadowski8292
    @yannsadowski8292 11 วันที่ผ่านมา +1

    Hi, you say it take you 5 hour with the Evolutionary tech. But with the RTX 4090 or another graphic card ?

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +1

      The evolutionary algorithm isn't GPU accelerated, it is running on the CPU (mutlithreaded). A big advantage of most of RL algorithms is that they are able to run on GPUs very efficiently.

  • @galacticlava1475
    @galacticlava1475 10 วันที่ผ่านมา +1

    Can you please post your code in the description? We’d love to tinker with it.

    • @PezzzasWork
      @PezzzasWork  10 วันที่ผ่านมา

      I will make my fork of Isaac Lab public soon and add the model featured in the video

    • @galacticlava1475
      @galacticlava1475 10 วันที่ผ่านมา

      @@PezzzasWork Thanks! Your AI content is some of the best on youtube rn. And I really commend people like you who keeps code open source so that we can all learn together.

    • @PezzzasWork
      @PezzzasWork  9 วันที่ผ่านมา

      @@galacticlava1475 thank you !

  • @LinkLaine
    @LinkLaine 10 วันที่ผ่านมา

    if you expand the problem to full 3d where you have cart on a 2d surface and the pendulum that can fall in 3d will that algorithm be so effective as in 2d?

  • @ErickTakada
    @ErickTakada 6 วันที่ผ่านมา

    Got a question: what if you add a slower reaction time? Like a human handling the pendulum?

  • @Nyihm226
    @Nyihm226 11 วันที่ผ่านมา

    Maybe look at realistic bounds for velocity or impulse, it looks like it requires a lot of motor acceleration to achieve its control

  • @RaaynML
    @RaaynML 10 วันที่ผ่านมา

    Seems like you can actually get surprisingly close with an order of magnitude less params if you are willing to train longer

  • @mozartantonio1919
    @mozartantonio1919 8 วันที่ผ่านมา

    awesome

  • @szeredaiakos
    @szeredaiakos 8 วันที่ผ่านมา +2

    So ... for non-linear, unstable system control .. you got a tool which requires more upfront investment and it is significantly more expensive to run to reduce the initial time investment. In certain applications it may be worth it, in your application it most certainly does not.

  • @theshadowx
    @theshadowx 3 วันที่ผ่านมา

    Do you plan on making prey vs predator 3?

  • @the_real_aaxel
    @the_real_aaxel 6 วันที่ผ่านมา

    honey wake up, pezzza made a new video

  • @Markus-r6g
    @Markus-r6g 11 วันที่ผ่านมา +1

    6:57 the "simple task" is the limit of humans because a double makes it impossible for a human to accomplish

  • @afdf96
    @afdf96 11 วันที่ผ่านมา

    What about n-pendulum with n > 2? Or n-body problem?

  • @HettyP
    @HettyP 11 วันที่ผ่านมา +1

    Hey man what tools are you using for your visualizations?

    • @PezzzasWork
      @PezzzasWork  11 วันที่ผ่านมา +3

      I am using a tool I wrote myself

    • @HettyP
      @HettyP 11 วันที่ผ่านมา

      @@PezzzasWork i see thanks!

  • @ofabiolima
    @ofabiolima 7 วันที่ผ่านมา

    Please make a prediction of three body problem

  • @ArnaudMEURET
    @ArnaudMEURET 9 วันที่ผ่านมา

    The cart motor seems extremely (unrealistically) capable. I wonder how the network would react with a more reasonable responsiveness of the cart.

  • @adamp3988
    @adamp3988 8 วันที่ผ่านมา +2

    I'm still saddened by the fact that Nvidia has so thoroughly cornered the market on AI hardware and consequently *software* . It would be better to have some viable competition, or to at least make these things *possible* on AMD cards to prevent vendor lock-in.
    Understandably, AMD is not a leader in the space for a reason --- that being its generally inferior hardware. But I feel like Nvidia is doing damage to the future of the field by creating an ecosystem of things that *only* work on its hardware while AMD tends to focus more-so on open standards that can be used on any card.