Sebastian Starke
Sebastian Starke
  • 18
  • 412 708
[SIGGRAPH 2024] Categorical Codebook Matching for Embodied Character Controllers
Translating motions from a real user onto a virtual embodied avatar is a key challenge for character animation in the metaverse. In this work, we present a novel generative framework that enables mapping from a set of sparse sensor signals to a full body avatar motion in real-time while faithfully preserving the motion context of the user. In contrast to existing techniques that require training a motion prior and its mapping from control to motion separately, our framework is able to learn the motion manifold as well as how to sample from it at the same time in an end-to-end manner. To achieve that, we introduce a technique called codebook matching which matches the probability distribution between two categorical codebooks for the inputs and outputs for synthesizing the character motions. We demonstrate this technique can successfully handle ambiguity in motion generation and produce high quality character controllers from unstructured motion capture data. Our method is especially useful for interactive applications like virtual reality or video games where high accuracy and responsiveness are needed.
Project Page: github.com/sebastianstarke/AI4Animation
มุมมอง: 6 687

วีดีโอ

[SIGGRAPH 2021] Neural Animation Layering for Synthesizing Martial Arts Movements
มุมมอง 25K3 ปีที่แล้ว
Interactively synthesizing novel combinations and variations of character movements from different motion skills is a key problem in computer animation. In this research, we propose a deep learning framework to produce a large variety of martial arts movements in a controllable manner from raw motion capture data. Our method imitates animation layering using neural networks with the aim to over...
[SIGGRAPH 2020] Sebastian Starke - Thesis Fast Forward
มุมมอง 28K4 ปีที่แล้ว
Thesis Fast Forward presentation at SIGGRAPH 2020 about my Ph.D. research on deep learning for character animation and control at the University of Edinburgh. The work was done in collaboration with Adobe Systems and Electronic Arts. No humans or animals were harmed during motion capture. Project Page: github.com/sebastianstarke/AI4Animation
[SIGGRAPH 2020] Local Motion Phases for Learning Multi-Contact Character Movements
มุมมอง 39K4 ปีที่แล้ว
Controlling characters to perform a large variety of dynamic, fast-paced and quickly changing movements is a key challenge in character animation. In this research, we present a deep learning framework to interactively synthesize such animations in high quality, both from unstructured motion data and without any manual labeling. We introduce the concept of local motion phases, and show our syst...
[SIGGRAPH Asia 2019] Neural State Machine for Character-Scene Interactions
มุมมอง 89K5 ปีที่แล้ว
Animating characters is a difficult task when it comes to interacting with objects and the environment. What if we used computer brains instead? In this research, we present the Neural State Machine, a data-driven deep learning framework that can handle such animations. The system is able to learn character-scene interactions from motion capture data, and produces high-quality animations from s...
[SIGGRAPH 2018] Mode-Adaptive Neural Networks for Quadruped Motion Control
มุมมอง 154K6 ปีที่แล้ว
We present a novel method for real-time quadruped motion synthesis called Mode-Adaptive Neural Networks. Our system is trained in an end-to-end fashion from unstructured motion capture data, without requiring labels for the phase or locomotion gaits. The system can be used for creating natural animations in games and films, and is the first of such systematic approaches whose quality could be o...
UltiDraw Data Visualisation
มุมมอง 2.6K6 ปีที่แล้ว
UltiDraw Data Visualisation
Real-Time Collision Avoidance using Bio IK
มุมมอง 3.9K7 ปีที่แล้ว
Real-Time Collision Avoidance using Bio IK
Real-Time Collision Avoidance on the KuKA LWR iiwa using Bio IK
มุมมอง 2.2K7 ปีที่แล้ว
Real-Time Collision Avoidance on the KuKA LWR iiwa using Bio IK
Low Poly Landscaper Tech Demo
มุมมอง 1.4K7 ปีที่แล้ว
Low Poly Landscaper Tech Demo
Aykie the Squirrel
มุมมอง 9037 ปีที่แล้ว
Did you ever dream of having a super fat squirrel to be the hero in your next game? Look no further, here is Aykie! This cute monster is living hidden in the woods, gathering nuts and always hungry for fresh food! Make sure to give him a lovely home and an appropriate role for a special squirrel in your next 2D platformer! He will make you go nuts about it! Aykie is a 2D animated character avai...
Bio IK for Generic Inverse Kinematics in Unity3D
มุมมอง 30K7 ปีที่แล้ว
The video demonstrates a novel biologically-inspired approach for solving the inverse kinematics problem on articulated generic kinematic geometries while providing real-time capability, robustness and scalability. Solutions are found through memetic evolutionary optimisation, and can be used for controlling characters in runtime. The asset is available at the Unity3D Asset Store. It is very ea...
Full-Body Inverse Kinematics on the NASA Valkyrie Humanoid Robot
มุมมอง 1.8K7 ปีที่แล้ว
This video demonstrates memetic evolutionary optimisation in solving full-body inverse kinematics on the NASA Valkyrie humanoid robot. Solutions are robustly found in real-time, and can solve multiple end effectors and constraints simultaneously without violating joint limits.
Maybe (Acoustic)
มุมมอง 9578 ปีที่แล้ว
Maybe (Acoustic)
Izuna, Tales of a Thief [Game Trailer]
มุมมอง 4958 ปีที่แล้ว
Izuna, Tales of a Thief [Game Trailer]
Collision-Free Navigation on Lego Mindstorms NXT
มุมมอง 1958 ปีที่แล้ว
Collision-Free Navigation on Lego Mindstorms NXT
Unity3D Inverse Kinematics Solver (Penguins)
มุมมอง 1.3K8 ปีที่แล้ว
Unity3D Inverse Kinematics Solver (Penguins)

ความคิดเห็น

  • @Something-lcl
    @Something-lcl 4 หลายเดือนก่อน

    Sorry to bother you. I replaced the 27-bone model you provided with the 22-bone Smplh model and Mixamo model. After setting up the Actor script, GenericIK script, Motion Controller, and Blueman script, the upper body became immobile, while the lower body bones could move, but the skin did not align with the bones. Then, when I increased the bones back to 27, the whole skeleton moved correctly, but the skin still didn't align with the bones. Could you please explain what might be causing this?

  • @bahshas
    @bahshas 5 หลายเดือนก่อน

    how come we don't see these implemented in videogames?

  • @bause6182
    @bause6182 5 หลายเดือนก่อน

    Good intersting paper + Code implementation = pur hapiness.

  • @leoako7775
    @leoako7775 5 หลายเดือนก่อน

    this is what's lacking in vrchat ! ugly movement really kills the experience so implementing this tech in game would be such a welcomed upgrade !

  • @TerraUnityCo
    @TerraUnityCo 5 หลายเดือนก่อน

    This is amazing Sebastian, great job as always 👏

  • @honglinchen4681
    @honglinchen4681 5 หลายเดือนก่อน

    Does anyone else also like the dance at the end of the video? lol

  • @BaseRealityVR
    @BaseRealityVR 5 หลายเดือนก่อน

    This is What I want for my own UE VR system that I'm developing, instead of having to A mix OF animation & procedural foot placement which can look robotic. The bending of the Knees in the correct directions here is very impressive because dynamic crouching can be a challenge

  • @r.m8146
    @r.m8146 5 หลายเดือนก่อน

    You were featured in 3 minutes papers. Congrats!

  • @r.m8146
    @r.m8146 5 หลายเดือนก่อน

    Absolutely amazing.

  • @ForexLearnerAI
    @ForexLearnerAI 5 หลายเดือนก่อน

    I'm literally speechless. No joke. I don't know what to say. This is amazing. I'll look into the repo for more info but I have a use case in mind where this would be so valuable when trained on a custom dataset. Thanks Starke and everyone involved for brining us one step closer to a thriving future.

  • @mbit5818
    @mbit5818 5 หลายเดือนก่อน

    if you implement this technology as a plugin for ue5 vr, I will definitely buy it!

  • @importon
    @importon 5 หลายเดือนก่อน

    Very nice! How can we try your "mirror demo" in our headsets? Is there a demo available?

  • @mattanimation
    @mattanimation 5 หลายเดือนก่อน

    been such a joy to watch how this work has evovled and be applied to so many areas

  • @ThaLiveKing
    @ThaLiveKing 5 หลายเดือนก่อน

    Is this just for VR?

    • @bause6182
      @bause6182 5 หลายเดือนก่อน

      You can customize the code in order to make a mocap for animations maybe

  • @ZadakLeader
    @ZadakLeader 5 หลายเดือนก่อน

    Sweet!

  • @skavenqblight
    @skavenqblight 5 หลายเดือนก่อน

    Impressive.

  • @tomhalpin8
    @tomhalpin8 5 หลายเดือนก่อน

    Absolutely fantastic as always!

  • @fanshengmeng6102
    @fanshengmeng6102 5 หลายเดือนก่อน

    Looking forward to the code/demo/dataset! Great work!

    • @paul-starke
      @paul-starke 5 หลายเดือนก่อน

      We are working on the polishing and will upload them soon, thanks!

  • @loveadeola
    @loveadeola 8 หลายเดือนก่อน

    Can these be integrated into a game engine? If yes, which ones?

  • @zk-mc
    @zk-mc 8 หลายเดือนก่อน

    Hi! Great work. Is there any documentation, papers on this example, or information we can read for the method of this particular implementation? Cheers.

  • @vladjurat
    @vladjurat 9 หลายเดือนก่อน

    This is so awesome!

  • @zeblanoue2642
    @zeblanoue2642 10 หลายเดือนก่อน

    This goes so hard

  • @zeon3d755
    @zeon3d755 11 หลายเดือนก่อน

    Could it be done using unreal engine 5?

  • @reaperdev9857
    @reaperdev9857 ปีที่แล้ว

    No plans for a comercial product? As an indie dev the idea of cutting down the budget and producing higher quality content faster is VERY attractive.

  • @Hello_352-v1h
    @Hello_352-v1h ปีที่แล้ว

    does it still have the features from 2019 and 2021 because i need all 3 of them?

  • @Hello_352-v1h
    @Hello_352-v1h ปีที่แล้ว

    Is there a way to controle the charecter like we used to use a xbox controller before

  • @oldleaf3755
    @oldleaf3755 ปีที่แล้ว

    amazing

  • @thomashalpin2251
    @thomashalpin2251 ปีที่แล้ว

    Many 3D characters come in a range of shapes and sizes, with different lengths for limbs. I wonder if there is a path that could train one set of motions then correctly apply it to all the different characters with no loss in fidelity. Hopefully a future paper :)

  • @Tenenx69
    @Tenenx69 ปีที่แล้ว

    What the dog doin

  • @PapaMama-dm9nl
    @PapaMama-dm9nl ปีที่แล้ว

    Wow

  • @The_Pugilist
    @The_Pugilist ปีที่แล้ว

    Amazing stuff!

  • @JayYeasmin
    @JayYeasmin ปีที่แล้ว

    has this been utilized in a game yet?

  • @jacekfr3252
    @jacekfr3252 ปีที่แล้ว

    this is what modern art really is. Applause!

  • @TheSeiris
    @TheSeiris ปีที่แล้ว

    Will test it my own and will be sharing the results if I find anything interesting. Thanks!!

  • @TheSeiris
    @TheSeiris ปีที่แล้ว

    Great! I think your work’s direction is valuable should deserve more attention. Seems like periodical modeling or idea of using fft worked really good. I think it makes sense, since bone positions naturally repositions to origin, it’s very reasonable to model it periodically. Very good xD.

  • @dihuang9849
    @dihuang9849 ปีที่แล้ว

    Pretty cool!

  • @nictanghe98
    @nictanghe98 ปีที่แล้ว

    I prefer transflower dancing skills.

  • @manollobango
    @manollobango ปีที่แล้ว

    This is so awesome! 😍 Is there any chance that indie developers like me will be able/allowed to use this technique and animation data set for commercial to an affordable price some day?

  • @EpicPianoArrangements
    @EpicPianoArrangements ปีที่แล้ว

    Is it just me or is the only stiff part about these AI animations... is the ankles? The ankles just seem a little stiff in their movement as if they only bend to half the flexibility they can really achieve. Other than that however, this technology is brilliant.

  • @marcostorres4416
    @marcostorres4416 ปีที่แล้ว

    Excellent! How can I recreate this with a feline quadruped in Unreal Engine 5?

    • @LuciousKage
      @LuciousKage ปีที่แล้ว

      i would love to do that too, did you found out how to do that ?

  • @sinanrobillard2819
    @sinanrobillard2819 ปีที่แล้ว

    Amazing paper! Great progress over the years! Do you think Phase embeddings could be applied to another dataset format such as audio inputs? Also, the ball has physics but what about the character? The question is: Would this simulated motion be applicable to robotic humanoids?

  • @Super-id7bq
    @Super-id7bq ปีที่แล้ว

    This is all incredibly impressive but I still can't stop staring at the red trunk dude's junk. He's got a whole gonzo nose situation. You can see it from behind!

  • @gricchastain6506
    @gricchastain6506 ปีที่แล้ว

    Awesome. Crazy I just hear about this.

  • @mattanimation
    @mattanimation 2 ปีที่แล้ว

    been fun watching the progress since 2017, awesome work!

  • @NycroLP
    @NycroLP 2 ปีที่แล้ว

    Amazing stuff. I wonder if this can also be applied to more specific animation like a small man doing a heavy axe swing.

  • @RyMann88
    @RyMann88 2 ปีที่แล้ว

    I know this is old. But hopefully this could become a plug with animation data (separately), to help indie studios achieve the same effects.

  • @clinkclink7814
    @clinkclink7814 2 ปีที่แล้ว

    Hello Mr Starke, I am really fascinated with applying AI and game engines together. I am far from your skill level and knowledge, but I was wondering If you could guide me on what I need to do to get into this field. I would assume I would have to take courses in Numerical Analysis, Computer Vision, and Computer Graphics. But I feel like these might not be enough. Would you be able to recommend other courses that I should take to get even an "inch" closer to your abilities/knowledge?

    • @Detril2000
      @Detril2000 ปีที่แล้ว

      Machine Learning knowledge would also be needed, and in turn you would need a decent understanding of Linear Algebra and some basic calculus. Ideally you would enroll in a complete Computer Science program with some focus on AI! But of course this is state-of-the-art stuff, post-doc level, so you'd be expected to first take on some Masters and Doctorate level research.

  • @eduard_soul
    @eduard_soul 2 ปีที่แล้ว

    Incredible !

  • @jokwon7343
    @jokwon7343 2 ปีที่แล้ว

    and just like that....thousands of animators screamed out interror

  • @importon
    @importon 2 ปีที่แล้ว

    UE5 where you at??!!!!