Immersive Home's implementation is really nice, he's done his own version of room discovery. Quest 3 can automate a lot of that now which we'll be covering in the near future :)
So glad we are finally moving to a world of using blend modes correctly. Esp on pcvr over link. Been pestering every HW manufacturer about it for 2+years. No reason the streaming connection needs to be saturated with camera image data as in the old pass through extension.
Do note that we don’t have any control here, we’re just emulating the blend mode approach on devices that don’t support it. Our main goal was to present this way of working to the developer to ensure cross platform conformity
This is on my list of todo's as it has become much easier in Godot 4.3 with the new composition layer functionality. In the mean time you can watch the following video that details how this worked in Godot 3. While a few things have been renamed, most of the steps are the same in Godot 4.x. th-cam.com/video/Kt1Wk8NWyF8/w-d-xo.html
No, only mentioning Hololens because it is a device that supports a certain type of AR. While we do have a DirectX backend now, Hololens requires a UWP build. As XBox no longer requires this, there is no incentive to make Godot 4 work on UWP.
I don't have any knowledge on Rokid but if they are planning OpenXR support for their glasses, Godot should be able to run on those. Godot does run on Magic Leap 2 glasses, is rumoured to run on XReal (though details are scarse as XReal doesn't have official OpenXR support yet to my knowledge) and supposedly should run on Meta Orion glasses as this supposedly run Meta HorizonOS (but I can't confirm this). Godot also runs on TiltFive glasses but using T5s own SDK through a plugin.
Omg that really cool thanks for your work and dedication, I’m working on an AR app and i’m still wondering if I should use UE5 or Godot I really want to use godot but I’m afraid I lack features compared to UE for advanced features.. Is there still some big features missing on godot ? Also i want to dev the AR app on the quest 2 and quest 3 at the same time I don’t know if it’s possible with godot
There is definitely an amount of things you'll need to do yourself, Godot for instance supports all the hand tracking functionality, but you'll need to build the interactions on top of that while in Unity you get that mostly for free (UE I don't know). Also a lot of features have only just been implemented like Metas scene discovery and anchor logic so there still is little documentation available, but there is a vibrant community willing to help. You can use the same project to deploy to Quest 2 and Quest 3. I often test between my Quest Pro and Quest 3.
Great Video, You are talking about markers, are they available in Godot 4.3? I really want to learn more about markers, but I can not find anything about it. (And markers as i see it is that the VR can track a marker or pattern and then show some 3D models connected to this marker)
This is on my todo list, I have an HTC Elite now to test with. There is already code for this written but it needs to be moved into the vendors plugin. If you have time before me then we'd very much welcome your contribution :)
Thanks Bastiaan! Btw, what software are you using for casting Quest's screen? I noticed it also outputs the passthrough cameras. I could so far only achieved that using scrcpy.
For content on WebXR I highly recommend checking out @snopekgames I'm not planning on adding WebXR content to my channel any time soon, my focus mainly lies with OpenXR.
Once you have everything setup, how do you debug AR through a Quest 3? I tried remote debugging but it plays the apk on the headset without any debugging tools, and when I use Link passthrough doesn't work
With remote debug you have to first go into the debug menu and enable the remote debug option to actually get debug information. It's a bit confusing for sure. You can also get passthrough working with Link for easier testing however in Quest Link you need to go to the settings, then the beta tab, and enable all the experimental features. I think there is a tickbox specific for passthrough there.
At this point I would say the Quest 3 as it is much newer then the PICO 4 and has the newer XR2 gen2 chipset. But the new PICO 4 Ultra spec wise is close to the Quest 3. I however haven't had a chance to try the PICO 4 Ultra so I'm not in a position to give an educated opinion on how it compares in actual use.
@@ADITYA3GAME no, currently mobile AR (e.g. phones) don’t use OpenXR. They have their own APIs. I have hopes that we’ll see an OpenXR phone runtime for Android someday but I have no idea if this is on Googles radar. If you want to do phone AR have a look at the new ARCore plugin being worked on for Godot. It’s basic feature set is now working but does require the latest Godot 4.4 dev build.
Android phones currently do not support OpenXR so you can't go down this route. We've been trying to get ARCore up and running for awhile and are pretty close to finally supporting this. Once that is available this can be used for passthrough on Android phones.
@@BastiaanOlij Once ARCore support is available, I would love to see a tutorial video! Thank you so much in advance, and looking forward to passthrough on Android! 🙌
@@hamsterfoxgamedev9737 any headset that supports OpenXR and OpenGL or Vulkan, and offers passthrough should work. I don’t know if this applies to the cosmos. As a PCVR headset that works with SteamVR it should work with Godot, but if it has passthrough available I don’t know.
How do I create a game where I can project the 3D elements on a table as if it were a board game and use my hands making gestures to move the pieces but that I can run on Android phones so I can play local multiplayer with the my friends? sorry for my english... hugs from Brazil Bastiaan thank you.
Note: I asked specifically about Android cell phones since here in Brazil it is much easier for people to have access to games of this type than to buy these expensive VR devices that are rarely used here.. thahk you
Sadly ARCore support, which would be needed for this, has proven difficult though we are making some progress on this front. I don't think ARCore supports hand tracking however, that really requires a proper headset.
Hey Bastian, i just started out using Godot 4.2 with pico 4 as VR headset. i've been following your videos but there's this problem i'm facing when it come to input. It seems Pico 4 controller got map only to Simple Controller in OpenXR Action Map, even after i deleted all other profiles leaving only Pico 4 Controller profile, instead it wont track anything. Have you face something similiar? It kinda bugged me since i cant find solution
No, it's working right for me, though admittedly it's been awhile since I tested with my Pico 4. Only thing that springs to mind that I still need to fix is that if you don't press save on the action map, it won't actually apply your changes.
Thx for the reply, I think saving on the action map is working just fine on me because if I remove simple controller profile it wont even track the controller. The only way I can use pico 4 on Godot is rolling back to 4.1 before pico controller profile seperated to pico 4 and pico 3 Neo. I've tried on 4.3, its still have the same problem like 4.2
Does anyone know if hand/finger tracking works on the Pico 4 (I'm trying using Godot 4.3 beta 1, Godot XR tools for Godot 4 (4.3.1) (for the hands and there skeleton), Godot OpenXR Vendors plugin v3.0.0 beta 1) I have yet to get the pico to not complain about the application not supporting hand tracking on launch (when not having a controller active) I have put the XRHandModifier3D into the skeleton of an inherited copy of the hands I have tried to put the hands in different parents, at least inside an XRController they can be moved around using the hand if they use the hand trackers as tracker. Have yet to get the fingers to move (also the orientation of the hands are wrong, but a simple rotation should fix that)
Note that OpenXR has a rather weird bone layout, so if you've created a hand in Blender, bones are probably oriented wrongly. This is why we now convert the bones to the Godot humanoid skeleton.
@@BastiaanOlij I used the godot-xr-tools/hands/model/hand_l.gltf and hand_r.gltf are the bones in those in the correct orientation? Also I mistakenly created a bug report on godot_openxr_vendors and dsnopek kindly informed me that the bug probably belongs to godot as it also handles hand tracking. Would you preferred that I copy the bug report over to godot or just leave it as a comment here (and as a closed bug in godot_openxr_vendors)
@@BastiaanOlij Would have sworn that I pushed the reply button yesterday, well apparently not. Anyhow I created a bug report on the wrong project (godot_openxr_vendors), thanks goes to dsnopek for informing me. If you prefer I can copy it to the godot project instead? With regards to the bones I used godot-xr-tools/hands/hand_l.gltf and hand_r.gltf for the skeleton and mesh.
I'm not sure what makes you say this. V(irtual) R(eality) encompasses creating a virtual world in which you are fully immersed. Everything you see is rendered which is not what we're dealing with here. A(ugmented) R(eality) is adding virtual content to the real world, you are augmenting what you see. Whether you are seeing the real world "naturally" with a glasses type device such as HoloLens (not supported by Godot), Magic Leap, XReal, TiltFive, Meta Orion and similar devices, with the virtual content superimposed through some sort of screen in front of your eyes; Or whether you are seeing the real world through "passthrough", e.g. one or more cameras recording the real world, displaying this as a background and rendering virtual content on top, such as is done with phone based AR (ARKit/ARCore/Etc), Meta Quest, Lynx R1, Pico, HTC, the list goes on and on; It has the same end result, we are augmenting the real world with virtual content. That we are using a VR style headset to "simulate" AR, is semantics. It is why often we talk about M(ixed) R(eality) to indicate the fusion of AR and VR, but I'm personally not a fan of that because it creates an artificial boundary that gets in the way of us creating augmented reality applications that do not care what device you are using. In that way I don't understand why people have no problems calling phone based AR, AR, even though this is technically passthrough, no different to what we're doing on "MR" headsets such as the Quest. This is off course only one piece of the puzzle, how we technically solve the issue of displaying virtual content within the real world. Another important piece of this puzzle is scene discovery, where sensors map the real world so we can accurately place virtual content in a way that makes sense. Put a hologram on a table, hang a virtual picture on the wall. Again scene discovery does not care what type of device you're using, its the functionality that counts. Here Godots OpenXR implementation currently only supports scene discovery for Meta HorizonOS but this means you can create AR applications targeting the Quest line of devices, and likely deploy your AR application as is on Meta Orion glasses as it runs the same tech stack. I can't verify this yet as I do not have access to Meta Orion glasses but in theory, it should just work (tm). As other devices expose scene discovery APIs in OpenXR, this functionality will come to those devices as well. That is the goal we set out creating this solution in Godot, to remove the artificial difference between "optical AR" and "passthrough AR" for the developer. The same code allows you to deploy on any device that allows you to add virtual content into the real world, while you as a developer no longer need to worry about the technical difference. With "optical AR" still emerging, it allows you to start creating meaningful AR applications while using devices such as a Quest that will work when affordable "optical AR" devices hit the market.
Seeing this legitimately has me so excited for what's possible. The smart home demo in the video is amazing!
Definitely subscribe to Nitwels channel, he's making loads of progress and vlogging about it.
@@BastiaanOlij Do you have a link?
@@fra9w3ll I also had some difficulties finding it. It's the Immersive home youtube channel
Great video showcasing what Godot can do with AR. I used this to make my "90's Room" experience take place in my actual room, thanks!
AMAZING WORK BY NITWEL! I can't wait to show this off to people!
Thank you Bastiaan! Looking forward to this and other anticipated features in 4.3 !
Amazing work. You guys never cease to amaze.
Nice! Thank you for all of your work!
This looks super cool. Thanks for pushing it along so far!
Really impressed how this has enabled immersivehome, fantastic!
Immersive Home's implementation is really nice, he's done his own version of room discovery. Quest 3 can automate a lot of that now which we'll be covering in the near future :)
Great work as always 🎉
Thank you so much, we love you!
So glad we are finally moving to a world of using blend modes correctly. Esp on pcvr over link. Been pestering every HW manufacturer about it for 2+years.
No reason the streaming connection needs to be saturated with camera image data as in the old pass through extension.
Do note that we don’t have any control here, we’re just emulating the blend mode approach on devices that don’t support it. Our main goal was to present this way of working to the developer to ensure cross platform conformity
Thanks for the video! and all the work on Godot XR and everything!
This really inspiring❤❤❤
That's beautiful
Thanks for your work! 👍👍👍
maravilloso sublime!. gracias por compartir. ☺
Thank you for your great work! I would love to see a tutorial for the UI as seen in the 01:00 time mark. Thank you in advance!
This is on my list of todo's as it has become much easier in Godot 4.3 with the new composition layer functionality. In the mean time you can watch the following video that details how this worked in Godot 3. While a few things have been renamed, most of the steps are the same in Godot 4.x.
th-cam.com/video/Kt1Wk8NWyF8/w-d-xo.html
Thank you Bastiaan.. thank you so much
Thank you very much. This is amazing
You mention hololens at 2:56 does this mean we be able to create things for hololens 2?
No, only mentioning Hololens because it is a device that supports a certain type of AR.
While we do have a DirectX backend now, Hololens requires a UWP build. As XBox no longer requires this, there is no incentive to make Godot 4 work on UWP.
I would like to know if Godot's OpenXR supports glasses like Rokid.
I don't have any knowledge on Rokid but if they are planning OpenXR support for their glasses, Godot should be able to run on those.
Godot does run on Magic Leap 2 glasses, is rumoured to run on XReal (though details are scarse as XReal doesn't have official OpenXR support yet to my knowledge) and supposedly should run on Meta Orion glasses as this supposedly run Meta HorizonOS (but I can't confirm this).
Godot also runs on TiltFive glasses but using T5s own SDK through a plugin.
Omg that really cool thanks for your work and dedication, I’m working on an AR app and i’m still wondering if I should use UE5 or Godot I really want to use godot but I’m afraid I lack features compared to UE for advanced features.. Is there still some big features missing on godot ? Also i want to dev the AR app on the quest 2 and quest 3 at the same time I don’t know if it’s possible with godot
There is definitely an amount of things you'll need to do yourself, Godot for instance supports all the hand tracking functionality, but you'll need to build the interactions on top of that while in Unity you get that mostly for free (UE I don't know). Also a lot of features have only just been implemented like Metas scene discovery and anchor logic so there still is little documentation available, but there is a vibrant community willing to help.
You can use the same project to deploy to Quest 2 and Quest 3. I often test between my Quest Pro and Quest 3.
Great Video,
You are talking about markers, are they available in Godot 4.3?
I really want to learn more about markers, but I can not find anything about it.
(And markers as i see it is that the VR can track a marker or pattern and then show some 3D models connected to this marker)
Not yet but it’s something i want to look into soon. Right now only magic leap, htc and varjo support markers if i’m not mistaken
@@BastiaanOlij OK, Not yet you say, but are you going to fix something for this in Godot or what do you mean when you say "I want to look into soon"
@@olanders_se the support will likely be added in the vendors plugin, won’t know for sure until i get to it and learn how it all works:)
@@BastiaanOlij OK And what vendor is that?
@@olanders_se the one i'll be trying first is HTC, using the HTC Elite
Got yourself a Q3, I see? Nice. (Still saving up for mine. 🤣)
Yes, Meta was cool enough to supply us with a couple of units, absolutely love it!
Sweet. And well-deserved.
This looks great! Could you provide steps to add platforms (VIVE XR ELITE?)
Happy to work thru some native code to get it working.
This is on my todo list, I have an HTC Elite now to test with. There is already code for this written but it needs to be moved into the vendors plugin. If you have time before me then we'd very much welcome your contribution :)
Thanks Bastiaan! Btw, what software are you using for casting Quest's screen? I noticed it also outputs the passthrough cameras. I could so far only achieved that using scrcpy.
It's the new casting feature in Metas Developer Hub, it works really well.
Thank you, could you also create this for WebXR?
For content on WebXR I highly recommend checking out @snopekgames
I'm not planning on adding WebXR content to my channel any time soon, my focus mainly lies with OpenXR.
Vendors extension doesn't show up for me, Any idea why that might be the case?
The latest version of the vendor extension plugin automatically works. If you can see the options in the export setup it has been installed properly.
Once you have everything setup, how do you debug AR through a Quest 3? I tried remote debugging but it plays the apk on the headset without any debugging tools, and when I use Link passthrough doesn't work
With remote debug you have to first go into the debug menu and enable the remote debug option to actually get debug information. It's a bit confusing for sure.
You can also get passthrough working with Link for easier testing however in Quest Link you need to go to the settings, then the beta tab, and enable all the experimental features. I think there is a tickbox specific for passthrough there.
What would you recommend ? PICO 4 or Quest 3 ?
At this point I would say the Quest 3 as it is much newer then the PICO 4 and has the newer XR2 gen2 chipset.
But the new PICO 4 Ultra spec wise is close to the Quest 3. I however haven't had a chance to try the PICO 4 Ultra so I'm not in a position to give an educated opinion on how it compares in actual use.
can this be used to make mobile AR apps as well? Has somebody already done that?
@@ADITYA3GAME no, currently mobile AR (e.g. phones) don’t use OpenXR. They have their own APIs. I have hopes that we’ll see an OpenXR phone runtime for Android someday but I have no idea if this is on Googles radar.
If you want to do phone AR have a look at the new ARCore plugin being worked on for Godot. It’s basic feature set is now working but does require the latest Godot 4.4 dev build.
How to implement the Passthrough function in Android phones?
Android phones currently do not support OpenXR so you can't go down this route. We've been trying to get ARCore up and running for awhile and are pretty close to finally supporting this. Once that is available this can be used for passthrough on Android phones.
@@BastiaanOlij Once ARCore support is available, I would love to see a tutorial video! Thank you so much in advance, and looking forward to passthrough on Android! 🙌
can i use another vr headset like htc vive cosmos?
@@hamsterfoxgamedev9737 any headset that supports OpenXR and OpenGL or Vulkan, and offers passthrough should work. I don’t know if this applies to the cosmos. As a PCVR headset that works with SteamVR it should work with Godot, but if it has passthrough available I don’t know.
@@BastiaanOlij okey, maybe i will try it first, because the vr headset provided by my lab is that
How do I create a game where I can project the 3D elements on a table as if it were a board game and use my hands making gestures to move the pieces but that I can run on Android phones so I can play local multiplayer with the my friends? sorry for my english... hugs from Brazil Bastiaan thank you.
Note: I asked specifically about Android cell phones since here in Brazil it is much easier for people to have access to games of this type than to buy these expensive VR devices that are rarely used here.. thahk you
Sadly ARCore support, which would be needed for this, has proven difficult though we are making some progress on this front. I don't think ARCore supports hand tracking however, that really requires a proper headset.
@@BastiaanOlij thank you
Hey Bastian, i just started out using Godot 4.2 with pico 4 as VR headset. i've been following your videos but there's this problem i'm facing when it come to input. It seems Pico 4 controller got map only to Simple Controller in OpenXR Action Map, even after i deleted all other profiles leaving only Pico 4 Controller profile, instead it wont track anything. Have you face something similiar? It kinda bugged me since i cant find solution
No, it's working right for me, though admittedly it's been awhile since I tested with my Pico 4. Only thing that springs to mind that I still need to fix is that if you don't press save on the action map, it won't actually apply your changes.
Thx for the reply, I think saving on the action map is working just fine on me because if I remove simple controller profile it wont even track the controller. The only way I can use pico 4 on Godot is rolling back to 4.1 before pico controller profile seperated to pico 4 and pico 3 Neo. I've tried on 4.3, its still have the same problem like 4.2
Very strange indeed. If you turn verbose output on, does it give you any errors in the logs about adding the interaction profiles?
FINALLY SOME ONE ELSE SEES IT
Does anyone know if hand/finger tracking works on the Pico 4 (I'm trying using Godot 4.3 beta 1, Godot XR tools for Godot 4 (4.3.1) (for the hands and there skeleton), Godot OpenXR Vendors plugin v3.0.0 beta 1)
I have yet to get the pico to not complain about the application not supporting hand tracking on launch (when not having a controller active)
I have put the XRHandModifier3D into the skeleton of an inherited copy of the hands
I have tried to put the hands in different parents, at least inside an XRController they can be moved around using the hand if they use the hand trackers as tracker.
Have yet to get the fingers to move (also the orientation of the hands are wrong, but a simple rotation should fix that)
Got the deprecated OpenXRHand to modify the hand bones (did "break" all my fingers but still :) (opening an issue for it on the issue tracker)
I need to find some time testing this all on Pico, it's likely missing a number of feature/permission flags.
Note that OpenXR has a rather weird bone layout, so if you've created a hand in Blender, bones are probably oriented wrongly. This is why we now convert the bones to the Godot humanoid skeleton.
@@BastiaanOlij I used the godot-xr-tools/hands/model/hand_l.gltf and hand_r.gltf are the bones in those in the correct orientation?
Also I mistakenly created a bug report on godot_openxr_vendors and dsnopek kindly informed me that the bug probably belongs to godot as it also handles hand tracking.
Would you preferred that I copy the bug report over to godot or just leave it as a comment here (and as a closed bug in godot_openxr_vendors)
@@BastiaanOlij Would have sworn that I pushed the reply button yesterday, well apparently not.
Anyhow I created a bug report on the wrong project (godot_openxr_vendors), thanks goes to dsnopek for informing me.
If you prefer I can copy it to the godot project instead?
With regards to the bones I used godot-xr-tools/hands/hand_l.gltf and hand_r.gltf for the skeleton and mesh.
There is AR support for Android?
Only for Android based HMDs that support OpenXR such as the Quest, HTC Elite, Pico 4, etc.
this is called VR not AR
I'm not sure what makes you say this.
V(irtual) R(eality) encompasses creating a virtual world in which you are fully immersed. Everything you see is rendered which is not what we're dealing with here.
A(ugmented) R(eality) is adding virtual content to the real world, you are augmenting what you see.
Whether you are seeing the real world "naturally" with a glasses type device such as HoloLens (not supported by Godot), Magic Leap, XReal, TiltFive, Meta Orion and similar devices, with the virtual content superimposed through some sort of screen in front of your eyes;
Or whether you are seeing the real world through "passthrough", e.g. one or more cameras recording the real world, displaying this as a background and rendering virtual content on top, such as is done with phone based AR (ARKit/ARCore/Etc), Meta Quest, Lynx R1, Pico, HTC, the list goes on and on;
It has the same end result, we are augmenting the real world with virtual content.
That we are using a VR style headset to "simulate" AR, is semantics. It is why often we talk about M(ixed) R(eality) to indicate the fusion of AR and VR, but I'm personally not a fan of that because it creates an artificial boundary that gets in the way of us creating augmented reality applications that do not care what device you are using.
In that way I don't understand why people have no problems calling phone based AR, AR, even though this is technically passthrough, no different to what we're doing on "MR" headsets such as the Quest.
This is off course only one piece of the puzzle, how we technically solve the issue of displaying virtual content within the real world.
Another important piece of this puzzle is scene discovery, where sensors map the real world so we can accurately place virtual content in a way that makes sense. Put a hologram on a table, hang a virtual picture on the wall. Again scene discovery does not care what type of device you're using, its the functionality that counts.
Here Godots OpenXR implementation currently only supports scene discovery for Meta HorizonOS but this means you can create AR applications targeting the Quest line of devices, and likely deploy your AR application as is on Meta Orion glasses as it runs the same tech stack. I can't verify this yet as I do not have access to Meta Orion glasses but in theory, it should just work (tm). As other devices expose scene discovery APIs in OpenXR, this functionality will come to those devices as well.
That is the goal we set out creating this solution in Godot, to remove the artificial difference between "optical AR" and "passthrough AR" for the developer. The same code allows you to deploy on any device that allows you to add virtual content into the real world, while you as a developer no longer need to worry about the technical difference. With "optical AR" still emerging, it allows you to start creating meaningful AR applications while using devices such as a Quest that will work when affordable "optical AR" devices hit the market.