Apple's impressive entry in to photogrammetry: Object Capture API
ฝัง
- เผยแพร่เมื่อ 30 ก.ค. 2021
- With OSX Monterey comes a new Photogrammetry API that allows developers to build applications around it. Even though OSX is still in beta there's already one application out!
Of course I had to give it a try and I'm happy to see the application and the API behind it work wonderfully!!
Access 3D files, textures, discounts and more through my Patreon page:
/ katsafouros
PHOTOGRAMMETRY RELATED VIDEOS:
Updates to Apple's Object Capture API: • Apple's new and improv...
Workflow after scanning: Retopology and texture maps: • Photogrammetry: The ne...
Using video instead of photographs to capture objects: • How effective is video...
Camera VS phone. Here's a comparison video: • Is a phone good enough...
Face scanning: • The cutting edge VFX o...
MY TUTORIALS & 3D ASSETS
🧱 3D scanned debris: www.marvelousdecay.com/gumroa...
🥐 3D scanned food assets: www.marvelousdecay.com/gumroa...
📼 Adobe Illustrator tutorial: www.marvelousdecay.com/gumroa...
PHOTOGRAMMETRY GEAR:
Genie Mini II: amzn.to/3LYsANu
FalconEyes RX-18T: amzn.to/35yviJT
Linear Polarised film for the light: amzn.to/38RLiZ9
This very much depends on the size of your light. If you have a smaller light you can get something smaller than A2. And of course the other way around.
Gobe lens polarise filter: amzn.to/3jAkZsr
Make sure you buy the right size for your lens. Or you can get a bigger filter and use step down rings for all your lenses
Note:
When you buy something using the affiliate links in this video or any of my other videos, I earn a small affiliate commission at no additional cost to you. Your support is appreciated!
---------
Host & Creator: Dimitris Katsafouros
Website: www.marvelousdecay.com
Twitter: / marvelousdecay - วิทยาศาสตร์และเทคโนโลยี
Imagine when photogrammetry/nerf gets improved enough that you can just walk around with your smartphone and 3d scan your city. It could be crowdsourced to scan the entire earth and build a simulator/game like GTA.
We are already doing it. There is a lot we can do with that data. Consider Nural Link + digital earth + VR life especially for those who are bed ridden.
Does want 🤤
That's a great video as always!
I really love the simplest and quickest way you explain Mac and C4d related. Good jog! 👌
Hello from Japan. Your advice about lighting has helped me enormously. Thank you.
This video is amazing! I had no idea this was a thing.
I'll be thinking of ways I can incorporate this type of tech in an upcoming design project.
This video is really thorough, well explained and useful! A 1000 thanks!
This is amazing. Was a PC user, with a top end iPad Pro, I hope this will be coming to iPad which has easily enough power to do the same.
It's impressive how easy these things have become, especially considering how quick it is. I wish they'd improve the amount of detail you get though.
Thank you, this was very helpful, and I'm quite excited for OS X Monterey!
My Mac is dead because of Monterey. It’s not ready yet
This experiment/tutorial was so smart, thanks for sharing.
What a sensational video and personality. Thank you!
Πολύ καλό βίντεο, εξαιρετική παρουσίαση. Μπράβο Δημήτρη, Α! και χρόνια πολλά :)
Thank you!
Holy crap. Amazing technique and really good video
love your music on this video! refreshing
Very informative and practical. Well done!
Absolutely fantastic vid!
Fantastic. I didn’t know this was coming in Monterey, but it comes at a good time, as I’m learning 3D and I aim to do photogrammetry; this might mean avoiding expensive software, certainly enough for my project. I’ve already studied up on the subject a bit, inc cross-polarisation. A trick I came across in relation to shiny featureless objects like that statue in the video is to flick flour on it with an old toothbrush, just to give it texture for the camera to capture. They were using a ring light with a polariser sheet cut to size.
Thank You very much, Dimitris!
Really cool walkthrough!!
Μπραβο μητσαρα ωραιο βιντεο καλη επιτυχια με το καναλι σου!
Σιγουρα αυτος ο τροπος φαινεται καλυτερος σε σχεση με τα παλουκια που περναγαμε παλια στα windows.
SUPER great, thanks for sharing :)
Great stuff. Thank you.
I've heard and seen some tutorials from Siemens recap that having a blank background isn't always best and applying multi coloured dot stickers can actually improve performance so it can differentiate where the background is....in this example it might make it pick up the background and not work...however putting the stickers on an object could improve some things.
How did you get your model from the app to a obj? Do I need to convert it? Thanks and great video!!
Now I have to update that spare Mac Mini to Monterey, purchase filters and try it out! Goodbye weekend (Thank you Dimitris)!
Hey Dimitri .... Efcharisto poly! I was searching the whole internet (nearly) to find a software for OS X ... and I didn´t find anything that was good for my needs. When i saw your video I couldn´t believe that a solution would be that easy...really deep THANK YOU!🙏
You're welcome! Have fun building your own assets!
daaang. so impressive. Will be a game changer for the multiverse.
Looks great!
Amazing
thank you for sharing :)
Thank you for the video
wtf man . how good is this please
Great video!
btw, what was the final file (average) size of those objects? are they usable to be displayed on a website? sort of like Ikea place
As mentioned towards the end, you’d need to retopologise to optimise the mesh, and to bake the details in, which makes it easier to animate. This is standard in the animation and game industry, you get the details but with a lighter file that is easier to compute.
Wow, can't wait for more apps like this!
Do you think the LiDAR enabled iPhones might provide more precise scanning?
Actually with iPhones and iPads LIDAR, things are less precise but there’s one befit with LiDAR. A more real time feedback of what is scanned. I’m sure things will get better in the future but for now LiDAR will give you less detail than the pure photo approach.
Sorry if you mentioned this somewhere but how did the program calculate the stucture of the side which is on the ground? Did you turn the object on its head and do a second photo series and then somehow merged the two series?
This is super cool
Great video! I wish to know what the result if we use this to create a digital double of a human, thanks in advance.
Super interesting ! have you tried transparent objects ? (glass, glasses, packaging...)
You’d need to treat the object so the software can pick up the surface, like splattering on flour with a toothbrush.
WOW -- that pinecone!!
I would like to use this for 3D printing
Might be wrong but you could probably get free polarized filter from broken LCD/LED Tvs... I have one with a broken LCD, will definitely try.
Photo gram tree looks good!
Helli Dimirti! Thanks for great video. Which software you are using at 12:35 (Model Editor)
Hey Sercan. Have a look at the description. The app is called Photocatch.
You said it's very fast, what Mac are you using?
From my experience working with metashape and capturereality the process needs a lot of hardware power and time.
It's a 10core iMac Pro with a Vega 64 16GB card. I didn't shoot a whole lot of images for each object so that's probably the reason.
This so cool Photogrammetry. I’m also curious about googles photogrammetry because google street view looks amazing.
Can you do a test of trying to capture a person? Excellent video by the way
Great video! How does the linear polarization filter work. Does the effekt work if you have a smaller filter over the light and then a softbox? (The filter inside the softbox)
Or do I need to do it like you, the filter direct on the light and no light shaper?
You would have to use the filter over the soft box so that will increase the costs by quite a lot. What you could do is move the light source closer to the subject and just use the light without a filter. This will give you the soft shadow you want. By moving the light closer you're essentially increasing the size of the light which is what the soft box is doing. By not having to use huge filters you'll get to save a ton of money!
@@marvelousdecay Thanks for your answer. I will try both. I going to buy a A2 size fitler over my flex light panel like you did and try a big softbox close to the subject. Will be fun to see the difference in the result.
Quite promising // thanks
I find the potential accessibility of this jaw dropping. Any benefit with doing it via the video capture option?
I don’t get how does it know what’s the bottom or the top after having flipped the object for the next set of photos… are they all exactly the same? I.e. 20 photos for the side and another 20 at a 90 degrees for example?
The software analyses the pictures and finds the same area in all the different photographs. That's why smooth surfaces are difficult to do with this technique. There is a workaround for that by dusting the object. This introduces more detail for the software to use.
The more pictures you shoot the longer it will take for the software to put everything together but the better the detail will be.
@@marvelousdecay thanks for the clear up.
Will be playing with this now that Monterey is just out.
Cheers!
Amazing
can you export to fusion 360 for engineering cad models?
Hi Dimitris, I’m having trouble modeling simple objects like box products or cylinders they all look like they have been on a fire. I try a stick off wood and it works perfectly. But I try to model a laundry detergent container and it came out like it has been burned. I haven’t tried polarizing the light I just ordered it. I hope that works. Thanks for the video.
Hey Carlos. Probably the smooth and reflective nature of the products is giving you a hard time. Try to powder the products to give them some distinct details. This will help the software to distinguish the different areas. That might solve your problem.
Hi, I'm guessing you could use blender to place these objects in? Thanks for sharing this. Fab.
Amazing results. Shame that i don't a Mac at my hand to try this out.
I have a dedicated photogrammetry application that I have been using for a year or two that was pretty expensive. It’s nice to see Apple baking the ability right into the OS. I get the feeling that the app I own has poor support for GPUs under MacOS, this of course would not be the case with an Apple centric solution as I am sure it will take full advantage of metal and the new GPUs in Apple Silicon. I am using photogrammetry to digitize my own sculpture, so I am a little concerned that they won’t be interested in making it capable of capturing ultra fine detail, (it seems to be more of a web retail kinda tool) but maybe that is up to the developers that build apps from the API to figure out.
Man! I don't know how Ryan Connolly has time for Film Riot AND this channel! JK, great video!
Hi Dimitris, thanks for this tutorial!! When I create a model, I get an absolutely perfect result, but when I export it, the model looks like it's melted / distorted.
But it seems like you manage to get the same quality out of PhotoCatch into C4D.. I'm not paying for the software, would that make a difference?
No you don't have to pay for the desktop version. I assume you export a lower res version and that's why you get this bad result. Just export the RAW file and you'll be good.
@@marvelousdecay Thanks for replying! 🙏 I've run the model on RAW and still get the same result. I've noticed that if you go into the 'preview' mode and change the Shading to Physically Based, that's the model that I get when exporting ( both .usdz and .obj )
Wow!
Would this app work for something like a building? Hat about a mix of photos like some from a phone, some from a drone?
Yep it will work with buildings. Have a look at this video here where I go through a scan of an interior. th-cam.com/video/xPkB6I30cxg/w-d-xo.html
As for mixing photos that does work too. Usually I mix photos using different lenses but I think it'll be fine with mixing different cameras.
I can't find software that will work on Monterey 12.6.7, (Mac book air) does such a thing exist ?
Possible to share the program used to take pictures of the pine cone and rocks? It appears it placed blue dots on matched corners or areas?
That's Panasonic's camera controller app but it's not doing anything fancy. The blue marks on the image just show which parts are focused.
what the heck! awesome
Is there any way to use this technology with an older mac? i have a MacBook Pro (13-inch, 2017, Two Thunderbolt 3 ports)
All Photogrammetry programms ive tried wont work with my system.
Very grateful for any tips, or programms that i could use
Unfortunately photogrammetry is resource heavy. It usually requires a beefy GPU or CPU. Maybe wait a few months until Apple announces the new Mac Mini and get that. It won't break the bank and it's going to be optimised for this type of work.
Thanks for the video. Promising. However I only can manage to get "PhotoCatch couldn't create a model with these images. View some tips and try again." Well, it sucks. I tried photos and videos. OK rather quick and w/o polarizing. But 0% success is way too low :(
Is it possible to do it on server side with swift once user uploads the image and make a 3d object on the server side itself?
You can do that. In photo catch (and especially the mobile version) you have the option to upload your images and to do the processing on the server side. To be honest though I don't see the need for it. The object creation takes little time so it's not an insanely time consuming procedure. Especially if you have very few images.
Hey! Can you tell how many polygons the objects ended up with by any chance?
I'm thinking about making a game but I'd love to use a setup to scan the objects like you did and use them in the game instead of learning how to use Blender and make them there. But depending on the complexity of the objects it'll be hard to use them in the game without problems. Do you know how many on average did they end up with? Thank you!
The objects I shot range anywhere from 50.000 polys to 250.000. It really depends on the amount of detail captured by the photos. I wouldn't recommend though using the objects straight out of photogrammetry. You should try to optimise the assets for use in game engines. That is relatively easy to do with software like ZBrush.
In the end you should aim for a low poly mesh with all necessary details in textures (diffuse, normal, bump displacement).
@@marvelousdecay After optimizing them do you think how low in terms of poly count can they get (while keeping the quality)?
@@branseed3479 Depends on the object but you will be able to reduce the number of polys by quite a lot. 1/6 or more of the original number
wow this is so promising
would like to see the mesh.
Nice video
I wonder how this might work with a drone? For mapping an area? I have a MAC and all the high end software for this stuff in Window based or subscription dependent.
Works without any problems. I’ve tried it with drone footage and there are no issues.
@@marvelousdecay I'll give it a try and thank you. I need to first figure out 'Dronelink'...this app ain't easy.
hi, did you use the GH5 to film yourself? and window light? thx
yes to both!
@@marvelousdecay Thanks
Thank you for this video - much appreciated. I subscribed to your channel. Now, I have a Q I hope you will kindly answer. How can I use this to display products? Do I have to make a video after photos have been taken - do I have to make a GIF? Or what should I do so the product can actually move around so my potential client can view my products from all angles automatically without having to do it manually? I should just add that each product has its own individual design on which means I have to find a way to export the 3D image to Photoshop, add my design, and then export to my website for display. How can I do that?
Hey Izak. What you're creating with this process is a 3D object. So you'll need 3D software like Blender or Cinema 4D to open the model and render it from whatever angle you want. Once your image or animation is exported you can open it in photoshop or any other image editor. Alternatively you can upload the model on Sketchfab and then send your client that link. That will allow him to view the model from all angles
@@marvelousdecay Thanks for your reply. So, I do not have the time to learn Blender or Cinema 4D (way too expensive anyhow), so I will just get someone on Fiverr to do the final work for me. Better yet, where can I find already made 3D mockups (t-shirts, mugs, etc) for my purpose?
Shoe product-collage: Running with two left feet looks uncomfortable, should have mirrored one of the shoes : ) Thanks for your Insights! 👟👟
You’re welcome 😊
woooooah, shit man, things are getting crazy in the 3D space
Thank you! Great tutorial 😀 Now the big question... Do you think I could use a drone to create a 3D model of my house? 😀
Yes you can! As a matter of fact that’s how the majority of big outdoor scans are done! Good luck with your project. You’re gonna have a blast!
@@marvelousdecay Thank you! Actually I just did it yesterday and it came out way better than I expected 😀
@@DavidSelesi What drone did you use?
@@DavidSelesi And what program?
@@alexsafayan7684 I used the DJI mini 2 (took between 100 and 200 photos) and PhotoCatch on a macbook pro. If you use a PC maybe try a different app because 1. PhotoCatch doesn't give you a lot of options and 2. When I tried making a more detailed model (aprox. 800 photos) i let it work all night but it never finnished the model.
Samsung Note10 plus from 3 years ago was already doing that
Do you need to enroll as a developer to be able to compile and run those apps?
No. You can just download OS X public beta and photocatch and you’re good to go
@@marvelousdecay Thank you very much! =)
Do you know what does the RAW format in PhotoCatch?
That gives you the highest quality possible, without giving you the extra produced normal, AO and displacement channels. Definitely use RAW over any other mode. If you’re short on time you can use the other modes but you can always produce your own maps later on from the RAW and in a better quality.
@@marvelousdecay thank you
What progam is he using?
Would be cool to be able to turn 3D into a 2D drawing?
Do you think Lidar will eventually get to this level?
Not anytime soon but who knows what the future will bring!
do you know what app is being used to take shots at 2:08?
My guess is it's just a prototype app for Apple to showcase the feature and demo it in their presentation.
🔥🔥🔥🔥
Καλημέρα Δημήτρη!!! Πού μπορώ να βρω την εφαρμογή Photogrammetry API ? Κάνω αναζήτηση στο App Store και δεν μου την βγάζει, ποιά ακριβώς είναι ?
Καλημέρα. Το API δεν είναι εφαρμογή. Είναι η βάση για να φτιαξουν οι developers το προγραμμα τους. Επομενως δεν θα βρεις κατι στο App Store. Μπορεις να χρησιμοποιησεις την εφαρμογη που εχω στην περιγραφη.
GH5 is great camera!
Apple VR is coming !
Yea, after seeing this, and reading more into apple's tech for capturing "objects" - it's not just apple VR that's coming, butr the next iteration of this technology will enable us to capture video this way. It's called volumetric video. Google calls a similar technology "light fields". I think Apple may have nailed this one. I'm kinda shocked.
What is the app on your screen in 5:36?
It's called Syrp but it only works with the motion control system so you need to have that in order to take advantage of it.
I need to know the music
Damn, my MacBook Pro with 16 GB has just 2 GB :( Any other great software?
You could try Metashape but I think it will have similar system requirements.
how can i exports the models to blender?
Either download a usdz importer for blender or just rename the usdz file to zip and unzip it. The folder will contain a usd file which you can open easily in blender.
What can I do if I don’t have a turntable
you can just move around the object. but be careful to have a lot of overlap in your pictures. that’s the only way for the system to successfully identify landmarks.
Photogrammetry is the science and technology of making measurements using photographs...
Super... and I love the choice of the music, not the usual banal music for this kind of videos... I'm not an expert of the Apple environment, does this app work on iPhone, too?
Yes it does! Didn’t cover that but an iOS app is already available on the App Store.
Yes the music..it’s amazing!!
@@volumeup2714 yeh have to agree, its called Smash by Shtriker Big Band
hey man this is nice and all....but tell me about your workstation specs... :)
the iMac Pro has a 10 core CPU, 64GB of RAM and the GPU is a Vega 64.
@@marvelousdecay if I may trouble you again, what specific vega 64 it needs to be able to do ray tracing
Does it work on Intel Macs?
It sure does. I’m on an intel Mac.
@@marvelousdecay thanks I wonder what I’m doing wrong 🤔
I’m on a 2017 IMac any thoughts?
I try to pair the iMac to the iPad Pro with IOS 15 installed? But it won’t pair, maybe that’s it?
123DCatch разве не сделали тоже самое 8 лет назад?)))
I was about to ask you about pricing, but I've found the info.
For those who might be interested: For a full quality object from 100 pictures you'll pay $16.99.
0.15 per picture + 1.99 per object.
You can always use the desktop app. The processing happens locally so it’s completely free.
@@marvelousdecay That's great! It's not clear visiting their website. Thanks for this info!!
@@marvelousdecay witch is the app?
Looks like a 30 years old QTVR application
great vid. one thing I dont understand though. instead of shooting images, can you not just roll video and extract an image sequence? I mean, video is literally 60 photos a second, should be overkill actually. Another thing, im surprised apple hasn't somehow combined their LiDAR Scanner with this app, and introduced LIDAR embed photos. I know the 13's have this with the false bokeh cinema mode. though the LIDAR is actually pretty small resolution for photogrammetry, still might help with alignment. anyway, exciting news. though im not quite ready to jump into Monterey. ill give her a few more months
You can actually shoot video and the program can use it to reconstruct 3d geometry out of it. But video is a bit tricky. Trickier than images. There are two issues mainly. The first one is the shutter speed. You don’t want blurry images so the shutter speed needs to be quite fast. Which in turn means that you need a ton of light. The second issue is the fact that you’ll be grabbing a lot of the background so this can mess things up.
But yeah there are people who use video for photogrammetry. It’s usually though external shots like statues etc.
@@marvelousdecay Yes, but you can shoot 4k video with your newer model iPhone at 60fps (which will reduce or eliminate blur altogether) and then used Photoshop to extract frames, probably in less than 5 minutes from capture to export. It's worth a try! In Photoshop: (1) File > Import > Video frames to layers... (2) Export > Layers to files (3) Done.
@@thwartd As I mentioned above, the app can already process video. There’s no need for frame extraction in photoshop.
Take the shine off the object, make it flat gray.
I wish I had my iPhone...Imageine that...
The app is not free, they charge you for processing the images. Something that could be simpy done by your M1 IPAD on the fly.
The app is free and the processing is being done on your computer. The mobile app and the desktop app are two different things.