Musk: Diffusion to Replace Transformers in Tesla FSD! HUH?! 🤔Transformers vs Diffusion vs CNNs
ฝัง
- เผยแพร่เมื่อ 14 ต.ค. 2024
- When Elon Musk recently tweeted that Tesla would move from using Transformers to Diffusion--and then on to something else, the big reaction from everyone was HUH?! What on earth does he mean by this? In this episode I take a shot at reading the tea leaves, discussing the differences between Transformers and Diffusion and Convolutional Neural Networks (CNNs), and why Tesla would "go back" to CNNs after working so hard on Transformers.
Join this channel to get access to perks:
/ @drknowitallknows
**To become part of our Patreon team, help support the channel, and get awesome perks, check out our Patreon site here: / drknowitallknows . Thanks for your support!
Go to drinkag1.com/d... to get started on your first purchase and receive a FREE 1-year supply of Vitamin D3+K2 and 5 travel packs.
Get 25% off Blinkist premium and enjoy 2 memberships for the price of 1! Start your 7-day free trial by clicking here: blinkist.com/d...
Get The Elon Musk Mission (I've got two chapters in it) here:
Paperback: amzn.to/3TQXV9g
Kindle: amzn.to/3U7f7Hr!
**Want some awesome Dr. Know-it-all merch, including the AI STUDENT DRIVER Bumper Sticker? Check out our awesome Merch store: drknowitall.it...
For a limited time, use the code "Knows2021" to get 20% off your entire order!
**Check out Artimatic: www.artimatic.io
**Want to get in on WeBull's awesome Crypto and stock fun? Check out this link, and get started trading stock and Crypto!
a.webull.com/i...
**If you are looking to purchase a new Tesla CAR, Solar roof, Solar tiles or PowerWall, just click this link to get up to $500 off! www.tesla.com/.... Thank you!
**You can help support this channel with one click! We have an Amazon Affiliate link in several countries. If you click the link for your country, anything you buy from Amazon in the next several hours gives us a small commission, and costs you nothing. Thank you!
USA: amzn.to/39n5mPH
Germany: amzn.to/2XbdxJi
United Kingdom: amzn.to/3hGlzTR
France: amzn.to/2KRAwXh
Spain: amzn.to/3hJYYFV
**What do we use to shoot our videos?
-Sony alpha a7 III: amzn.to/3czV2XJ
--and lens: amzn.to/3aujOqE
-Feelworld portable field monitor: amzn.to/38yf2ah
-Neewer compact desk tripod: amzn.to/3l8yrUk
-Glidegear teleprompter: amzn.to/3rJeFkP
-Neewer dimmable LED lights: amzn.to/3qAg3oF
-Rode Wireless Go II Lavalier microphones: amzn.to/3eC9jUZ
-Rode NT USB+ Studio Microphone: amzn.to/3U65Q3w
-Focusrite Scarlette 2i2 audio interface: amzn.to/3l8vqDu
-Studio soundproofing tiles: amzn.to/3rFUtQU
-Sony MDR-7506 Professional Headphones: amzn.to/2OoDdBd
-Apple M1 Max Studio: amzn.to/3GfxPYY
-Apple M1 MacBook Pro: amzn.to/3wPYV1D
-Docking Station for MacBook: amzn.to/3yIhc1S
-Philips Brilliance 4K Docking Monitor: amzn.to/3xwSKAb
-Sabrent 8TB SSD drive: amzn.to/3rhSxQM
-DJI Mavic Mini Drone: amzn.to/2OnHCEw
-GoPro Hero 9 Black action camera: amzn.to/3vgVMrH
-GoPro Max 360 camera: amzn.to/3nORGYk
-Tesla phone mount: amzn.to/3U92fl9
-Suction car mount for camera: amzn.to/3tcUfRK
-Extender Rod for car mount camera: amzn.to/3wHQXsw
**Here are a few products we've found really fun and/or useful:
-NeoCharge Dryer/EV charger splitter: amzn.to/39UcKWx
-Lift pucks for your Tesla: amzn.to/3vJF3iB
-Emergency tire fill and repair kit: amzn.to/3vMkL8d
-CO2 Monitor: amzn.to/3PsQRh2
-Camping mattress for your Tesla model S/3/X/Y: amzn.to/3m7ffef
**Music by Zenlee. Check out his amazing music on instagram -@zenlee_music
or TH-cam - / @zenlee_music
Tesla Stock: TSLA
**EVANNEX
Check out the Evannex web site: evannex.com/
If you use my discount code, KnowsEVs, you get $10 off any order over $100!
**For business inquiries, please email me here: DrKnowItAllKnows@gmail.com
Twitter: / drknowitall16
Also on Twitter: @Tesla_UnPR: / tesla_un
Instagram: @drknowitallknows
**Want some outdoorsy videos? Check out Whole Nuts and Donuts: / @wholenutsanddonuts5741
Outstanding! Thanks to you and James for supporting geeks who love AI but aren't living in the machine learning weeds.
Agreed. So helpful!
A little Dr know it all to cheer up in rainy Sweden. Awesome!
The cloud watching metaphor.
I have never heard such a clear and intuitive explanation of the topic.
Very nice!
Thanks. I understood about 25% of that. I did notice that the CNN looked like a Sudoku puzzle.
I'm glad that there are people like you who understand all this stuff, it's way above my walnut brain but maybe bits of it will stick.
The walnut-shaped amygdala does work to protect us from threats to our bodies or minds. In my case, my mind is second-hand, so it's seen a lot of threats 😱
It seems they are going all in on the general world model (aka DriveGPT). Its amazing that they can pivet like this and most likely abandoning a load of work they have done previously. Although, I would hope most of the important stuff will still fit into this new model so wont be wasted.
I think once they get this working on cars, the speed of improvement will be incredible. End to End models are the holy grail for self driving, it seems they now have the tech/knowledge to pull it off.
That's how Elon operates. It's why he's so frustrating for some people to follow and understand, but it's also part of the genius. Once he figures out that what he is currently doing will not get him to his goal, he's fully capable of dropping everything where it sits, performing a 100% pivot, and starting over from scratch, or from what is seemingly a position way behind where he was.
Not only DriveGPT, more RobotGPT.
Pivot
Thank you. You have a wonderful ability to help a common person make sense out of esoteric concepts. I also think I would enjoy watching Lex Friedman interview you.
This makes sense to me. If you render from white noise to reality, the solid objects you don't want to hit come into focus first.
Everything else takes a longer to resolve and is either further away or irrelevant. When we drive, we're responding to inputs in the millisecond range. The car has to be faster than that, or it is flawed. I don't think anyone is within a decade of Tesla, and in 10 years, barring an unforeseen breakthrough, they'll still be 8-10 years behind the data curve and probably further behind the tech curve.
Excellent point.
A little over my head! You did a very good job explaining, thanks!!!
Diffusion networks have famously high latency (because of the sampling process).
Diffusion networks are extremely smart, but I won't be surprised if they drop it too, because of the high latency.
Diffusion will be a stepping stone. Once you have the needed data for AB test the most efficient model will be a perceptron deep network.
Years ago I took a one-week image processing course at Georgia Tech. The most impressive item was a fuzzy, unidentifiable image. When the projector was DE-focused, the face of Abraham Lincoln was unmistakedly visible.
worth remembering that there can be multiple neural network "parts" used in FSD
As a database engineer for the last couple of decades I was so confused when you started talking about backing that thing up.
Thanks for the great video and the chuckle :)
This hits the sweet spot of challenging but still understandable.
Thank you for sharing this video. I have learnt something new today and you have definitely intrigued my curiosity. 🎉❤
Great video. Great explanation. Great disclaimer. Great speculation!
In the Twitter footer, it now says, "X Corp" and I can't lurk any more. Need to create an account ...
Thanks John, I'm hoping we're closer now to an FSD interview with Elon. Please Elon 🙏
Nice shirt, what brand is that?
Always enjoy your channel. Consider sunscreen.
So, giving foundation to intuition in a sense. So very Tesla. Thanks Doc.
Isn't the process that you described in diffusion exactly what our visual cortex does?
The signal coming in from our eyes has a very narrow focus in the center of our field of vision. Objects not in the focus point are blurry (diffuse). However our brains have been trained to get rid of that blurriness by building the reality of what we are seeing into something sharp.... based on our remembrance of what blurry images like that would actually look like if they were in sharp focus.
We humans actually construct the reality that is in our field of vision but that is not in sharp focus. This is why we are often surprised when we notice movement in our peripheral vision that was unexpected.
For example if there is a bird on the ground in our peripheral vision, that we were unaware of, our visual cortex may have completely eliminated that bird out of what we are thinking we see because it may think that is just noise. However if the bird should move we see that movement and turn our heads to focus on the object, once recognized we can turn our heads away and our brains now keep the bird in our picture of reality even if the bird is again being still.
This begs the question of whether we actually want vehicles to invent reality out of a diffuse background? I guess if it is better at that than a human then it would be a better driver, but can we ever trust it 100%?
FSD can save lives at much less than 100%; humans set a pretty low bar. The tradeoff on 100% is cost and efficiency in response time and compute power; i.e., EM's recent tweet about compute per watt. It is impressive they are working so hard to ensure FSD will run on the 4.5 M vehicles with HW3's limited memory. Awesome.
100% is going to be a problem just accept a 99.9999% accuracy. The nice thing about neural networks is that you can compute your loss function so Tesla can train a neural network and confidently verify that the model only has 1.21 hallucinations per million miles.
@@kazedcat Better than the pot smokers here in Oregon for sure! LOL
@@nickmcconnell1291 Yah! I've seen those potheads blasting down the road - at 4 miles an hour!
@@FrunkensteinVonZipperneck LOL.... while eating!
I like your take over James’s. Let’s see how this ages.
One minute per Elon Musk word in a tweet. That’s low bandwidth! Great speculation (and translation) video
Good speculation. This is a good match to the deconvolution "heads" you discussed in a recent video.
Again, super interesting and explained for a 5 year old, absolutely me here. Thank you!
Imagine Dragons: wow, Radioactive! ;-)
Outstanding, even I, a mere electronics engineer, understood this explanation. This would work out well for Tesla if correct. 😅
Backing up the reaction to "noise" is far behind where I would have expected by this time. I had a 1,000 pound bit of noise (a cow) walk onto the road as I was driving along at about 65 mph. The car did not seem to react. I had to quickly and firmly apply my brakes to avoid hitting the cow as she was right in front of me, taking her sweet time to cross the road. I would have expected my Tesla Model S to react to her crossing the road quicker than I did. The talk about getting the best cost per watt with AI is great for the future but we need to get to the point of noise recognition and reaction to perfection first.
I think that this what Tesla is trying to fix. I think Transformer need to have all the data with unknown object on the road blocking it, but the new model using the diffusion allows the FSD to determine that there is an object in front of your car and take action without the FSD needing all that training using labels, which is basically using tagged situations but may not be able to be used in your situation to resolve the problem since the conditions and situations is not familiar to the FSD program.
@@1flash3571 Thanks for the explanation but noise reaction could be adjusted. I came across another situation that I found annoying yet humorous. A dead skunk in the middle of the road made my car react by moving over as to avoid passing directly over the noise. Though the middle of my car avoided the dead skunk my right tire ran directly over the skunk. My car stank for a day.
Silicon Valley don't got many cows. Hard to catch a cow in a neural net.
diffusion networks often use attention too, mixed with convnets
I need a transfusion after this transformer diffusion confusion 😅😂
They've been very busy evolving the "local maximum."
interesting discussion. In the beginning, Musk talked about not locking designs down and that tesla designs were there for others to use, his goal was just to get everyone into electrification. Now we see others using his AI day presentations to gain footings and make changes, do you think its still Musks goal to keep it free and open or do you think that AI day is no longer a good idea if it gives ideas to what is now, the competition
Mr Musk recognizes that patents/innovations are quickly copied, so instead or resting on successes, he focuses on the speed of innovation - more is more.
Great stuff John thanks very much that has really clarified it in my mind and of course a big thank you to James as well.
Most interesting insight - thanks for sharing, from the outside with the rest of us - AI compute/W efficiency/performance crucial for Robot endurance! But you know that;-) - word count!
Labels are your foundation control for interaction, as iterations continue you must always maintain a control. Otherwise, it becomes quantum where a percentage is useless noise wasting watts and you'll never know why. Even as we utilize photon gates to increase speed, you still need a foundation control for interaction. Just my thoughts.
Isn't the process of removing noise called integration
The best way to remove noise is to turn off all the beeping machines in McDonalds.
Phil and Ashoka’s presentation hinted towards no more labels. Similar with elons tweet, video in - controls out.
Will there now be potential for a significant saving; Tesla NOT having to update those who have pre-paid FSD -AND- have hardware 3 / owners willing to pay for h/w4, won’t now (speculation) need a hardware update to realise FSD - saving a ton of effort, expense, and opportunity cost; while maintaining the resale value of the hardware 3 fleet & maintaining momentum towards a future of FSD?
Really good stuff, good real stuff。 Much better than all the shallow price guessing and pumping youtube videos
Thank you, John. Nice lecture.
Very nice, thank you.
We dont need to have ever seen a dinosaur to know we better not hit it fight it or fk it.
Great I am working on it?
I think another useful concept is annealing, which I think would smooth out he FSD process out significantly. The current system is highly underdamped.
Well you know what they say. If you can't dazzle them with science, baffle them with bullshit
They may be preparing for the massive increase in computing power that Dojo will provide.
It seems to me that labeling is important to facilitate the human interface, and human comprehension of what a process is doing. If there's a way for humans to supervise training with no way to reference classes of objects, I'm having difficulty imagining it. But it seems that a goal in training technology is to automate labeling because using humans for this doesn't scale.
Interesting! And yes, I think you may be correct!
Thanks for some good explanations.
Lots of info here but I get the gest of this and it's going to be amazing to watch what Tesla finally works out and where the research takes them.. and like the diffusion model for the computer network you talk about I'll be watching this one over a few times while I clear out the noise and grasp the concept completely. LOL
Fantastic video - apart from the fact that I only understood maybe 20%.... sheep are so much easier to understand! Regards from uk
What did the NavLab 1 (1986) use as a model? Transformers or Diffusion?
Anyone?
It's unfortunate that the A.I. Industry is using terminology like hallucination; that means something specific when discussing human mental function and or dysfunction, as this can lead to confusion and poor understanding of the process of A.I.
Thanks
Followed by Spiking Neural Networks? Doesn't that have the best performance per Watt?
Could just go straight there.
Thank you. Really interesting AND informative! It feels to me like Tesla is getting closer and closer to FSD. I thought you did a very good job of explaining something quite complicated. I want to thank you for trying, but I feel like you succeeded.
… please! Please have James Delma explain this stuff!
Seems to me Elon is prepping FSD for integration into Optimus space, generalizing the model opens a LOT of doors, not just a Model Y's...😉
Fascinating
🤯
The diffusion model that Tesla is using in their FSD is called "Stable Diffusion". It is a type of generative model that was developed by OpenAI. Stable Diffusion works by adding noise to an image and then gradually removing the noise until the desired image is produced. This process is called diffusion, and it is what gives the model its name.
The Stable Diffusion model that Tesla is using is based on the "ImageNet-1k" dataset. This dataset contains over 1 million images from 1,000 different categories. The model is trained on this dataset to learn the statistical distribution of images. This means that the model can generate images that are similar to the images in the dataset.
Tesla is using the Stable Diffusion model in their FSD to generate images of the road ahead. The model is used to help the car's self-driving system identify objects and obstacles in the road. The images that the model generates are used to train the car's self-driving system to better understand the world around it.
This is per Google Bard.
Is that like, a "Stable Genius?" And I don't mean the smartest horse...
Wow .❤Thanks for breaking it down, to someone that merely has a B.S...and 26yrs tech experience 😅
I got lost immediately
Interesting video. Diffusion is typically used on images that would be million dimensional data so for the driving policy Tesla would try to apply it for maybe 10 dimensional data if 10 values can describe their immediate path plan tentacle. Presumably a much simpler function that what midjourney amazingly pulls off but I'm not sure the diffusion aspect is particularly potent in their use case. Even for images diffusion is probably a hack and not the final solution but for images it kind of makes sense to jiggle out a solution among near infinite valid solutions. Driving is different. My hunch is that it will probably work fine, similar to current approach but no better.
You obviously know a LOT more about this than I. But what I see is that Elon builds systems, not cars. Cars are a part of the system, but his goals are way more long-term. The real hoot would be seeing an Optimus driving a Model 3 in 'manual' mode on its way to work at the Megapack factory before getting home in time to make dinner for its family, who are at home watching the live feed from the Optimusses/Optimi(?) building Elon's Mars Sanctuary...🙃
@@thesoundsmith hehe, AI wont parallel human life like that. It's not a person. Never will be. That it's impossible to inflict pain on a computer proves our spiritual nature. I suspect that the future wont have many androids, they will be more specialized and technology driven. Why have hands and legs or even a body if you can move things with fields.
How long will I be able to process FSD at a top level with my 2022 S Plaid ? I have a first week cyber truck order placement back in November 2019.
Well if the doc is correct then the focus for Tesla is still to make hw3 achieve FSD so you should be fine with your hw3 equipped car.
6:50) the "CNN camp?" If Jeffrey Toobin has anything to do with Neural Nets - count me out...
Nice shirt
Does a diffusion approach remove one of their moats? Do you still need a fleet?
Yes you still need data. All this talk is about how to cook an egg but no matter what you need eggs to cook an egg.
@@kazedcat but do you need a million eggs every 14 hours?
@@DanielASchaeffer Sadly yes current state of AI is not as good as humans in learning so you need millions of data to compensate.
@@kazedcat if you can generate any training input you need, why do you need a fleet?
@@DanielASchaefferbecause you can't generate any input you need. In addition the fleet allows shadow mode to be run on any vehicle with suitable hardware.
seems a bit weird. You'd think having AI transformed to a robust vector/occupancy representation then all you need is a decent higher order physics model to project physics tendencies of the vector representation's changes in velocity, position etc into the future objectively and not need some diffusion "reimagining", as real world objects like cars house people etc are typically sufficiently reliably persistent and trackable that deviations in objective physical attributes due to driver decision should be bounded by the same physical contextual tendencies eg making turns with fast on coming traffic will cause drivers to be more hesitant similarly vehicular size or occluded spaces will influence other drivers behaviour in a way which could be reasonably aligned with normal AI or statistics too versus some recourse to imagination... It's not like other objects go from 100-0 or magically disappear when integrated from multiple frames of data even if AI occasionally misidentifies. I mean that's my higher mental expectation when I interpret the world which seems to work for me lol....
Thanks John. It is unlikely that FSD can be solved by using a single AI model. Most of us aren't thinking broadly enough. FSD requires a paradigm shift. I am sure that Elon is snickering at our flatland musings. Prepare for multidimensional AI.
Best wishes to you and your family. Darwin
I’d think if evolution came up with 2 cameras for real world navigation(eyes), it may be too much to put 12 on a car, thus requiring exorbitant bandwidth.
Koalas are not bears. Sorry. Just thought you would want to know (it all) 👍
Deconvoluting Elons tweet.
Nice upsampling
Did we figure out who's going to be the next president 🤔 diffusion or transformers?
👍🏻🙏🏻❤️
Yes, diffusion is not a competitor to transformers. E.g., Stable Diffusion (an image generator) uses both transformers (for text input in CLIP) and convolutional kernels for the VAE (because images can be any size, after all).
Also, a lot of people get this wrong, but convolutional networks are better than transformers because of Divine Right. Basically, because God gave convolutional networks the right to be better, they are. It's that simple, but escapes a lot of so-called "experts." :-)
name ne a paper where convnets are better then vits?
and with DOJO Elon can process that data faster than anyone.
The title makes no sense, diffusion is a learning objective and defines the forward inference process, but it's not something that can replace a network architecture like CNNs or transformers/attention.
Can A.I. keep teslas secrets..secret.
HUH?
The only significant point in the tweet is that Tesla is nowhere near solving FSD. Switching models this late means they have reached a local limit so they have to use diffusion. Saying he might drop diffusion later tells me they are desperate to find a solution.
The computer per watt statement could be interpreted as... there are other companies that have advanced self driving compute, just not as efficient. Seems like the competition can switch models too.
What are you talking? Diffusion and Transformers are the current state of the art in generative ai, with diffusion for images or sound and transformers for text. Tesla however needs generative ai for routes, so both methods do apply. Diffusion is actually expensive at inference, except in latent space. It is interesting that Tesla hasn't yet found their final generative model. But that's the hardest problem indeed. Once they figure that out, they may actually keep it as a trade secret ...
The most you talk about applies to classification and segmentation (occupancy space etc.). That's not generative ai, transformers and diffusion do NOT apply here! You seem to not get something fundamental here ...
You seem to assume that the creation of the occupancy model requires generative ai. I do not think so as this is a more classical 3D classification problem considered solved.
Maybe I am wrong. But I am sure: the crucial challenge is generative ai for route creation. I hope that Tesla is NOT talking about something more mundane when talking about diffusion. That would be deceptive.
I thought you were gonna say the twitter sphere was all atwitter 😂
Did you ever wonder how the twitter sphere gets a round?
wow how you breakdown stuff to commoners...you should be a preacher of the gospel
Economics ultimately guides the future.
Economics is the joke that is not funny for anyone.
@@FrunkensteinVonZipperneck If you treat it as a "joke" you're making a big mistake.
so now that musk has finally tweeted out some technical jargon, we should be convinced he is finally right about autonomous driving. LOL. sorry. I have been fooled by him too many times before. self-driving is many years away. it does not take an expert to realize that. just basic pattern recognition. I could be wrong, but I doubt it. if I am wrong hallelujah, I am so ready to not own a car.
Hi Doc, I appreciate the diffusion explanation but the Elon Tweet you noted blew me away. I’m not a Twitter user but it looks like I might have to be. During both AI days, I felt the level of detail being disseminated was irresponsible. I may have even commented about it on a couple of Tesla TH-cam channels. Tesla really needs to back off the hubris and practice more security management. As an investor, I find this whole situation very disturbing. I won’t even ask how Elon knows the Chinese copied his architecture.
Well we know about Xpeng, right?
Just remember that Neural Nets were invented 80 years ago - why the AI industry still uses them as a foundation for AI is beyond me.
Anytime you have to "train" AI to ID something, you can bet a Neural Net is being used as a horse-and-buggy way to ID things.
A much better way is to use the millions and millions of things that any CAD CAM program can access. These are 3D models of almost everything a robot/car would see daily. That rendering can be viewed from 360 degrees in both axes. So AI should easily be able to ID everything around it without "teaching" it a single thing.
Anyway, FSD will fail - it uses the wrong hardware and software and will never reach Level-4 autonomous driving.
Since you are an EXPERT where you can say that Tesla's way of doing FSD is wrong, why don't you give us the CORRECT method, hardware, and software for it to work Successfully? I dare you.
Don’t pay attention to these types of comments
Who cares? Show me the money!!!
I understood very little of this but thanks for sharing
Guy has no clue what he's talking about
Good Job. Thanks
Thanks
Thank you so much!