Dojo: Tesla’s Massive Supercomputer Explained with James Douma #23 (Ep. 419)
ฝัง
- เผยแพร่เมื่อ 16 ก.ย. 2021
- I’m joined by James Douma as we discuss Tesla Dojo (training supercomputer), why its important, fundamental architecture, and its future implications on autonomous driving, Dojo as a service, Elon Musk’s Humanoid Tesla Bot aspirations, and more.
View full unedited video with James Douma on Tesla Dojo (9/16/21), • James Douma full inter...
View Tesla AI Day full presentation, • Tesla AI Day 2021
James Douma on Twitter, / jamesdouma
James Douma on AI playlist, • James Douma on AI
Social
🐦 Twitter / heydave7
🎧 Apple Podcast: podcasts.apple.com/us/podcast...
🎙️Spotify open.spotify.com/show/2iin015...
📸 Instagram / heydave7
Dave Lee on Parenting, / @daveleeonlife7659
💁🏻♂️ Disclaimer: All content on this channel is for discussion and illustrative purposes only and should not be construed as professional financial advice or recommendation to buy or sell any securities. Should you need such advice, consult a licensed financial or tax advisor. All views expressed are personal opinion as of date of recording and are subject to change without responsibility to update views. No guarantee is given regarding the accuracy of information on this channel. Neither host or guests can be held responsible for any direct or incidental loss incurred by applying any of the information offered. Author is long TSLA, LMND, SQ, and other stocks at time of original video publish date. When you buy through links in this video description, author may earn an affiliate commission.
Open an brokerage account at Interactive Brokers:
www.interactivebrokers.com/mk...
Buying a Tesla?
🚗 Car: Use my referral link to receive 1,000 free Supercharger miles with the purchase of a new Tesla. ts.la/david7556
☀️ Solar: You’ll get $100, I’ll get $500 for a new install of a Tesla solar system. ts.la/david7556
Check out my archived articles/posts on Tesla and investing: teslamotorsclub.com/tmc/threa...
![ลักพาตัว เอาลูกเอิร์นกลับมา...!! [เอิร์นไดเม่]](http://i.ytimg.com/vi/f4NVKlZ5qkk/mqdefault.jpg)
Dave’s interviews with James are my favourite by far. So fascinating
Dave made James a legendary guest, as well as Emmett, and other. Thanks for bringing such quality minds to our attention.
Emmett Peppers doesn't get nearly enough shout outs. Well said DS... well said.
Always love James’ take on everything. He should have his own channel
At one point in my career, I was responsible for end user hardware and software product specifications for bleeding edge genetic engineering equipment. This means that I only had to define end results, and not the how to do it designs. While I still understand 98% of the issues James discusses, even after 25 years of being out of the game, I never disrespected the magic of the operators who implement like James is probably able to do. Thank you James for a great presentation! I respect Dave Lee's attempt at drawing out information without a solid base of the issues. Good Job Dave! My question to James, if he reads these comments, is what is the difference between writing code in assembly for DOJO vs using higher level code for the compilers necessary for best Dojo performance?
Man, I didn't even think about how good these dojo chips would be at blocked matrix math.
Great job James breaking it down without too many technical terms.
David and James doing deep dives are always my favorite videos to watch!
And so great for my insomnia! Haha! Joking.
Mine too. I guess most of Dave's audience won't understand most of what James was saying :) Look at the number of views only 1K so far!
Yeah I trust James to separate the hype from the substance. He has a perspective that I find very trustworthy
That laserpointer is low bandwith, high latency.
lol - yea definately an issue when James is using it - pretty undecipherable and useless audio for most of the time that he is using it. Hopefully, Dave and or James will not use it in the future (without figuring out how to work around it).
😂😂😂
Audio freaks out for James description of the parallel chips. Somewhere around 44min. Lasts a minute or two.
Think that was on James' end when he was using that drawing tool.
Douma is the best. Could listen to him all day.
Been waiting for this since AI Day! Thank you both for taking the time to put this together.
You know what I look forward to in interviews with James by Dave? James explanation to questions from a slightly more technical perspective and Dave summarizing it in a more layman-understandable way. I think this is a really great way to learn personally. Please keep these interviews up @Dave!
They fit well together. Each makes the other better.
Unreal! I find this the most stimulating exchange online today. Dave and James are delivering a depth of information and knowledge that we can simply geek out on! Invaluable. I think we’re seeing the actual birth of the AI age in real time. Thank you both so much.
James is an asset that we're only beginning to fully understand and appreciate. The more we learn of Teslas master plan the more we need James to decipher it for us. His ability to make the super complex world of neural networks and the systems they use to run them understandable is a very special gift. That he takes the time to share that gift with us is amazing. Thank you James.
I want James' t shirt! Another fabulous session with James! I would be interested in a future session to have him explain a bit about why matrix multiplication is such a key part of the processing. It's been a while and I am a little rusty on my linear algebra!
James is so educating!! Such clearity of thought and great delivery! Love love love this
This is very very intellectually heavy material. Takes a lot of paying attention. Excellent stuff Dave and James. Thank you.
Just want to let everyone know that there is an uncut version in the description as well, 30 min longer ;)
Whats in the unedited thats not in the edited? Is there info or just extra that wasn’t necessary for the full video?
@@harsimranbansal5355 I went for the shorter one haha, I just saw it in the description, sorry
Damn. I would have watched the uncut version first had I known.
Many thanks!
(Dave if you're reading please consider pinning a comment mentioning the extended version!)
Can't wait for the rest of the videos with James in the series.
Yesss... Thank you James and Dave. Love you both ❤️
Compliments to both of you, had to watch this intv several times to sink in..
I was so excited for this video! Tesla bot is so much speculation, but Dojo is real and will be operational not too long from now
Professor Douma strikes again 🍿. Geek out time🤓
Thanks so match for all the time and effort from both James and Dave. I took the time to carefully listen to all the video and really enjoy it. So much helpfull information for Tesla investors but also for anyone looking to be relevant at the future workplace.
Love the in-depth research of this channel! 🔋🚗⚡️
Dave and James! The best combo in TH-cam!
Dave and James together! This is going to be a treat!
Crazy how much James understands Dojo, almost like he worked on the design 🤯
Boom! Been waiting for this
Absolutely wonderful. (full of wonder). Thank you both.
29:00 Driving down computing costs by achieving very large scale is kind of what Amazon did with AWS.
This is an excellent video as always - great questions and clear answers. As a future topic, it would be great to talk through how compilers can be optimized to take advantage of this architecture. Tesla talked about being able to use standard ML libraries and languages - how easily can will these map these onto the Dojo architecture?
The 1980s processor James referred to when discussing communicating with its neighbour was called a "transputer", made by Inmos. There were some jobs which were eminently suitable for it but overall it was limited by the fact that its intercommunication speed was slow. The transputer did have four interconnections though, just like Tesla's nodes.
Thanks James and Dave, good stuff!
Thanks Dave and James! 🙏
Insightful interview as always, Dave and James!
Great job. I love the talks with James.
James is a national treasure for investors.
I suggest watching the uncut version (link in the description). The cuts make the content hard to understand at times. Thank you James & Dave!
Brilliant! Thank you ❤️🙏🏼
I'd listen to James all day long... 😎
🙏🏻🙏🏻🙏🏻 bravo, gentlemen. Bravo!
I was looking forward to this!
Very informative episode. Is that an FSD screenshot T-shirt that James was wearing?
I was wondering when this image would become art or design. 👍
@@JamesDouma nice! Thanks for the information. Great interview and thanks for your time.
Thank you for another great episode. But just a heads up. I think there was a connection lag while using the drawing tools. While James was explaining. A side note for future videos. Around the 45 min range
Thank you for interview!Advanced in Tech,...Tesla❤🌍
James rocks well done
Awesome, Dave and James! Will quantum computing ever be combined with Dogo?
Great episode, thanks!
Ok Tesla, you convinced me to join the Tesla team.
Can you please go inte more detail on the other mammal sensory inputs that could be used for a general AI?
Multiplying 2 matrices can be highly parallelized. In theory every single multiplication could run on it's own compute node. The results of (billions of) these multiplications then need to multiplied again and so forth. Therefore, Dojo's core arch (min ~40) matches this parrallelisation so well b/c each node can handle a portion of these huge matrices. It resembles what Joe Armstrong once said: 1 core per function would be ideal and then message passing between these cores
On the surface it seems there is no limits to escalate. My question would be what is the dividing line in terms of performance where one single supercomputer had reaches his practical limits? Lets say for example instead of having one with 10000 nodes or having 2 with 5000 nodes each.
Inmos had a general purpose Transputer that had amazing connectivity compared to what was on the market at the time. That used Occam parallel programming language, which was fascinating. Sadly, that project died a premutare death, probably for the reasons James says, it it was hard to formulate programs to take advantage of it.
How does Tesla ensure that (or estimate whether-) the size of the net running on cars is large enough to capture ("understand") the depth/complexity of level 5 autonomy?
Can someone explain that if DOJO is to address computational needs that Tesla will have in the future, how does that square with the FSD rollout timeline? Which one has to or can come first?
If Dojo is needed for FSD and Dojo is going ot take 2 years to be online what does that say about when FSD will be ready?
Anyone know who's selling the t-shirt James is wearing?
what are you going to do for episode 420?
Where is the DOJO computer located?
How long would the best competitor need to get where Tesla is today with FSD?
Another thing is that communication takes a *lot* of energy. So the massive amount of energy use by the dojo chip is going to be used more efficiently than with a competing solution.
Could you two loop in Anastasia in tech and do an episode on TSMC, Intel, Samsung, ARM, AMD, Nvidia - stocks to invest individually, basket or an ARK fund. This seems like guaranteed growth so 15% over 5 years is sound for me if the risk is limited which it seems true as a basket.
Go out and enjoy my weekend, or... da hell, click on this!
James is awesome
In regards to economy of scale building server farms, there is a big difference between a commercial server farm and a supercomputer.
A regular server farm uses off the shelf cabinets and cooling and servers and networking and power supplies, hooked up in standard ways inside buildings built with standard blueprints and engineering.
They are the very definition of economy of scale because everything in them is priced in volume.
If Tesla only builds one Dojo, nothing will have economy of scale, because everything is custom and limited production.
This means that Tesla has an incentive to build multiple of them, probably with improvements on each one, and then rent out the additional capacity.
Unless they are hand making the parts, might as well leave the machines running turning parts out since you have already invested in the design and manufacturing capabilities.
At 29:55 you are describing how AWS was born. Would Tesla use an existing cloud, losing profits or build their own??? Given their proprietary hardware, etc I suspect build their own using open source. I think the architect in this manner and if aI AAS manifests itself then they pivot or expand to that. Building this type of platform now is the key point and I feel certain they understand this. Note, data ingress and egress between Tesla and cloud is critical design point .
Anyone know James Douma background?
Nice job, thanks to you both. Would like to see next talk cover the data changes between 2D and 4D on neural network. Seems like others are still processing 2D or are others processing 4D in volumes Tesla is capable of? Read speculation somewhere that data could use anywhere near the data they have available. AI I was involved with in IT had training with images being the most data intensive and we served with more GPU's where non-photographic data got along well with more CPU intensive nodes. They are basically now processing movies plus car data and other non-graphic data. Seems to me they will need more horsepower just to drive the new massive data pipeline at their disposal. Better AI seems a better potential now with several combined advances in the HW, SW and Data stacks. Version 10 is good, but give them a year and version 11 will be a step function advance. Will it take a year?
James at 1:12 are you looking at a clock on the wall to see what time it is? lol
Finally man!
Would there be any benefits to have sound detection in the cars for better spatial awareness?
Doppler shift to calculate objects trajectories for one, yes, no?
@@Mellowyellow8888 I dont know..
I just reason from my own perception of spatial awareness and i think hearing helps me focus my vision in the right direction.
Audio gets really bad at about 44 minutes. Put in subtitles at this point and reupload?
Warm regards, Rick.
Thank you so much…this is greatly helpful for my better understanding of where Tesla stands with their current development. I continue to feel confident in further investment even at the current stock price level, considering most of my investment happened starting in 2015. As a disclaimer, I know you are not giving investment advice but rather information to understand Tesla.
heads-up - The audio when screensharing or whatever that was (~45 min) was unintelligible
James needs his own channel
Anybody remember the INMOS transputer?
Wow James does such a good job explaining these advancements to us uninformed NN people.
Tesla DOJO is killing it.
I love Tesla!
Chasing the 9s..
So no competent fsd until dojo starts chipping away.
Well impressed with V10.x
I want to know James edu. Is he PhD from cal tech ? Also, have him say something in Japanese. He is my favorite guest !!!
I'm about 30 mins in and you are discussing economies of scale of Dojo - I'm coming to this conversation late but much of it is still relevant - people are now talking about the limiting factor for compute becoming not chips but electricity - the beauty of this is that Tesla already vertically integrated into the power generation stack. Will we see enormous Dojo centres colocated with the world's biggest solar farms somewhere in e.g. Arizona ? With huge battery packs to smooth supply overnight ? With all the equipment having Tesla labels on it and all contributing to the economies of scale of the respective products ?
I've just updated myself.- EM is now saying that the limiting factor on powering compute (read Dojo) will be electrical transformers to step down from distribution voltage to corporate voltage to rack voltage to chip voltage etc. If we play Elon's game of going back to basics then the problem actually becomes getting enough power out of a battery (charged by a solar panel) to run a chip - (this also avoids having to convert ac to dc) - so if the batteries were custom designed to do nothing but run a Dojo you could perhaps do away with all the intermediate transformers and the heat that they generate and the equipment used to remove that heat and the power requirements for that equipment etc... Suddenly a solar powered Dojo begins to look much more feasible... And now a step too far - perhaps you could build the chips, the batteries and the panels into a single unit - a tile - and just lay out acres of them on a flat piece of desert somewhere - maybe floating on an artificial lake of nice cool water to keep them perfectly level...
I really think is in Tesla's interest to leverage idle Dojo time for something more than just renting out that time to others. Maybe they get a free license for a GPT-x model for allowing access or they get.a stake in company etc. Though I wouldn't put it past Elon to make something huge that is indisputable considered the biggest/fastest system in the world even at FP16 to rival AWS and Jeff who.
You really should pay tribute to the Transputer and the companies which built the early supercomputers with them, like Parsytec. Esp. when it comes to this 2D network topology …
Fun fact: Parsytec managed to create a fault-tolerant 3D grid out of the 2D grid components. Fault tolerance is key to get these large networks to boot up and work reliably btw.
Gpu support for video call and using Google drive not good.
I understood some of those words
Dojo is the machine that builds the AI machine and is put into the physical machines built by other machines
FINALLY!!!
Dave what would we do without you
I found it funny how while explaining bandwidth bottlenecks at the 45 minute mark, he was experiencing a bandwidth limitation.
One question - James - where did you get the cool FSD T-shirt? That's the extent of my understanding on this subject ;-) On the other hand you don't need to understand what James is talking about - you just need to understand his positive vibes about what Tesla are doing - to know they are on the right track, and my TSLA shares are worth holding onto.
The highest use of AI could be to help us find our better selves and preserve it.
Just a thought🤔
Maybe the TeslaBot is the first segment of a path that leads to AGI. But I doubt it. Teslabot if implemented is the equivalent of the advent of the personal computer back in the 80's. If the tesla bot is capable of easily navigating most environments with little to no code. While providing a simple way to program higher level behaviors. That unlocks ALOT of potential. We dont really need AGI to get all the way to or damn near most of the way to that robot driven future we talk about as being brought on by the advent of AGI.
We have so many technology savy people available that having them build out a library of skills that you can easily swap in and out of your tesla bot is world altering all on its own.
He didn't even talk about the power savings, compute clusters use millions of dollars worth of energy per year, its not the biggest issue for now, but if you scale up the current compute by a thousand then you start saving hundreds of millions per year.
51:50;
a small improvement. Dojo is not a node, because
Dojo is a set of nodes, as he later correctly reports.
Nodes within nodes within nodes interlinked.
I need that t-shirt
Where did James buy his shirt?
Yes please!
Where can I get James’ shirt?
I’ve not been able to find them. Very cool. Does anyone know where we can buy them?
I’ve been looking all over the Internet. Can’t find it. I would buy one if I knew how to order it. Please chime in if you know where to get it.
Next step; The Fabulous TeslaFab!
Please; Robotaxi . The name 69 , Chinese numerology.
Symbolism , There and Back in English.