Monorepos - How the Pros Scale Huge Software Projects // Turborepo vs Nx
ฝัง
- เผยแพร่เมื่อ 31 พ.ค. 2024
- Big companies, like Google & Facebook, store all their code in a single monolithic repository or monorepo... but why? Learn how to use tools like NPM or Yarn workspaces, Learna, Nx, and Turborepo to scale your codebase fireship.io/pro
#software #js #compsci
🔗 Resources
Turborepo turborepo.org/
Nx nx.dev/
Lerna lerna.js.org/
Google's Monorepo cacm.acm.org/magazines/2016/7...
📚 Chapters
00:00 The World’s Largest Codebase
00:49 Benefits of Monorepos
01:50 The Problem with Monorepos
02:18 Yarn & NPM Workspaces
02:48 Lerna
03:30 PNPM
03:48 Nx vs Turborepo
06:10 Turborepo Tutorial
🔥 Get More Content - Upgrade to PRO
Upgrade to Fireship PRO at fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.
🎨 My Editor Settings
- Atom One Dark
- vscode-icons
- Fira Code Font
🔖 Topics Covered
- Monorepo Pros and Cons
- What is a monorepo?
- Who uses Monorepos?
- Should I use a monorepo?
- Nx compared to Turborepo
- How Google's Monorepo works. - วิทยาศาสตร์และเทคโนโลยี
Finally, I can have a giga-scalable monorepo and build tools for my 100-line Discord bot written in Python
I was actually thinking of using this for my Discord bot lol. It has a REST API to manage a roster of members in a DB and I wanted to make a website to interact with the database directly as well. Except in this case, the Discord bot would be a tool for the entire application
Hahaha. Yeah context is VERY important as to whether this is the recommended route someone chooses, for highly practical reasons.
***written in a single Python file
And 40 of those lines are imports.
This is literally most of my uni coding experience
This is the way
Software teams then: ‘This codebase is too coupled together… let’s break it up’
Software teams now: ‘This codebase is too broken up… let’s couple it together’
When the moores law died (as in single core SISD performance, cuz its too hard to make parallel code), code performance became important again
That one guy on the team : "... this goes against everything that I know to be right and true ..."
At the risk of sounding like I don't get the joke; a little context for the uninitiated:
We're still decoupling as much as possible, we're just doing it in a smarter way which allows us to make simpler use of a single source of truth. (1 monorepo, but still many micro-services)
do then undo is work
Google's codebase: moaaaaai
This is by far the best channel to get enough knowledge in order to to make it sound like you know what you're talking about.
@@mikkelens Yea I've spent enough time in front of online courses during the various lockdowns to know how demanding it is. I've spent countless hours on React and I still feel like I don't _actually_ 'get it'.
I like these, more for entertainment than learning. My preferred learning style mainly involves diving in, writing code, and looking at documentation.
Fake it till you make it!
@@noskillzdad5504 i do this when I'm manic lol. Sometimes it's good. Sometimes not
@@reed6514 It's good to discover things and know if you want to dive deep into them, like if a video showcases a new framework which has some particularities you can't live with, if you were able to spot them in the video it may save you hours of reading the docs.
Same for this one, most people probably went to the conclusion that Monorepos weren't yet a fit for their projects.
These videos are handy in a field with so much (sometimes useless) novelties
You explained TurboRepo better than the creator himself, huge respect to Fireship 🎩
At this point, all software should feature a Fireship video in their mainpage
@@andresramos7965 I couldn't agree more, they should pay this guy to make more understandable explanations.
@@oussamasethoum1665 I already pay him to explain stuff to me 😭
@@oussamasethoum1665 Maybe you mean WE should pay him
You're comparing apples to oranges here. Engineers are nutoriously bad at explaining their own work. Fireship is a seasoned technology communicator.
What people call best practices in software engineering amuse me.
Backend: Everything is a microservice
Frontend: Use micro frontends
Git: Put everything in a giant monorepo
The thing is about performance and not everything going down at once, where you store the codebase doesn't matter
Infrastructure and code have different requirements
no! its all about decoupling. microservices to enforce a api and maybe consider an integration layer.
and you split your frontend because you dont want to compile for minutes at a time and have a 100gb node module installed for fixing a a hardcoded link in a selfservice tool which no one seems to use anymore but people manage to complain about somehow
@@matheusvictor9629as linus torvalds said:
Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it
Thank you for covering this. Recently vercel acquired turbo repo and I was confused what was it as I use vercel a lot.
As somone who works at a certain large software company with proprietary source code manager, repository and build tool I can tell you it is no fun at all. It is a nightmare dealing with dependencies when you don't have Maven or Gradle.
Sounds like someone has a case of the NIH blues.
"Maven or Gradle" Look with Gradle I can't comment but with Maven? are you serious? Do you know how complicated it is to simply add a project and the problems it gives looking for its dependencies? Sometimes you have to solve the problems yourself by hand (looking at the dependency tree, and that I don't even talk about projects with Wildfly is a horror) and it makes me lose time from work when all I wanted was to install a library
would this proprietary build tool happen to share a name with a certain south american country
@@bobobo1673 Be thankful that atleast it downloads the versions you mention in pom. The build system I work with needs a kind of dummy project just to deal with dependencies. It then does some black magic, pulls the wrong jars in a secret location and then shoves it into the cryptic buildpath directly during environment level global build 🤣🤣
@@not_nerp Maybe....not
Another fantastic video, just one nit pick: it wasn't super clear (to me) early on that this was about repositories for web tech specifically. When you started out with 2 billion lines of Google code I had their C++, Go etc code in mind - a major driver of Go's toolchain design was long C/C++ build times on huge code bases - but Turborepo is js/ts only.
Using build tools like maven/gradle and scripting I found is the best way to go for multiple projects of different programming languages
Thanks for pointing this out, I completely missed this point. It's somewhat ironic that a tool written in Go to manage repos doesn't actually support Go. Does NX or Lerna support a larger set of languages?
@@tmthyrd Lerna is based on npm, so probably not.
There is only JavaScript 😂
But Bazel that google use does actually handle multiple codebases like Java, C++, Go, Android, iOS
Having recently left a company that was built around a monorepo, here's my takes:
Pros:
* Visibility - it's easy to see where things are used and discover stuff.
* Uniformity - no need to manage versions between things used internally.
Cons:
* Upgradability - you can only upgrade something once *all* of its users are upgraded.
* Maintenance - if you don't invest enough talent into setting up and maintaining your monorepo tools, it's easy to get to a point where they are no longer usably fast.
* Speed - if a simple `git status` command takes multiple seconds to complete, nothing in your project will feel snappy.
I've actually started using monorepos for a couple of solo projects, due to the ease of packaging and dependency sharing
I scouted the entire internet a few months ago to get the exact same monorepo build setup as I'm used to at Google, but here you are, who explained it all so beautifully in a video under 10 minutes. Bless you.
how long have u beeing coding before getting into google? is it still worth it?
Just few days back , i saw the Fireship comment in the TURBOREPO launch video. Now we have a complete tutorial on it...
Thank You dude
I was waiting for so long to finally see this uploaded. Thank you!
Another issue with monorepos is the access management. Especially with the public available tools, you cannot easily define what directories one should have access to.
Doesn't github's codeowners do that?
@@devnol I know that they can limit access to specific repos, but not directories inside a repo. Don't think they can limit your access to specific directories inside a repo (unless I missed and not aware of something)
@@devnol Codeowners can limit write / merge / PR rights. But not read access.
Aside from fringe cases (like fraud rules), why would you want to manage read access?
@@muizzy For example if you use freelancers or outsourced development for the frontend or for integrations, but don't want them to see your proprietary backend algorithms.
You have no idea how helpful this video is and how much time it saved me. THANK YOU.
I'm probably really late on this but congratulations on reaching 1m subs!! I watch daily and always learn a new thing or two. I hope that you keep making amazing content!
I feel like your tracking my Google search wow. Been researching monorepo since the weekend and now you post this 🤣 . Love it
Thank you for the video I was confused when vercel acquired turbo repo and what it was, this cleared up a lot of things
So much information in one video. Huge respect. ❤️
turborepo is great. I just want a demo where everything is done from scratch instead of the boilerplate to better explain it.
me too
That is my only problem with Fireship. Everything else is great.
their docs have a good step by step process of this and their examples help a lot
Man your channel is pure gold.
ive been feeling the need for a monorepo for quite some time. this is helpful.
You are really doing some great explanations! How about a video about how to plan software and what tools to use? Would love to see it👍
That’d be pretty cool actually
I have been setting up a typescript monorepo for one of my personal projects ... And lets just say there are some pains you are gonna feel that you wont forget.
.. cant wait to get through these pains in my next vacation.
You're videos are great. Congratulations for your 1M subscriber mark. 🎉🎉🎉
I'm loving the frequent uploads
0:30 I like that the codebase is elvoving blindfolded _and_ being pulled side to side by different teams _and_ will irrevocably topple over with crushing force after just one pull too hard or in the wrong direction. Great metaphor.
I was searching for this to decide if I should use this approach or not. OMG either you can hear me or you have amazing skills to access my search history
Your videos are my non-guilty pleasure, I always leave having learned cool stuff and enjoying it, it's like having celery for dessert.
You just give me an idea for a personal project, kudos... this is a nice video to watch.
Top quality as always 👌 waiting for your video on webpack module federation, another hyped tool on the market
Your videos are spectacular, congratulations in advance for 1M
you saved my life by explaining how monorepos works. Thank you so much.
Would love to see a video covering Micro Frontends. Having said that, always a big fan of the quality of videos. 10/10 college degree material right here. I should start adding fireship io under the education section of my resume now.
Second this!
I work with a micro frontend every day (though admittedly on the backend). Do you have any specific questions? I may be able to answer them.
@@muizzy Do you think micro frontends improved your development experience in you project? How did micro frontends improved your team workflow? Can you talk about some trade offs in comparison to currently traditional frontend architectures?
Watched "Java in 100 seconds".
- Proficient in Java
Would check out Jack Herrington’s videos on this (e.g., th-cam.com/video/lKKsjpH09dU/w-d-xo.html where he goes through a React example in detail)
you literally read my mind on the video I wanted to see. Incredible!
Frikin love you man! It's been 84 years. That was frikin hilarious bro. 💪❤️👍
Serious question. Why would I use this instead of NX's built in CLI tooling to run the "affected" aspects of my code base. I use NX for both of my side projects
I'm interested in this as well.
Turbo would probably be used instead of Nx, not alongside it. If you're already using Nx, turbo won't do anything for you that Nx doesn't do.
Great video Jeff, really wanted to see this.
So two questions mainly. How do those tools help solve the massive git stack of changes and vscode to keep up with them, and do nx and turborepo override lerna? Or is lerna supposed to work in tandem with any of those? From what I can read in turborepo's site it seems like it can do all that lerna does?
So much to understand in this build flows .. thanks for sharing it..
I've got 3 apps, possibly a 4th coming if i finish the current project soon all along the same theme, i might try creating a monorepo for them to share some of the common functionality. It's most likely overkill at this stage but each app was to learn something, so i may as well do it and see what i learn
Worked in both mono and traditional repos. Monorepo removes so many dependency and versioning headaches.
It can go to an opposite way. 1 app only supported old version while the other uses the new
THANKS! Great video! I have to test this now on my monorepos. :D
Amazing,
I think I am trying Turborepo soon! Thanks for your videos!
I like how the narrator sounds like text-to-speech software...it's one of the many appealing aspects of this channel.
i knew this was coming when i saw you put "nice" on Vercel's stream on Turborepo :D
Thank you for this. It's what's I need right now
I usually only use Monorepo on things like Microservice based systems. I kind of feel like having one repository for ALL your products kinda is overkill. But I guess not, after watching this.
What's overkill is going through the effort of maintaining different repos for a small project.
If your web app, your mobile app and your data processing jobs all use the same data representation, why not store everything in the same repo and keep everything in sync?
Please build a course around it (mono repo) with all the tools you have mentioned.
take 1 big project and architect the same in all tools like nx, turborepo, lerna, yarn, blaze, webpack module federation etc
Time to use turbo repo for my todo app.
Amazing & concise explanation.
Just need this！ Thanks so much for the explanation
I wonder what are the main advantages of using Turborepo against the standard yarn workspaces? We are using the latter, but we will move to a new tool if we see obvious benefits.
We only run one app at a time during development, so the process doesn't seem to differ much from the one described in the video.
We use nx at our firm, it’s really cool and definitely helps maintain our mono repo. 😀
🙌
@@middlefloor Why are you raising your hands? Are you catching a basketball?
@@skyhappy yes!
@@middlefloor well this is a tech channel not a basketball one
@@skyhappy Jeffrey Cross is one of the maintainers of Nx...
Awesome vid. Good idea to build something like this for flutter as well
Melos?
@@chinmaykabi We use melos but it's not a build tool, it's more accurately defined as a "script coordinator". You're bundling commands for multiple packages under a single command. No caching, no smart rebuilds, etc. It will always execute all commands that you defined.
Get the Linux DX with remote containers ;) you can even clone the repository to the docker instance to get the speeds
Our man Jeff becoming funnier and funnier (and informative!) every video. Way to go Jeff, love you
I just converted most of my company's front end code to Nx. It's been a game changer
Awesome to hear!
"You might know him with packages like Formik" Telling me that after 3 weeks of trying to solve issues with fomik and eventually having to refactor my entire code base to use react-hook-forms (which is MUUUUCH better) makes me wanna ditch Turborepo even if I know nothing about it hahaha Cheers for the video!
Formik has its own issue but it created the whole form library ecosytem
That's literally what we've been doing in the last couple weeks haha
Thank you so much for this content!
too much of information in just 10mins that's why I love you
you should always think big and leave an extra space in your app (if possible) so it's growth-friendly
great vid, as always!
you should always think small, and iterate it to the better.. otherwise you end up in perfection and you need a lot of time instead of just shipping a lot of small parts
Keep it lean and modular
Me finishing after 3 days my MonoRepo setup. 2 hours later, this video got uploaded….
Sir you are just awesome pro pro level knowledge.....
The weirdestthing for me is seeing big companies modify distributed version control systems so that they operate more like centralized version control systems. All the problems with git scaling is from git's own setup. Really glad I no longer work at one of the big companies mentioned here. Central version control systems manage monorepos so much better in terms of speed and out of the box access management.
And this particular company's proprietary build tool had such poor documentation that it was almost tribal knowledge when you needed someone to setup the build pipeline or fix a bug.
Great video!
You’re amazing! How can you put out such high quality videos so quickly?
His amazing intelligence and work ethic
My man fireship has a brain so dense, that rumors says in the center there is a blackhole
Monorepo coupled with almost perfect IaC is a god send, it gives amazing visibility to everything. Yes it is complex, and the on-boarding learning curve will be steep, but once you are familiar with it, you don’t even need to leave your IDE to find anything.
Quality video that everyone should watch
Thanks Jeff, this video was very cache
One thing I felt it went missed on this video was about the easy integration with microfrontends and testing tools with Nx.
Hi, great content!!!
Do you have any video where you explain the differences between monorepo and microfrontend? and if it is possible to use both?
Monorepos are great but how to control visibility of the code & access to the code if everything is in one repository.
Like some departments are not allowed to see specific directories/code & things like that
Yes, I was also thinking about that. Everyone can see everything
Is there anything similar for when you have to mix languages? E.g. if your back-end isn't written in JavaScript.
Thanks for fulfilling the request
Had my eye on Turhorepo, didn't know they went open source!
perfect, now I just need to start a big project. maybe next weekend. yeah.
I was so stoked on this but when I tried adding a “ui” package to the default starter I wasn’t able to get it to work. Only the default “Button” component worked across projects. Which was a bummer. I wasn’t able to find any docs on how to add one. For turborepo
You talked about microservices in that, can you do a video on it? That sounds really interesting
Been watching for ages, but not subscribed. I don't know how!
Can one use turbo with other languages (Java, PHP, Python, ...) as well, or is it just for JS projects?
Hey, I used that in creating a blockchain 😃 Thanks
Funniest thing is that today I was thinking exactly about monorepos & multi repos. Fireship reads our minds.
@Fireship, How do I add a new project in this monorepo? Does it have to be configured manually? What if I want to migrate my existing repos here in this pattern, what are the steps I must follow?
YES finally monorepos are getting more coverage
Cool - like Gradle!
Except that build caching sharing - that sounds like fun
Awesome video, thanks for the info. Do you know if there's a similar tool for Java codebases? or, I don't know if it makes sense, heterogeneous code bases?
Gradle's probably the closest Java-first tool. Most of the tools in the video can be used for multi-language codebases, but Bazel and Nx have plugins that make it easier to work with different technologies/languages.
i'm a php dev with about 10 foss packages, many depending on eachother, and composer (php's dependency manager) works great for me. & I think gitlab has a built-in feature for mono-repos, though idk how feature rich it is.
Okay, i have quite a few more foss packages than 10 ... but i estimate only about 10 of them are useful ... i just open source most stuff i write.
what do you use to highlight your cursor?
Setting up a monorepo for all my unfinished projects
Thank you for the video :)
I have worked with monorepos built with lerna. OMG are they difficult to maintain!
Best tech TH-camr on the planet yo
I knew it coming, since you starred the repo 2 days ago!
Really interesting, thanks
Nice explanation.
Let's get fireship to the moon 🚀
would these problems that monorepos have just be solved by creating different repos and some dockerfiles? instead of having to use all these different tools?
thank you so much
Excellent video!