The company I work for uses monorepos. Most of the time it's a blessing as I am able to read frontend code and check what the React guys are doing. Our code bases are not highly complex though, so there's no need to project encapsulation. Great video, as always!
I've found a mix of both could be also a good solution, here an example: github.com/mateodelnorte/meta I've used it to gather all projects in the squad or all repos related to a topic (infra).
I have used both... Mono repos are ok for small tightly coupled projects... as a rule they are problematic... An automated K8s infrastructure deployment can be a good candidate for such a mon-repo... In my last project I was the architect and lead software engineer and made that decision. We used a Mono-repo (mostly) for our K8s infrastructure automation that was rather large, however we had about 10 sub directories in the mono- repo for different types of functions or pieces of the infrastructure (logging verses, SecOps, verses provisioning, etc.)... We also had the code call other repos for a few peices of the build, some via Git submodule, some via chg dir Git checkout and run. This worked fairly well as the developers (DevOps engineers who were all writing code) did not step on each other's code... Our complicated build took 14 mintues to build and deploy a very large and K8s infrastructure with automated testing. Then we would delete test infrastructure and Tag the successfull code. I tested and approved almost all Merge Requests (PRs), occasionally I would just test them after others had merged their code... (I was pressured to allow others to merge code) I tagged all versions that successfully built and required that we deploy from a tag only. However, when I moved to a different specialized sub-project on the same team, my replacement(s) did NOT read and test the code... They abandoned tagging because my configuration managment rules were considered too difficult.... The standard was EVERYDAY at the beginning and the end, all developers were to run a script... Git status, Git stash, Git Pull, Git checkout Main (Master), Git pull, Git checkout you feature branch, Git merge main (Master), Git stash pop... Took 2 -3 mintues to run... Merge Conflicts would be SMALL... and were rather rare actually... That was considered so difficult (Draconian) to run the Bash script twice a day... Within a couple of weeks of me focussing on a different off-shoot project, the repo was trashed. The master branch diverged and couldn't be built.. . A Release branch had to be made and it tooks months for them to fix the main (master) branch...This lack of discipline wasted a LARGE Amount of developer time (money)... but that is what they decided to do... A Diaster project... On my newest project (which I worked on three years ago) , The team decided on a Mon-repo for both the K8s infrastrucutre AND the 38 Charted applications that are developed by 26 different software development teams ( 120 developers total)... The Charts (not including the custom Docker containers) when rendered (into a compiled tar.gz file) are 54K lines of code and deploy via the "Replace Method". This is a Diaster... A working build takes a week or more to put together and fails often... I am currently taking their diaster build and separating it all out into individual char repos AFTER they make their conglomerated mess via automated Helm Rendering adn pushing to seperate repos). The other team is separate and the client hired us to make this suite of applications work correctly ( A Ci/CD Process taht takes a week or more to codify changes and build with lots of problems is not CI/CD)... I am scipting the rendering of all the charts into loosely coupled applications in seperate repos and checked via Tag so deployment will be automatically handled by ArgoCD...
Rightly or wrongly we went one step further and employed multiple organization namespaces based on function for example: frontend-org, backend-org, infra-org, security.org, sso.org etc. All our apps are loosely coupled microservices. We have people, friends and acquaintances coming and going , helping us out (self bootstrap'd start-up) so we just provide access to the specific org's and repo's that they require. It also works quite well organising in vscode :-) BTW, sent you Linkedin connection request. edit: One of the smallish issues that come to mind in our case is the requirement to manage multiple deploy keys for each repo, in Github. But this also introduce security a benefit of polyrepos over monorepos. If someone unwanted gains access to a monorepo they pretty much have a complete view of your business application. If they gain access to a polyrepo (a subset of the total system) they most probably will spend an eternity scratching their head. ;-)
This video might be 2 years old but it's still very relevant, and I found it very helpful. It expresses all of the concerns I have around mono vs multi-repo, but I couldn't quite put into words as you have so well elucidated. Thanks so much for making this.
Many of the problems you describe happen when working on loads of applications in the same project. I'm using monorepos for a single app with its backend, frontend etc and I've found it really helpful. Then again we are a very small team. Another problem I've heard is that CI build needs to build everything as it cannot detect what projects changed. Though this could be good for spotting regressions relating to build quickly.... maybe? As you say all of these issues can be mitigated but it's important to consider them. Thanks for the video, very informative.
I think you're spot on here, I'd just call out that not everyone uses the Microservices architecture, so I think it's key to mention that the repo strategy you choose should take your chosen architecture into consideration. Regardless of architecture, a good separation of concerns/related functionality should be universal, regardless of whether that's at the repo level or not.
Oh yeah. There is no single solution or approach that works for all cases. It's all about understanding the "big picture" and be able to adapt depending on specific scenarios. If any of those approaches matches needs of many, you can still choose a different one for the needs of a few.
I disagree with a few of the downsides presented here: 1. Difficulty in discovery. The idea that it's harder to find stuff in a single place seems off. Whether your code is in one repo, or multiple, you will still need someone to explain which repos / [directories you do and don't need. The fact that there might be millions of lines of code in a part of the repo that doesn't relate to you shouldn't matter if your team lead / hiring manager tells you which parts do relate to you. In fact, I would argue that with poly-repos it can harder to discover that you're missing a necessary dependency. All you know is your build fails, but figuring out that the build fails because you're missing another repo, rather than an external package is an extra step. 2. Searching. While it's true that you can use built-in tools in various VCS's to search through code, those search features are often very half baked. When you have a single repo it's trivial to do complex searches using regex, exclusions, and scripts, and extra processing layers. If you're using a web interface, then you only have access to the features that this web interface offers. If you need multiple processing steps, forget about it. Granted, you could just clone all the repos into a single directory and go from there, but then you're just re-creating the structure of a mono repo from parts. 3. Dependency building. If you have a project with extremely long build time, you can still have an external dependency registry for some dependencies. 4. Refactoring. With sufficiently advanced tooling, you can do massive refactoring using context-aware programs. Generally any time I've done massive refactoring pushes, the cause was something along the lines of "we're upgrading the language / core module / framework used by a lot of systems" or "we've adding an additional static analysis step, and we need to fix a huge mistake we've been making across the entire code base." Granted, frequent refactoring of all your smaller projects is certainly a code smell, but on the other hand not all cross-project refactoring needs to be universal. I've frequency seen smaller refactoring efforts that touch an interface layer between two distinct projects. That sort or refactoring effort isn't necessarily a bad thing. We don't want dozens of systems depending on each other, but having a few closely related systems communicating is generally a desired state of affairs. Granted, and API should be versioned, but the idea that every service should be a wholly distinct thing that we should leave along is a very contractor-centric type of thinking. When an executive comes along and says that we need a new feature next month, they usually don't mean "we want 25% of that feature next month, then another 35% the month after, then another 10% in 2 weeks, and the last 30% the following quarter." I understand that's how a lot of consultancies work, but that honestly usually only creates additional friction between staff developers and executives when the employees of the company have to explain to the execs that the new shiny feature they promised to their clients only has partial back-end support, and the only reason the story is marked "completed" is because the people implementing this feature didn't have any insight or visibility into another critical service. Also in terms of the disadvantages: 1. Slow cloning. This one is correct, but only really for the initial clone. Realistically as long as you have a fairly up-to-date version of the repo, you should never be in a position where you pull and have to wait for longer than a few seconds. If you do, then most likely someone is using the repository wrong. Additionally, this sort of problem is inherently one that's going to happen only in *extremely* large companies, who are likely going to have large teams to mitigate it. If you don't have those sort of resources, then chances are good you're simply not going to have that large a repo. It's a classic chicken and egg scenario. 2. Configuring change tracking pipelines. If you have a smart repo structure, this is trivial in any CI system I have used. In fact there is extensive tooling and support for this in open source. Beyond that; as much as you don't want a universal build on every commit, the simple fact that you *can* fire off a universal build fairly trivially is a major advantage. You can set it on a schedule to run once a day at night, or once a week over the weekend, and get an update snapshot of overall system health. What more, as projects are added, you only have to modify the one global pipeline. You can even set up rules to validate that new projects update these configurations. This is much easier than having to track down multiple distinct configurations, which all reference distinct repositories, and must be update as part of an update process which must be documented and executed by god knows what team. 3. Pull requests. If you want to direct PRs to code owners, most VCS's have the concept baked in. This is arguably less work than adding new users to multiple repositories, since you can generally configure this in one place for all repos. If you're getting notified about all the changes in the system, that really means you just haven't set up your monorepo to deal appropriately with a large system. Basically the entire video seems to be focused on the idea that if you have huge, multi-repo organization, and you want to move to a monorepo layout, then you will need fundamental changes to your process. This is a perfectly valid point, but it's sort of different from what the title of the video seems to suggest. If you're starting a major project and you need to decide between multirepo and monorepo, then you will need to set up a lot of tooling either way, and that tooling will require an extensive time and money investment no matter the choice you make. If you're thinking of moving TO monorepos, then sure, all the points make sense, but that's not really a fault with monorepos. That's just a problem with large, multi-project refactoring efforts. I just take an issue with presenting a video about the issues with monorepos, and only mentioning at the end that the entire thing is about *switching to monorepos*.
Can you expand a bit this: " If you have a smart repo structure, this is trivial in any CI system I have used. In fact there is extensive tooling and support for this in open source." how would a smart structure be and what tooling do you have in mind?
@@berduel A smart structure is basically any directory structure made with the entire codebase in mind, and kept consistent as new projects are added. This is in contrast to an ad-hoc structure where everyone treads the codebase as their own personal playground, and adds files and directories wherever they want following their own personal conventions. The latter leads to lots of arguments, and makes it impossible to find anything that didn't come from you or your team. Your goal is to have well defined rules about where to add new code, new projects, shared libraries, shared assets, and so on. That way there's no need for debate or for creative solutions when you want to add something new. You just open your rules, and follow the guidelines, and if you see a PR that doesn't follow the guidelines you can just reject it and say "follow the guidelines plox." This in turn will let you have a single, centralized set of configurations which will let you resolve paths to projects, and which in turn can be used to launch per-project pipelines from one repo. As an example of configurations tools you can use, if you're using GitHub Actions there's this one which I use in my active project: github.com/dorny/paths-filter. I have done something equivalent in CircleCI, Azure Devops, and Jenkins though I'd have to go digging for those, so I will leave that as an exercise to the reader.
Sorry for not responding earlier. For reasons I do not comprehend, TH-cam moved your last message to spam so I did not see it until now. I fully agree that rules about organizing repositories, directories, files, etc. is very welcome. Knowing what is where helps the people in a team work together. The part I dissagree is that "in turn will let you have a single, centralized set of configurations". Having guidelines and following them does not necessarily mean that those guidelines must result in centralization. Sometimes that is the case, while in others it is not. In big part, that depends on team topologies, the way of working and cooperation, etc. What I'm trying to say is that guidelines are a must, but they do not always lead to "a single, centralized set of configurations".
@@DevOpsToolkit That's fair, just having a centralized source alone isn't enough. At the minimum you will also need well documented style and organizational guidelines, and likely a lot of effort to correct PRs before the rest of the team fully internalizes the structure. Unfortunately there isn't a single, universal panacea that will let you manage an effective development team. There's a myriad of approaches, each of which suit different types of teams, staffed with different personalities. The key part is that without such structural and organizational elements having a monorepo is likely to lead to a lot of headaches. So I suppose it would be more correct to say that if you want to have a monorepo you will at the very least need centralization, and like a lot of extra work on top of that. However, if you don't put an emphasis on centralization then trying to keep all your code in one place will be a nightmare.
We have a monorepo at work from before I joined. Not all of company's project are in it but a bunch of related applications are. The dream of single PR for changes in multiple apps is just a dream. Even in our monorepo, I see two PRs for the same feature change - one by the frontend dev and on by the backend dev. I am not bothered by it, in fact I prefer it that way as I review backend code with higher rigor. Other than frontend-backend code, generally projects should be very loosely coupled and in that case a single PR makes no sense. So the advertised benefit of single PR across projects will be useful only when you are doing things wrong i.e. when you have tightly coupled code. The commit noise is just too much in a monorepo. Like you said, it is difficult (or not possible) to assign commit or PR rights to individual projects inside a monorepo. So the first thing I have to check in every PR is whether the dev is making modifications in projects he/she should not be touching in context of that PR. This problem would never occur in a polyrepo. When you hear someone praising monorepo, ask what kind of apps they are building. Monorepos supporters are usually working with monoliths or distributed monoliths where all apps are updated in lock-step with each other. Such projects have no need to solve for situations where each app/microservice can be deployed at different times, In my case, we have several microservices in a monorepo. Monorepo are a bad fit for Microservices. I am managing it by creating tags for each version of a specific app.
A point to Dependency Management : with using monorepo we have some tools which can auto bump-up the dependency version. While with polyrepo you have to do that individually.
Very interesting video, as usual :). I've had a similar argument with a relatively unexperienced team where developers were in favor of monorepo. My main concern was that it will encourage bad practices and thigh coupling in microservices architecture (which at the end it did!). Once very interesting aspect you've mentioned is API and versioning. A typical (micro)service development process has business logic and externally facing APIs in the same code base deployed together. Understanding that APIs must be versioned and backwards compatible encourages using patterns like API management systems, gateways, etc to allow decoupling of the backend code from externally facing API. Complexity in CI/CD pipeline is even more important with cloud native applications and distributed systems.
Awesome video! I agree on every point but I have understood these things through many difficulties. If I had found this video earlier it would have helped me tremendously. Your presentation is clear, precise and methodical. Thanks for your contribution!
The last place I worked at, we used polyrepo approach with a central repo having submodule repos. While the parallel development went easier with this approach, the code organization became messy over time with a lot of submodule commit versions to track. After researching more, I now believe monorepo approach (though it has its own downsides too) would be more suitable in scenarios like this.
Hi Viktor. Thanks. I always appreciate your style of explaining things. Some important points to think about when it comes to the organization of software projects. I like that you always keep the balance of black and white. As you said, it's almost never black or white. It's mostly grey.
My biggest pain point has been access control. Since the apps and libraries in a mono-repo are tightly coupled with each other it is difficult if not impossible to give separate access to people.
I just about coughed up my cereal at the end when prompted to subscribe lest I get notifications of any change to your monorepo hahaha. Had I not already been subscribed, this threat would have served as sufficient call to action for me to subscribe. Awesome channel, you are a walking encyclopedia.
6:55 exactly! Your deployables should be mapped 1:1 to repo. Refactoring in a monorepo is harder, and the scaling makes this harder. Instead of changing the API of one library/service and then changing how its dependencies use it, you need to make all of those changes at the same time, slowing down your entire org while the change happens.
I enjoyed this video a lot because I am someone who is against mono repo to build microservices! I open a folder and use a git branch for per microservice in my single repo as main branches of those micro services and from those sub main microservice branches it becomes very easy to work and build pipelines and also maintains a single repo for each microservice project! Branched based poly repo strategy is very useful but don't know why many famous tech company like google and fb don't use it.
First of all, I'm a regular viewer of your channel and love your videos! Even though I disagree with a lot of what you say in this one :-) Just out of curiosity: how much experience do you have with monorepos and the related tooling? Now for my comments... Being able to search the entire codebase is indeed easier on a monorepo. Not sure I understand how the presence of stuff you aren't always interested in makes searching harder though. You can restrict your search to the folders you're interested in. Cross project atomic commits are as least as big a benefit to collaboration I find. For example, if you work on a service that requires backend and frontend changes, they can be included in the same PR. This reduces the chances of breaking changes and helps you to staying end-user focused. The problems you describe around dependency management aren't monorepo specific I think. In a polyrepo setup you would also need to rebuild everything that depends on reused code. The dependency tree you describe exists in a polyrepo situation too, unless you decide not to reuse any code or APIs, but that doesn't like a recipe for success. Yes, you need specialized tools to describe and make use of the dependencies, but for a monorepo those tools exist at least. Indeed, if you frequently need system wide refactorings, that is an architectural smell. And since monorepo makes the use of internal libraries easier, it requires extra discipline not to end up with a distributed monolith. The basic rule for internal libraries is: don't share domain specific classes or logic, only horizontal technological layers. Services should talk to each other via APIs not via libraries. But I have seen distributed monoliths in a polyrepo setup too. This was a result of sharing internal domain oriented libraries, which is just as possible, but with additional versioning headache. So while I agree that frequent system wide refactorings are a smell, they may still be necessary to get out of the bad situation. And within a monerepo you can at least *consider* doing them. WRT the disadvantages: - Slow git operations: this really depends on the size of your codebase. In my experience this is not an issue on medium size (maintaned by say 50 engineers) codebases. Incremental pulls only take seconds, unless somebody has comited binaries, which shouldn't be done anyway. For pristine CI builds it _can_ take some effort to keep the clones fast enough: cached snapshots, shallow checkouts, etc. - Encourages tight coupling: agreed. It makes the use of shared libraries easier and therefore also the abuse of shared libraries. However, disentangling tight coupling is a lot easier in a monorepo too. - Pipeline issues: I disagree pretty strongly with this one. For very simple projects, with services that can live in splendid isolation pipelines are simple. But that's not what we are talking about. As soon as you have services that depend on each other, you need to orchestrate your CI. This is virtually impossible to do with polyrepos. On a monorepo master can be broken in different ways. If any of the leaf projects is broken, this does not affect anybody else and doesn't block development. If a widely used library is broken, this will stop the work, as it should, if you want to practice continuous integration. Monorepos will make it much less likely to happen though. Imagine publishing a shared library in a polyrepo setup. How do you even know if it breaks any of the clients using it? With respect to the noise monorepos create, that can easily be solved with a folder structure, code owner files or automated folder based PR labeling. Yes, I am in the monorepo camp as you may have gathered by now :-) If I have to summarize the polyrepo vs monerepo tradoffs in one sentence: while there are a few bad practices that monorepos make easier to do, they illuminate architectural problems that tend to stay hidden with polyrepos while at the same time making it much more possible to address them.
I like the idea of a monorepo per team, but the freely available build tools for Typescript have not supported that. (Are there any new developments on the horizon?) The best supported cases for monorepo are for collections of related modules, such as modularized libraries, e.g., babel, or apps consisting of a core plus a bunch of *related* plugins.
I try to avoid monorepos as much as possible, and instead clearly separate my project into components, each of which then lives in its own repo. I think that in the past it might have been easier to pack everything into one file tree and figure out separation of concerns for yourself, but nowadays with the open-source ecosystem on GitHub, modern CI/CD tools and containerized applications that rely on a multitude of programming languages, polyrepos are the simpler solution. The one major downside is that there are more steps involved in making changes across components (say you want to introduce a new feature to the API and simultaneously develop the UI that consumes it), but then again linking dependencies locally ain't that hard and the worst you'll have to deal with after that is coordinating a few PRs at once (so that the dependencies will merge and build before the depending components). I think that tradeoff is waaayy better, especially as I expect the tooling for cross-repo work to improve a lot in the near future. PS: Great video! PS2: I think we are supposed to call it the "main" branch nowadays ;)
I agree with all the points. The upsides of using monorepo are easily overshadowed by the downsides. I'm doing my best to call it main or mainline but `gh` is holding me back. Whenever I create a new repo with `gh repo create`, it creates the `master` branch instead of `main`.
Nice video, Victor! In our project, we got similar conclusions in our discussions about monorepos vs polyrepos. I think monorepos are more suitable when you are working with a monolith. When you are working with microservices makes more sense to use polyrepos.
Those are my thoughts as well. If I do not have any additional information and would make a decision only based on architecture, I would say that monoliths and distributed monoliths (faking microservices) are better suited for mono-repos, and microservices for poly-repos. It's not always that easy though, but that could be a rule of thumb.
I can't agree with dependencies&monorepo thing. You should always build everything. You should always make sure that "that small dependency" does not break everything. You should test everything after the smallest change as well. Yes, it takes time. But if you want to deliver high-quality and stable products you need to do it.
I use monorepo for infra and monorepo for serverside code, and monorepo for frontend, and etc. Is it multimono? :) But I doing this comment just to say that what is the nice place you are sitting!! Clean, silent, adorable... Love it
If you have a repo that contains more than one project (more than one app that is built and deployed separately), you're having a monorepo. One can have one monorepo for everything or multiple monorepos or a combination of monorepos and polyrepos. As for the place where I recorded... It's beautiful. We go there at least a couple of weekends every month.
The biggest issue I have with it is that project managers hear about google using it and try to use it for a couple of small projects. The devs end up trying to figure out how to solve version conflicts in dependencies of dependencies of dependencies of dependencies and why nohoist seems to be doing nothing at all. Combine that with the company want their devs to work on a trendy mac so npm/yarn takes a hundred times longer to complete and you have a team not programming anything but instead tinkering with lerna and yarn workspaces instead.
The problem with "Google does it" is that Google was not doing it always but evolved over time. Even if the "Google way" is indeed something someone should take, that means not only doing what Google is doing today but also walking a similar path as Google was walking throughout the years before it became what it is today. One cannot just teleport itself into the current state of Google (besides the question of whether one should have it as the goal in the first place).
The only reason why I want to try monorepos at all is because I highly dislike the developer experience you get when working with 'microservices' or should I say distributed monoliths.
I recently had an interview with a Lead Dev of a publicly traded company. He told me they refactored everything the company has into a monorepo. I just thought, boy they should fire you and we'd not be friends ^^.
Thanks, Victor! I found myself at this crossroad a while back and eventually went with Polyrepos. However, what are your thoughts on using git subtrees to manage these multiple repos in a single base repo. My personal experience with git subtrees is that it eventually becomes hell to manage.
@@DevOpsToolkit what do you think about dev containers (containers.dev)? I am using bazel and monorepo for a couple of months, but it doesn't feel right if you have to write custom rules for languages such as Node, TypeScript, Rust, C#...
Thanks! The flip side of easy collaboration across teams is perhaps security (access control) isn't it? Everyone in a team gets access to all the code of all other teams. Depending on the organization and the type of product being developed this may or may not be desirable. For example, Google can't possibly have their top secret search engine algo code openly available as part of a monorepo. So, security should be listed as a potential downside in my opinion.
Why you think there is one master branch? You don't need to have just one branch. There are tools that help build projects and related libraries to them, not the whole project at once like Nx. Nx cloud caching in cloud already built projects. So if you pull it, it will all be built already.
How about MonoPoly? One repo with multiple projects, developers gain access/own to folders based on their ranks/achievements in their development journey :-).
... and if you collect enough achievements, your PRs are merged to the mainline and deployed to production automatically without anyone's intervention.
Are you using a monorepo, polyrepos, or both?
The company I work for uses monorepos. Most of the time it's a blessing as I am able to read frontend code and check what the React guys are doing. Our code bases are not highly complex though, so there's no need to project encapsulation. Great video, as always!
I've found a mix of both could be also a good solution, here an example: github.com/mateodelnorte/meta
I've used it to gather all projects in the squad or all repos related to a topic (infra).
I have used both...
Mono repos are ok for small tightly coupled projects... as a rule they are problematic...
An automated K8s infrastructure deployment can be a good candidate for such a mon-repo...
In my last project I was the architect and lead software engineer and made that decision. We used a Mono-repo (mostly) for our K8s infrastructure automation that was rather large, however we had about 10 sub directories in the mono- repo for different types of functions or pieces of the infrastructure (logging verses, SecOps, verses provisioning, etc.)... We also had the code call other repos for a few peices of the build, some via Git submodule, some via chg dir Git checkout and run. This worked fairly well as the developers (DevOps engineers who were all writing code) did not step on each other's code... Our complicated build took 14 mintues to build and deploy a very large and K8s infrastructure with automated testing. Then we would delete test infrastructure and Tag the successfull code. I tested and approved almost all Merge Requests (PRs), occasionally I would just test them after others had merged their code... (I was pressured to allow others to merge code) I tagged all versions that successfully built and required that we deploy from a tag only.
However, when I moved to a different specialized sub-project on the same team, my replacement(s) did NOT read and test the code... They abandoned tagging because my configuration managment rules were considered too difficult....
The standard was EVERYDAY at the beginning and the end, all developers were to run a script... Git status, Git stash, Git Pull, Git checkout Main (Master), Git pull, Git checkout you feature branch, Git merge main (Master), Git stash pop... Took 2 -3 mintues to run... Merge Conflicts would be SMALL... and were rather rare actually... That was considered so difficult (Draconian) to run the Bash script twice a day...
Within a couple of weeks of me focussing on a different off-shoot project, the repo was trashed. The master branch diverged and couldn't be built.. . A Release branch had to be made and it tooks months for them to fix the main (master) branch...This lack of discipline wasted a LARGE Amount of developer time (money)... but that is what they decided to do...
A Diaster project...
On my newest project (which I worked on three years ago) , The team decided on a Mon-repo for both the K8s infrastrucutre AND the 38 Charted applications that are developed by 26 different software development teams ( 120 developers total)... The Charts (not including the custom Docker containers) when rendered (into a compiled tar.gz file) are 54K lines of code and deploy via the "Replace Method". This is a Diaster... A working build takes a week or more to put together and fails often... I am currently taking their diaster build and separating it all out into individual char repos AFTER they make their conglomerated mess via automated Helm Rendering adn pushing to seperate repos). The other team is separate and the client hired us to make this suite of applications work correctly ( A Ci/CD Process taht takes a week or more to codify changes and build with lots of problems is not CI/CD)... I am scipting the rendering of all the charts into loosely coupled applications in seperate repos and checked via Tag so deployment will be automatically handled by ArgoCD...
Rightly or wrongly we went one step further and employed multiple organization namespaces based on function for example: frontend-org, backend-org, infra-org, security.org, sso.org etc. All our apps are loosely coupled microservices. We have people, friends and acquaintances coming and going , helping us out (self bootstrap'd start-up) so we just provide access to the specific org's and repo's that they require. It also works quite well organising in vscode :-) BTW, sent you Linkedin connection request. edit: One of the smallish issues that come to mind in our case is the requirement to manage multiple deploy keys for each repo, in Github. But this also introduce security a benefit of polyrepos over monorepos. If someone unwanted gains access to a monorepo they pretty much have a complete view of your business application. If they gain access to a polyrepo (a subset of the total system) they most probably will spend an eternity scratching their head. ;-)
Polyrepo and recently moving to mono. Thanks for the video. Great to have experience in both knowing what to use and when to use
This video might be 2 years old but it's still very relevant, and I found it very helpful. It expresses all of the concerns I have around mono vs multi-repo, but I couldn't quite put into words as you have so well elucidated. Thanks so much for making this.
You're shooting location is pure awesomeness!
Apart from the video is really helpful (:
Many of the problems you describe happen when working on loads of applications in the same project. I'm using monorepos for a single app with its backend, frontend etc and I've found it really helpful. Then again we are a very small team.
Another problem I've heard is that CI build needs to build everything as it cannot detect what projects changed. Though this could be good for spotting regressions relating to build quickly.... maybe?
As you say all of these issues can be mitigated but it's important to consider them. Thanks for the video, very informative.
I think you're spot on here, I'd just call out that not everyone uses the Microservices architecture, so I think it's key to mention that the repo strategy you choose should take your chosen architecture into consideration. Regardless of architecture, a good separation of concerns/related functionality should be universal, regardless of whether that's at the repo level or not.
Oh yeah. There is no single solution or approach that works for all cases. It's all about understanding the "big picture" and be able to adapt depending on specific scenarios. If any of those approaches matches needs of many, you can still choose a different one for the needs of a few.
I disagree with a few of the downsides presented here:
1. Difficulty in discovery. The idea that it's harder to find stuff in a single place seems off. Whether your code is in one repo, or multiple, you will still need someone to explain which repos / [directories you do and don't need. The fact that there might be millions of lines of code in a part of the repo that doesn't relate to you shouldn't matter if your team lead / hiring manager tells you which parts do relate to you. In fact, I would argue that with poly-repos it can harder to discover that you're missing a necessary dependency. All you know is your build fails, but figuring out that the build fails because you're missing another repo, rather than an external package is an extra step.
2. Searching. While it's true that you can use built-in tools in various VCS's to search through code, those search features are often very half baked. When you have a single repo it's trivial to do complex searches using regex, exclusions, and scripts, and extra processing layers. If you're using a web interface, then you only have access to the features that this web interface offers. If you need multiple processing steps, forget about it. Granted, you could just clone all the repos into a single directory and go from there, but then you're just re-creating the structure of a mono repo from parts.
3. Dependency building. If you have a project with extremely long build time, you can still have an external dependency registry for some dependencies.
4. Refactoring. With sufficiently advanced tooling, you can do massive refactoring using context-aware programs. Generally any time I've done massive refactoring pushes, the cause was something along the lines of "we're upgrading the language / core module / framework used by a lot of systems" or "we've adding an additional static analysis step, and we need to fix a huge mistake we've been making across the entire code base."
Granted, frequent refactoring of all your smaller projects is certainly a code smell, but on the other hand not all cross-project refactoring needs to be universal. I've frequency seen smaller refactoring efforts that touch an interface layer between two distinct projects. That sort or refactoring effort isn't necessarily a bad thing. We don't want dozens of systems depending on each other, but having a few closely related systems communicating is generally a desired state of affairs. Granted, and API should be versioned, but the idea that every service should be a wholly distinct thing that we should leave along is a very contractor-centric type of thinking. When an executive comes along and says that we need a new feature next month, they usually don't mean "we want 25% of that feature next month, then another 35% the month after, then another 10% in 2 weeks, and the last 30% the following quarter." I understand that's how a lot of consultancies work, but that honestly usually only creates additional friction between staff developers and executives when the employees of the company have to explain to the execs that the new shiny feature they promised to their clients only has partial back-end support, and the only reason the story is marked "completed" is because the people implementing this feature didn't have any insight or visibility into another critical service.
Also in terms of the disadvantages:
1. Slow cloning. This one is correct, but only really for the initial clone. Realistically as long as you have a fairly up-to-date version of the repo, you should never be in a position where you pull and have to wait for longer than a few seconds. If you do, then most likely someone is using the repository wrong.
Additionally, this sort of problem is inherently one that's going to happen only in *extremely* large companies, who are likely going to have large teams to mitigate it. If you don't have those sort of resources, then chances are good you're simply not going to have that large a repo. It's a classic chicken and egg scenario.
2. Configuring change tracking pipelines. If you have a smart repo structure, this is trivial in any CI system I have used. In fact there is extensive tooling and support for this in open source.
Beyond that; as much as you don't want a universal build on every commit, the simple fact that you *can* fire off a universal build fairly trivially is a major advantage. You can set it on a schedule to run once a day at night, or once a week over the weekend, and get an update snapshot of overall system health. What more, as projects are added, you only have to modify the one global pipeline. You can even set up rules to validate that new projects update these configurations. This is much easier than having to track down multiple distinct configurations, which all reference distinct repositories, and must be update as part of an update process which must be documented and executed by god knows what team.
3. Pull requests. If you want to direct PRs to code owners, most VCS's have the concept baked in. This is arguably less work than adding new users to multiple repositories, since you can generally configure this in one place for all repos. If you're getting notified about all the changes in the system, that really means you just haven't set up your monorepo to deal appropriately with a large system.
Basically the entire video seems to be focused on the idea that if you have huge, multi-repo organization, and you want to move to a monorepo layout, then you will need fundamental changes to your process. This is a perfectly valid point, but it's sort of different from what the title of the video seems to suggest. If you're starting a major project and you need to decide between multirepo and monorepo, then you will need to set up a lot of tooling either way, and that tooling will require an extensive time and money investment no matter the choice you make.
If you're thinking of moving TO monorepos, then sure, all the points make sense, but that's not really a fault with monorepos. That's just a problem with large, multi-project refactoring efforts. I just take an issue with presenting a video about the issues with monorepos, and only mentioning at the end that the entire thing is about *switching to monorepos*.
Can you expand a bit this: " If you have a smart repo structure, this is trivial in any CI system I have used. In fact there is extensive tooling and support for this in open source."
how would a smart structure be and what tooling do you have in mind?
@@berduel A smart structure is basically any directory structure made with the entire codebase in mind, and kept consistent as new projects are added. This is in contrast to an ad-hoc structure where everyone treads the codebase as their own personal playground, and adds files and directories wherever they want following their own personal conventions. The latter leads to lots of arguments, and makes it impossible to find anything that didn't come from you or your team.
Your goal is to have well defined rules about where to add new code, new projects, shared libraries, shared assets, and so on. That way there's no need for debate or for creative solutions when you want to add something new. You just open your rules, and follow the guidelines, and if you see a PR that doesn't follow the guidelines you can just reject it and say "follow the guidelines plox."
This in turn will let you have a single, centralized set of configurations which will let you resolve paths to projects, and which in turn can be used to launch per-project pipelines from one repo. As an example of configurations tools you can use, if you're using GitHub Actions there's this one which I use in my active project: github.com/dorny/paths-filter. I have done something equivalent in CircleCI, Azure Devops, and Jenkins though I'd have to go digging for those, so I will leave that as an exercise to the reader.
@@berduel NX, Turborepo, etc... allow you to run a command to see what's affected and only test, build, deploy relevant sub-repos.
Sorry for not responding earlier. For reasons I do not comprehend, TH-cam moved your last message to spam so I did not see it until now.
I fully agree that rules about organizing repositories, directories, files, etc. is very welcome. Knowing what is where helps the people in a team work together.
The part I dissagree is that "in turn will let you have a single, centralized set of configurations". Having guidelines and following them does not necessarily mean that those guidelines must result in centralization. Sometimes that is the case, while in others it is not. In big part, that depends on team topologies, the way of working and cooperation, etc.
What I'm trying to say is that guidelines are a must, but they do not always lead to "a single, centralized set of configurations".
@@DevOpsToolkit That's fair, just having a centralized source alone isn't enough. At the minimum you will also need well documented style and organizational guidelines, and likely a lot of effort to correct PRs before the rest of the team fully internalizes the structure. Unfortunately there isn't a single, universal panacea that will let you manage an effective development team. There's a myriad of approaches, each of which suit different types of teams, staffed with different personalities.
The key part is that without such structural and organizational elements having a monorepo is likely to lead to a lot of headaches. So I suppose it would be more correct to say that if you want to have a monorepo you will at the very least need centralization, and like a lot of extra work on top of that. However, if you don't put an emphasis on centralization then trying to keep all your code in one place will be a nightmare.
We have a monorepo at work from before I joined. Not all of company's project are in it but a bunch of related applications are. The dream of single PR for changes in multiple apps is just a dream. Even in our monorepo, I see two PRs for the same feature change - one by the frontend dev and on by the backend dev. I am not bothered by it, in fact I prefer it that way as I review backend code with higher rigor. Other than frontend-backend code, generally projects should be very loosely coupled and in that case a single PR makes no sense. So the advertised benefit of single PR across projects will be useful only when you are doing things wrong i.e. when you have tightly coupled code.
The commit noise is just too much in a monorepo. Like you said, it is difficult (or not possible) to assign commit or PR rights to individual projects inside a monorepo. So the first thing I have to check in every PR is whether the dev is making modifications in projects he/she should not be touching in context of that PR. This problem would never occur in a polyrepo.
When you hear someone praising monorepo, ask what kind of apps they are building. Monorepos supporters are usually working with monoliths or distributed monoliths where all apps are updated in lock-step with each other. Such projects have no need to solve for situations where each app/microservice can be deployed at different times,
In my case, we have several microservices in a monorepo. Monorepo are a bad fit for Microservices. I am managing it by creating tags for each version of a specific app.
Very clear articulation, thank you!
A point to Dependency Management : with using monorepo we have some tools which can auto bump-up the dependency version. While with polyrepo you have to do that individually.
Very interesting video, as usual :). I've had a similar argument with a relatively unexperienced team where developers were in favor of monorepo. My main concern was that it will encourage bad practices and thigh coupling in microservices architecture (which at the end it did!). Once very interesting aspect you've mentioned is API and versioning. A typical (micro)service development process has business logic and externally facing APIs in the same code base deployed together. Understanding that APIs must be versioned and backwards compatible encourages using patterns like API management systems, gateways, etc to allow decoupling of the backend code from externally facing API.
Complexity in CI/CD pipeline is even more important with cloud native applications and distributed systems.
Super useful pros and cons of monorepo vs poly or multi, thank you!
Awesome video! I agree on every point but I have understood these things through many difficulties. If I had found this video earlier it would have helped me tremendously.
Your presentation is clear, precise and methodical.
Thanks for your contribution!
The last place I worked at, we used polyrepo approach with a central repo having submodule repos. While the parallel development went easier with this approach, the code organization became messy over time with a lot of submodule commit versions to track. After researching more, I now believe monorepo approach (though it has its own downsides too) would be more suitable in scenarios like this.
I am not fond of submodules and would choose monorepo almost always.
Hi Viktor. Thanks. I always appreciate your style of explaining things. Some important points to think about when it comes to the organization of software projects. I like that you always keep the balance of black and white. As you said, it's almost never black or white. It's mostly grey.
Oh yeah. It's never black or white. One of the reasons i started this channel is due to me being disappointed at people painting it like that.
Grazie.
Thanks a ton
My biggest pain point has been access control. Since the apps and libraries in a mono-repo are tightly coupled with each other it is difficult if not impossible to give separate access to people.
Oh yeah. The bigger the repo, the hard it is to separate into teams no matter that is access control, PR reviews, or anything else
Thank you for the video, I discovered the existence of such a thing as service mesh!
Finally.. we can talk about this!!
Monorepos don't prevent you from having backwards compatible apis. They give you the opportunity, but not the obligation, to refactor
Awesome video, thanks a lot. Information was conveyed very clearly .
I just about coughed up my cereal at the end when prompted to subscribe lest I get notifications of any change to your monorepo hahaha. Had I not already been subscribed, this threat would have served as sufficient call to action for me to subscribe.
Awesome channel, you are a walking encyclopedia.
turborepo kill it all... great video 👍
6:55 exactly! Your deployables should be mapped 1:1 to repo. Refactoring in a monorepo is harder, and the scaling makes this harder. Instead of changing the API of one library/service and then changing how its dependencies use it, you need to make all of those changes at the same time, slowing down your entire org while the change happens.
Much of the concerns regardingMonoRepos can be addressed with Nx (on the front-end).
I agree. Especially the "pipeline issues" concern. Nx makes it wicked easy.
Thank you for a great explanation. There is no black and white. It depends on requirements and resources.
I enjoyed this video a lot because I am someone who is against mono repo to build microservices! I open a folder and use a git branch for per microservice in my single repo as main branches of those micro services and from those sub main microservice branches it becomes very easy to work and build pipelines and also maintains a single repo for each microservice project! Branched based poly repo strategy is very useful but don't know why many famous tech company like google and fb don't use it.
First of all, I'm a regular viewer of your channel and love your videos! Even though I disagree with a lot of what you say in this one :-) Just out of curiosity: how much experience do you have with monorepos and the related tooling?
Now for my comments...
Being able to search the entire codebase is indeed easier on a monorepo. Not sure I understand how the presence of stuff you aren't always interested in makes searching harder though. You can restrict your search to the folders you're interested in.
Cross project atomic commits are as least as big a benefit to collaboration I find. For example, if you work on a service that requires backend and frontend changes, they can be included in the same PR. This reduces the chances of breaking changes and helps you to staying end-user focused.
The problems you describe around dependency management aren't monorepo specific I think. In a polyrepo setup you would also need to rebuild everything that depends on reused code. The dependency tree you describe exists in a polyrepo situation too, unless you decide not to reuse any code or APIs, but that doesn't like a recipe for success. Yes, you need specialized tools to describe and make use of the dependencies, but for a monorepo those tools exist at least.
Indeed, if you frequently need system wide refactorings, that is an architectural smell. And since monorepo makes the use of internal libraries easier, it requires extra discipline not to end up with a distributed monolith. The basic rule for internal libraries is: don't share domain specific classes or logic, only horizontal technological layers. Services should talk to each other via APIs not via libraries. But I have seen distributed monoliths in a polyrepo setup too. This was a result of sharing internal domain oriented libraries, which is just as possible, but with additional versioning headache.
So while I agree that frequent system wide refactorings are a smell, they may still be necessary to get out of the bad situation. And within a monerepo you can at least *consider* doing them.
WRT the disadvantages:
- Slow git operations: this really depends on the size of your codebase. In my experience this is not an issue on medium size (maintaned by say 50 engineers) codebases. Incremental pulls only take seconds, unless somebody has comited binaries, which shouldn't be done anyway. For pristine CI builds it _can_ take some effort to keep the clones fast enough: cached snapshots, shallow checkouts, etc.
- Encourages tight coupling: agreed. It makes the use of shared libraries easier and therefore also the abuse of shared libraries. However, disentangling tight coupling is a lot easier in a monorepo too.
- Pipeline issues: I disagree pretty strongly with this one. For very simple projects, with services that can live in splendid isolation pipelines are simple. But that's not what we are talking about. As soon as you have services that depend on each other, you need to orchestrate your CI. This is virtually impossible to do with polyrepos.
On a monorepo master can be broken in different ways. If any of the leaf projects is broken, this does not affect anybody else and doesn't block development. If a widely used library is broken, this will stop the work, as it should, if you want to practice continuous integration. Monorepos will make it much less likely to happen though. Imagine publishing a shared library in a polyrepo setup. How do you even know if it breaks any of the clients using it?
With respect to the noise monorepos create, that can easily be solved with a folder structure, code owner files or automated folder based PR labeling.
Yes, I am in the monorepo camp as you may have gathered by now :-) If I have to summarize the polyrepo vs monerepo tradoffs in one sentence: while there are a few bad practices that monorepos make easier to do, they illuminate architectural problems that tend to stay hidden with polyrepos while at the same time making it much more possible to address them.
I like the idea of a monorepo per team, but the freely available build tools for Typescript have not supported that. (Are there any new developments on the horizon?) The best supported cases for monorepo are for collections of related modules, such as modularized libraries, e.g., babel, or apps consisting of a core plus a bunch of *related* plugins.
I try to avoid monorepos as much as possible, and instead clearly separate my project into components, each of which then lives in its own repo.
I think that in the past it might have been easier to pack everything into one file tree and figure out separation of concerns for yourself, but nowadays with the open-source ecosystem on GitHub, modern CI/CD tools and containerized applications that rely on a multitude of programming languages, polyrepos are the simpler solution.
The one major downside is that there are more steps involved in making changes across components (say you want to introduce a new feature to the API and simultaneously develop the UI that consumes it), but then again linking dependencies locally ain't that hard and the worst you'll have to deal with after that is coordinating a few PRs at once (so that the dependencies will merge and build before the depending components). I think that tradeoff is waaayy better, especially as I expect the tooling for cross-repo work to improve a lot in the near future.
PS: Great video!
PS2: I think we are supposed to call it the "main" branch nowadays ;)
I agree with all the points. The upsides of using monorepo are easily overshadowed by the downsides.
I'm doing my best to call it main or mainline but `gh` is holding me back. Whenever I create a new repo with `gh repo create`, it creates the `master` branch instead of `main`.
Nice video, Victor! In our project, we got similar conclusions in our discussions about monorepos vs polyrepos. I think monorepos are more suitable when you are working with a monolith. When you are working with microservices makes more sense to use polyrepos.
Those are my thoughts as well. If I do not have any additional information and would make a decision only based on architecture, I would say that monoliths and distributed monoliths (faking microservices) are better suited for mono-repos, and microservices for poly-repos. It's not always that easy though, but that could be a rule of thumb.
I can't agree with dependencies&monorepo thing. You should always build everything. You should always make sure that "that small dependency" does not break everything. You should test everything after the smallest change as well. Yes, it takes time. But if you want to deliver high-quality and stable products you need to do it.
I don't agree with that. You should build apps that changed, not everything. The exception is when you have a distributed monolith.
Excellent explanation!
I use monorepo for infra and monorepo for serverside code, and monorepo for frontend, and etc. Is it multimono? :)
But I doing this comment just to say that what is the nice place you are sitting!! Clean, silent, adorable... Love it
If you have a repo that contains more than one project (more than one app that is built and deployed separately), you're having a monorepo. One can have one monorepo for everything or multiple monorepos or a combination of monorepos and polyrepos.
As for the place where I recorded... It's beautiful. We go there at least a couple of weekends every month.
The biggest issue I have with it is that project managers hear about google using it and try to use it for a couple of small projects. The devs end up trying to figure out how to solve version conflicts in dependencies of dependencies of dependencies of dependencies and why nohoist seems to be doing nothing at all. Combine that with the company want their devs to work on a trendy mac so npm/yarn takes a hundred times longer to complete and you have a team not programming anything but instead tinkering with lerna and yarn workspaces instead.
The problem with "Google does it" is that Google was not doing it always but evolved over time. Even if the "Google way" is indeed something someone should take, that means not only doing what Google is doing today but also walking a similar path as Google was walking throughout the years before it became what it is today. One cannot just teleport itself into the current state of Google (besides the question of whether one should have it as the goal in the first place).
One more thing... The moment project managers start making technical decisions is the moment one should consider changing the company.
Great explanation, thanks!
I think ETL pipeline based on microservices is best suited to mono repo since the context of the pipeline is bounded
Thanks Viktor 😊
The only reason why I want to try monorepos at all is because I highly dislike the developer experience you get when working with 'microservices' or should I say distributed monoliths.
I recently had an interview with a Lead Dev of a publicly traded company. He told me they refactored everything the company has into a monorepo. I just thought, boy they should fire you and we'd not be friends ^^.
If the project is not that big, your team is small or it is just only you, if load is not gonna extremely big... than monorepo is fine
Thanks, Victor!
I found myself at this crossroad a while back and eventually went with Polyrepos. However, what are your thoughts on using git subtrees to manage these multiple repos in a single base repo. My personal experience with git subtrees is that it eventually becomes hell to manage.
I really do not like subtrees. I would rather do monorepo.
@@DevOpsToolkit what do you think about dev containers (containers.dev)? I am using bazel and monorepo for a couple of months, but it doesn't feel right if you have to write custom rules for languages such as Node, TypeScript, Rust, C#...
Thanks! The flip side of easy collaboration across teams is perhaps security (access control) isn't it? Everyone in a team gets access to all the code of all other teams. Depending on the organization and the type of product being developed this may or may not be desirable. For example, Google can't possibly have their top secret search engine algo code openly available as part of a monorepo. So, security should be listed as a potential downside in my opinion.
That's correct. I should have added that one.
Monolith to Microservices to Monorepo. Why do we need to be happy anyway?
I think about using a mono repo, but I found a problem and discarded my idea.
My problem now is, that I forget the problem I found 😅
Why you think there is one master branch? You don't need to have just one branch. There are tools that help build projects and related libraries to them, not the whole project at once like Nx. Nx cloud caching in cloud already built projects. So if you pull it, it will all be built already.
There is only one mainline (previously called master). That does not mean that there are no other branches.
What, if you ave a family of products in a monorepo…
Are those products independent of each other or more like a distributed monolith?
i love your video i wana be like you thankiu very much, polyrepo all the wayy!!
How about MonoPoly? One repo with multiple projects, developers gain access/own to folders based on their ranks/achievements in their development journey :-).
... and if you collect enough achievements, your PRs are merged to the mainline and deployed to production automatically without anyone's intervention.
@@DevOpsToolkit sounds like a new feature request for your new product 😁
Do you live in Malta?
It's not Malta. I live in Barcelona and now you made me curious. What made you guess that it's Malta?
Git submodules has enabled best of both worlds imho.
|11.59 you're not Google, don't use monorepo
|12.00 use microservices
😂😂😂
Yeah. Monorepo is typically used with distributed monoliths disguised as microservices.
Excellent explanation !