Yup. Really loved the talk, and how Boris slowly and incrementally, no pun intended, presented it. Actually clarified quite a lot of things about build in general, though I felt I was always familiar with them, looks like I had some spots to fill. Thanks Boris!
Boris is pointing out the "implementation language: Values" (20:59), but in my opinion the language, runtime, etc. and it's own values (java, python) should not conflate with the values of the final product. For example Google's bazel is written in java, and please.build is written in Go - but these systems (along with pants, gn, buck, etc.) do not target just one language (C/C++), but multiple - including python, bash scripts, etc. While bazel might be real slow the first time, since it keeps a server in memory, it can be very fast on subsequent runs. Also it doesn't have to use the standard java packaging, but it's own (e.g. it's own tool, that just happens to be written in java). I feel that this "raciality" where everything you do use must come, or be written in the main language you use (or the main programming, not scripting) is simply not beneficial. Yes, usually the contra argument is that - writing the build system in the language I know would help me change it easier, and that may be true - but the question you should be asking yourself is - how often would that happen to a typical user of that build tool.
I was not advocating for a C++-only build system. However, these days a C/C++ compiler is native to all the platforms one cares to run the build system on (i.e., it does not require another "system" as Python or Java would). Which makes it a natural implementation language for a general build system and definitely the preferred language for C++-centric projects.
I get what you are saying, but seeing Rust, Go going with their own build tool, and taking dog-fooding to the extreme takes away all the language inter-operability... But as it goes (and thanks for mentioning Bryan Cantrill's talk - had to watch it first, and then go with yours) - different values, drive different decisions - Certainly I would like to be in the place, where I can use different languages to build systems that talk to each other, and not to be limited by one. Obviously not the prime use case for "desktop", or "mobile", "console/game" oriented apps (although for games, you may have tools, and then runtime code), but certainly for micro-services - it scales up - because you are no longer limited by one language, you can use C, C++, rust, go, java, python - let these express dependencies among themselves, and given good IPC/RPC/whatever mechanism to talk to each other, you can achieve it. Really it's about micro services, and the culture behind it, than anything else (like I said, good ole' classic "desktop" apps).
54:30 - How to check your build system is working properly?I have one answer on that - you can easily check make -n output (or compiler database, if you have any). If build system for one particular configuration outputs just the same compiler and linker invocation (create a tool that compare arguments without checking order) - it probably correct. I do not talk about messing with invocation order - running yacc before compiling generated code, etc. I have written cmake scripts to build ffmpeg with all dependencies, so I tested mostly two things: "make -n" are identical and all generated files are the same. After that, I could not have a ready answer, how to check all possible configurations for new build systems, but I think it possible to create "configure tests" which check dry run output and generated files.
great talk, lots of good points few questions/comments from my point of view if someone knowledgable about build2 will read this: :) 1. We are migrating to conan.io in my company. Are there any plans and even is that a possibility to use both conan.io and build2? Conan.io to manage deps and build2 as a build system. 2. Project composability, subprojects etc, Not sure if this is what I am thinking of, but in conan.io workspaces are now being introduced, which is a feature i fell in love with while working in rust. Can you please link or tell me more about this topic? My most common scenario in c++ right now is as follows: I am working on a project, this project contains multitude of lirbaries. every library is located on separate git repository. Before workspaces each time i made a change to library i had to deploy project in order to use that change in upstream projects, now with conan.io workspaces i am able to work on workspace made of 10+ projects, which is super great. I achieve that doing git submodules and workspace configuration from conan.io. Upon changing library binary detects that change and recompiles everything as needed. is this something that you were talking about? in b2 is it possible to use submodules like that? 3. Wildcards - always tough topic. :) I am using cmake globs, even though they are not recommended, because its cmake everytime i add file /change branch /rename/remove i have to reload cmake. From what I understood b2 supports wildcards so in my opinion thats great. Meson build system is against wildcards, but honestly I think for most of the project the cons of having these are outweight by pros - not having to look at the build file multiple times a day is great, not to mention conflicts on file lists... 4. Unfortunatelly i will not be pushing b2 as a backend even if it worked with conan.io because of lack of IDE support. I am using clion now, and its getting better, finally using clang as a backend for understanding code.
for build systemt to succeed it needs to come with the language out of the box i wanna try c++ ? install cppbuild, and done, you can create project, add package, test, etc new script language ? NO, i don't want to learn new language, either you use c++ or use data json/xml
IDE support in general, and Visual Studio support in particular, is critical to the success of any new build system. There are a huge segment of C++ programmers who live in Windows-land, and you need to have *excellent* VS support to even be considered. Many of these programmers will *also* want a great CLI, but it has to be optional. Also... IDE integration doesn't just mean pressing the "build" button and having it run your build tool. Project management needs to be fully integrated into the IDE GUI. Editing build files and manifests in the IDE text editor doesn't count as "integration"! One consequence of this is that build files need to be both human and programatically editable. Makefiles and CMakeLists.txt are awful in this regard -- there's no way for an IDE to safely update them. JSON (or XML) work much better.
I think that's exactly the point Boris was talking about - the IDE should not be the driving force of your build. Just because it happened to integrate (UI-wise) with adding/removing/etc. your source files, doesn't mean is the only way, or the correct way... and long term the sane way to do things. We do have a rather elaborated system with MSBuild files with custom .props, .targets, to the point where the IDE would get confused about project references, and if you try to use the IDE to add, remove, edit these - it most likely won't work. Do we wanted the IDE to work with them in the first place? We wanted - but then our software (really a bunch of project files) gets shared across different teams, and they integrate them in different ways, folders, pick certain parts, ignore others - and then we had to use more advanced features - like PropertyTargets (MSBuild) - and this is where the connection with the IDE breaks... Surely, maybe you can spent some other time to make it IDE-friendly - but several tried, and it's a lost battle.... A better approach would be a build system, model where the IDE issues commands - I need to add, remove, change this entry - "You do it", then give me the "view" of how things are looking now... Surely - not the fastest, probably prone (at first) way - but had lots of success with CLion/IntelliJ and bazel (blaze back then) at Google. If BUILD2 offers such introspectability, reflection, whatever you call it, surely one can integrate it easy in any IDE (given the time)... probably not Visual Studio, but probably easier in CLion, and others like it.
I think you're missing my point. A substantial chunk of C++ programmers live in Windows-land and use Visual Studio. I'm not saying this is a Good Thing -- I'm saying its Reality. Given that reality, any build system that hopes to become "the one C++ build system to rule them all" needs to have a really good Visual Studio integration story, or it's just #15 ( xkcd.com/927 ).
I thought the same, I came from gamedev shop, then spent 2-3 years at Google, doing java, and saw that 40,000+ engineers do not actually use Visual Studio, but other things I've never used before in my life - like Eclipse, CLion, IntelliJ - and build systems I never thought. But the biggest point was, that the build system actually worked (somehow) quite well, allowing people to reuse code, reference it, etc. - and worked across languages, runtimes, etc. (granted just several few accepted languages), and wasn't really cross-platform oriented (like what a gamedev console developer would like), but going there... So the point of just delivering system that would work for Visual Studio, and it's user is not something breakthrough - you already have it. These new build systems, are not coming to accomodate these people, but rather let you compile/build/test/run your project under plethora of different operating systems, devices, or deploy them across services... From what I've seen, most seasoned MSBuild users do not restrict themselves to what the IDE supports, or might not support... Yes you still get your Intellisense, code naviagation, etc. - but you can't expect to be able to control everything through the IDE - rather you would go and touch a local Build.Directory.props (or something like this), this way changing the way things are build globally, setting a new version, or directing all packages to use suddenly a new version of something used across - e.g. you don't do such things through the IDE... Also when was the last time you were able to review build changes in a code review... given what the IDE did? I'd rather see BUILD (build2, msbuild, cmake, bazel, etc.) whatever change that human did, than what the IDE did - because the latter is often indecipherable - ... case in example the .sln generation (which is an abomination, and I think Microsoft are soon or later going to retire it).
Though, you are right about one thing that might definitely piss off engineers, and I've seen it actually :) (but I think could be addressed). So often Ctrl+F7 to single compile file is good "debugging" tool, especially when dealing with some weird compilation hurdles. Now a generator, like cmake, premake -> .vcxproj would have no problem there, but a build system on it's won (make, build2, bazel, scons, etc.) might not, or you might need to add this "Extra Tool" to do it. Then you have to spread out the knowledge to the folks how it should be done... Though recently been seeing that Microsoft are pushing OpenFolder, and Visual Studio Code is going even more ahead in that respect...
+1000 for requiring to explicitly list .h-eader files, and the fast pre-proc, post-proc scheme. Great talk. Thank you!
Great talk!
Yup. Really loved the talk, and how Boris slowly and incrementally, no pun intended, presented it. Actually clarified quite a lot of things about build in general, though I felt I was always familiar with them, looks like I had some spots to fill. Thanks Boris!
Boris is pointing out the "implementation language: Values" (20:59), but in my opinion the language, runtime, etc. and it's own values (java, python) should not conflate with the values of the final product. For example Google's bazel is written in java, and please.build is written in Go - but these systems (along with pants, gn, buck, etc.) do not target just one language (C/C++), but multiple - including python, bash scripts, etc. While bazel might be real slow the first time, since it keeps a server in memory, it can be very fast on subsequent runs. Also it doesn't have to use the standard java packaging, but it's own (e.g. it's own tool, that just happens to be written in java). I feel that this "raciality" where everything you do use must come, or be written in the main language you use (or the main programming, not scripting) is simply not beneficial. Yes, usually the contra argument is that - writing the build system in the language I know would help me change it easier, and that may be true - but the question you should be asking yourself is - how often would that happen to a typical user of that build tool.
I was not advocating for a C++-only build system. However, these days a C/C++ compiler is native to all the platforms one cares to run the build system on (i.e., it does not require another "system" as Python or Java would). Which makes it a natural implementation language for a general build system and definitely the preferred language for C++-centric projects.
I get what you are saying, but seeing Rust, Go going with their own build tool, and taking dog-fooding to the extreme takes away all the language inter-operability... But as it goes (and thanks for mentioning Bryan Cantrill's talk - had to watch it first, and then go with yours) - different values, drive different decisions - Certainly I would like to be in the place, where I can use different languages to build systems that talk to each other, and not to be limited by one. Obviously not the prime use case for "desktop", or "mobile", "console/game" oriented apps (although for games, you may have tools, and then runtime code), but certainly for micro-services - it scales up - because you are no longer limited by one language, you can use C, C++, rust, go, java, python - let these express dependencies among themselves, and given good IPC/RPC/whatever mechanism to talk to each other, you can achieve it. Really it's about micro services, and the culture behind it, than anything else (like I said, good ole' classic "desktop" apps).
54:30 - How to check your build system is working properly?I have one answer on that - you can easily check make -n output (or compiler database, if you have any). If build system for one particular configuration outputs just the same compiler and linker invocation (create a tool that compare arguments without checking order) - it probably correct. I do not talk about messing with invocation order - running yacc before compiling generated code, etc.
I have written cmake scripts to build ffmpeg with all dependencies, so I tested mostly two things: "make -n" are identical and all generated files are the same.
After that, I could not have a ready answer, how to check all possible configurations for new build systems, but I think it possible to create "configure tests" which check dry run output and generated files.
This was really helpful xx
great talk, lots of good points few questions/comments from my point of view if someone knowledgable about build2 will read this: :)
1. We are migrating to conan.io in my company. Are there any plans and even is that a possibility to use both conan.io and build2? Conan.io to manage deps and build2 as a build system.
2. Project composability, subprojects etc, Not sure if this is what I am thinking of, but in conan.io workspaces are now being introduced, which is a feature i fell in love with while working in rust. Can you please link or tell me more about this topic?
My most common scenario in c++ right now is as follows:
I am working on a project, this project contains multitude of lirbaries. every library is located on separate git repository. Before workspaces each time i made a change to library i had to deploy project in order to use that change in upstream projects, now with conan.io workspaces i am able to work on workspace made of 10+ projects, which is super great. I achieve that doing git submodules and workspace configuration from conan.io. Upon changing library binary detects that change and recompiles everything as needed. is this something that you were talking about? in b2 is it possible to use submodules like that?
3. Wildcards - always tough topic. :) I am using cmake globs, even though they are not recommended, because its cmake everytime i add file /change branch /rename/remove i have to reload cmake. From what I understood b2 supports wildcards so in my opinion thats great. Meson build system is against wildcards, but honestly I think for most of the project the cons of having these are outweight by pros - not having to look at the build file multiple times a day is great, not to mention conflicts on file lists...
4. Unfortunatelly i will not be pushing b2 as a backend even if it worked with conan.io because of lack of IDE support.
I am using clion now, and its getting better, finally using clang as a backend for understanding code.
for build systemt to succeed it needs to come with the language out of the box
i wanna try c++ ? install cppbuild, and done, you can create project, add package, test, etc
new script language ? NO, i don't want to learn new language, either you use c++ or use data json/xml
IDE support in general, and Visual Studio support in particular, is critical to the success of any new build system.
There are a huge segment of C++ programmers who live in Windows-land, and you need to have *excellent* VS support to even be considered. Many of these programmers will *also* want a great CLI, but it has to be optional.
Also... IDE integration doesn't just mean pressing the "build" button and having it run your build tool. Project management needs to be fully integrated into the IDE GUI. Editing build files and manifests in the IDE text editor doesn't count as "integration"! One consequence of this is that build files need to be both human and programatically editable. Makefiles and CMakeLists.txt are awful in this regard -- there's no way for an IDE to safely update them. JSON (or XML) work much better.
I think that's exactly the point Boris was talking about - the IDE should not be the driving force of your build. Just because it happened to integrate (UI-wise) with adding/removing/etc. your source files, doesn't mean is the only way, or the correct way... and long term the sane way to do things. We do have a rather elaborated system with MSBuild files with custom .props, .targets, to the point where the IDE would get confused about project references, and if you try to use the IDE to add, remove, edit these - it most likely won't work. Do we wanted the IDE to work with them in the first place? We wanted - but then our software (really a bunch of project files) gets shared across different teams, and they integrate them in different ways, folders, pick certain parts, ignore others - and then we had to use more advanced features - like PropertyTargets (MSBuild) - and this is where the connection with the IDE breaks... Surely, maybe you can spent some other time to make it IDE-friendly - but several tried, and it's a lost battle.... A better approach would be a build system, model where the IDE issues commands - I need to add, remove, change this entry - "You do it", then give me the "view" of how things are looking now... Surely - not the fastest, probably prone (at first) way - but had lots of success with CLion/IntelliJ and bazel (blaze back then) at Google. If BUILD2 offers such introspectability, reflection, whatever you call it, surely one can integrate it easy in any IDE (given the time)... probably not Visual Studio, but probably easier in CLion, and others like it.
I think you're missing my point. A substantial chunk of C++ programmers live in Windows-land and use Visual Studio. I'm not saying this is a Good Thing -- I'm saying its Reality. Given that reality, any build system that hopes to become "the one C++ build system to rule them all" needs to have a really good Visual Studio integration story, or it's just #15 ( xkcd.com/927 ).
I thought the same, I came from gamedev shop, then spent 2-3 years at Google, doing java, and saw that 40,000+ engineers do not actually use Visual Studio, but other things I've never used before in my life - like Eclipse, CLion, IntelliJ - and build systems I never thought. But the biggest point was, that the build system actually worked (somehow) quite well, allowing people to reuse code, reference it, etc. - and worked across languages, runtimes, etc. (granted just several few accepted languages), and wasn't really cross-platform oriented (like what a gamedev console developer would like), but going there... So the point of just delivering system that would work for Visual Studio, and it's user is not something breakthrough - you already have it. These new build systems, are not coming to accomodate these people, but rather let you compile/build/test/run your project under plethora of different operating systems, devices, or deploy them across services... From what I've seen, most seasoned MSBuild users do not restrict themselves to what the IDE supports, or might not support... Yes you still get your Intellisense, code naviagation, etc. - but you can't expect to be able to control everything through the IDE - rather you would go and touch a local Build.Directory.props (or something like this), this way changing the way things are build globally, setting a new version, or directing all packages to use suddenly a new version of something used across - e.g. you don't do such things through the IDE... Also when was the last time you were able to review build changes in a code review... given what the IDE did? I'd rather see BUILD (build2, msbuild, cmake, bazel, etc.) whatever change that human did, than what the IDE did - because the latter is often indecipherable - ... case in example the .sln generation (which is an abomination, and I think Microsoft are soon or later going to retire it).
Though, you are right about one thing that might definitely piss off engineers, and I've seen it actually :) (but I think could be addressed). So often Ctrl+F7 to single compile file is good "debugging" tool, especially when dealing with some weird compilation hurdles. Now a generator, like cmake, premake -> .vcxproj would have no problem there, but a build system on it's won (make, build2, bazel, scons, etc.) might not, or you might need to add this "Extra Tool" to do it. Then you have to spread out the knowledge to the folks how it should be done... Though recently been seeing that Microsoft are pushing OpenFolder, and Visual Studio Code is going even more ahead in that respect...
I'll take hand crafted build system over IDE generated gibberish any day...