Sorry for the delay! If you didn't see my community post, my editing software of choice (DaVinci Resolve) unpredictably took almost 9 hours to render this video for some reason. This topic is the foundation for everything else we're going to do, so I hope you can excuse this extremely dry explanation which definitely got way more technical than I anticipated... Once we're done with all the ground work, some real exciting sh*t will wait for us in the end!
As a small time graphics TH-camr myself, it’s great to see this space get more attention. I really like the systems perspective you bring with your videos!
Regardless of whether you think this way of going about things is effective/efficient or whatever else metric you could think of(I'm just fascinated, and feel like there will be a lot of value, and reusability, in larger blocks, once assembled), the breakdown of subject matter is *invaluable* . Thrilled for more!
I don't really understand the reusability argument, how are nodes more reusable than functions/methods? I'll keep watching because I still think it's interesting but I don't think it's a really useful concept, at least as far as making things more reusable goes. It's worth exploring though I guess that's the point, even if not for that there may be other benefits :D
You can think of a node as one complex compiler macro. Instead of moving functionality to different functions, which may come with additional runtime overhead, we're moving them to building blocks outside of a programming paradigm. If everything is put into functions within functions, even the best compiler will eventually stumble and your final executable will have a bunch of stack and register cleanup code + jump / call / return code in it, not even mentioning the likely instruction cache invalidation that would cause. Yes, some languages allow you to explicitly make a function inlined - but not all of them, and we're trying to target all of them. By essentially using macros, all these function related code pieces don't exist, if we don't explicitly want them. Besides potential performance benefits, it's also a much more readable visualisation once we get into multiple targets. Imagine mixing C++ source code and Javascript in some text-based file format... We're using a completely separate form of visualisation, outside of the standard programming interfaces, so we can combine different forms of byte sequences under the same hood. :D
@@Kronark Mm I see, my intuition was that at the end you'd just end up with a lot of unoptimized code if you just combine nodes that way. Or if you've addressed that maybe I didn't understand, I'll try rewatching these later ^^ But apparently you have a nice plan laid out so I'm excited to see how you go about this concretely then 👍 Best of lucks with the thesis too 😄
Haven‘t addressed optimisation yet, don‘t worry :) there‘ll be separate videos wjere we optimise stuff :D Optimisation will work the same as umoptimised output - instead of naiv approaches, we‘ll simply output optimised bytes. For example, if we multiply something by a power of two, we‘ll output a bitshift instead of multiplication instruction inside our multiplication abstraction! Thanks, I‘ll need it! 😅
@@Kronark yeah good luck, hope you finish that thesis early, so you'd be able to relax a bit at the end, and so we could have that cool looking language
Maybe you should give each line/connection a different color to make the whole project a little clearer. Or use different colored backgrounds for different topics in the video! Love your videos! Keep going! 😊
Connections aren‘t colorable in this prototype unfortunately, and adding that would be to much of a detour i‘m afraid! But the release build will have ui customisation ☺️ I‘ll try adding some highlighting effects for the dev-logs coming up! And thank you! :)
Glad to hear it! I think I've gone over the PE format so many times now, I'm really unsure how well it will go until next week. I will probably split one hefty part into a separate video though: the checksum calculation - simply because there are no good sources on that algorithm and It'd be nice to have that separate for easier visibility on search engines... Hope the remainder doesn't result in another 13 minute video ^^
I know it's too early for this kind of feedback... but a concept that can help to master the complexity of the network, when it starts to get overloaded, is to adopt balloons that abstract functions, with inputs and outputs, which have an open mode (to visualize the internal network) and a hidden-spam mode to reduce visual complexity... it's just an aesthetic tool, but it can significantly increase expressive power. Another very useful and widely used functionality is 'while', for me it could be simplified and would avoid programmer errors, if it were based on only 2 types, the range type n (if n
I think your balloons are the core mechanic of this tool... isn't that exactly what we're doing in these videos? Grouping nodes together to form new nodes? Regarding array implementation, there won't be dedicated graphics for it. We will be using something called a repetitive socket though, which allows us to provide multiple values for a single input.
Thanks! and sorry about that :/ unfortunately those will probably not be replaced any time soon... don't have time to redesign the video aesthetic right now... maybe next year!
I've been thinking about visual programming languages a lot lately, particularly how in many respects it could actually simplify language development, i can't wait for you to explain in more detail how you compile these nodes
gonna be a while until we get into that stuff, will focus on my masters thesis until February / March, probably do some more graphics stuff after that and then eventually do compiler implementation again - work on the release product. definitely need a break from compiler design after the prototype is done
I'm a 3D art enthusiast, I use blender and I'm familiar with Nodes. I'm fascinated by programming and I'm seeing your videos that are very interesting and well done, but I can't understand much about them, what do you recommend me to read or see in order to understand more of what you're doing? I know a little bit of programming, but I don't know anything about low-level languages.
If you have a grasp on general programming concepts you‘ll find in every language (ifs, loops, math, etc.) and you‘re really interested, i‘d recommend looking into how interpreted and compiled languages work i.e. how your text actually gets turned into something a computer can use. So stuff like lexing, parsing, abstract syntax trees… It all really depends on what you already know to be honest. If it‘s too heavy stuff, I‘d just start with how some data structures work, because you usually have to think about memory and operations etc when tackling those. Once you understand that pipeline you should somewhat understand what i‘m trying to do here :)
In this video he is encoding x84 instructions. Learn assembly. It is not that hard to write for small projects. Then look at the output in a hex editor.
Do not confuse "low level languages" with "low level languages". C, C++, Rust are "low level languages" that compile to "low level languages" like assembly. But assembly is more of an intermediate language these days. Frankly. You only use it for a few lines to write a boot loader for operating systems that then are written in C or Rust. Or to accesses hardware functions that then abstracted by C or Rust anyways. So unless you want to relive the glory days of Wolfenstein 3D or classic Quake and Doom that had in addition to C code some assembly in it in performance critical places because you are John Carmack. Other then thay assembly is not a "low level language" but just another intermediate language that low level languages like C and Rust compile to before they become final machine code.
@@techpriest4787 There are nearly no languages that compile to assembly. Assembly is machine code with direct translation to mnemonics for readability. Compiled languages just compile directly to machine code.
Is your idea that you should be able to build abstractions upon abstractions to effectively make it so you can write high level code that is built up by lower and lower levels of abstractions?
Im not sure if this is happening to anyone else but the graphics have this flickering effect sometimes. Great video im wondering if the format could be changed slightly, i felt like i got lost at the begining/middle and then when you tied everything up with a goal and an example it made more sense. Maybe a graphic that kind of breaks down each concept and we can see where it will be used in relation to the end goal is needed. Its hard to keep many things in my head when i dont know how/where they will be used
Hiya mate, yeah I‘m aware of some weird rendering artifacts - I currently just don‘t know how to prevent them. I may experiment with render settings if I have enough time for the video next week. Speaking of the video next week, it seams like it‘s going to be similar in complexity to the video this week. So I‘ll definitely try to implement your feedback there - I was in a bit of a rush for the one this week. Thanks!
Really interesting project, at this point I think I'm even able to follow you but im really interested in how you will make the step to compiling actual programs because I don't get how you will be dealing with register usage and not having everything being inline. Im kindof familiar with low level programming/assembly but I have never worked with node based programming. Maybe thats what hinders me right now. I don't want to criticize your way of explaining everything from the ground up but at least for me the bigger picture is really not clear at all. Looking forward to more videos:)
We‘ll use a global state system built into the compiler for register assignment and stuff like that! It‘ll all become clearer eventually I hope :D The bigger picture will be more apparent eventually as well ;)
@@Kronark Too kind. I barely get it too, but then I'm old enough to know assembly and co. Yet, not skilled enough to do anything useful - not even with high level languages XD But I'm curious to see where this goes. It can be ANYthing, the potential is almost endless.
Question: GCC and other compilers perform a LOT of optimisations on C and C++ Code when converting it to assembly, which improves efficiency, However, your solution requires the user to effectively use assembly directly. So how would you implement these compiler optimisations?
Users won't be required to use assembly directly for optimisation or anything else. It will be hidden behind multiple layers of abstraction just like in a traditional programming language. However, the difference in this tool is, that you always have access to the lower levels of any given building block - so you *can* use the underlying assembly directly, but you don't have to. We'll get into optimisations further down the road, but the vast majority of optimisations essentially boil down to "replace x with y, because y is faster". A simple example would be the multiplication of some value by a power of two. Unoptimised code might simply compile that down to an "imul" instruction with a register and an immediate value as parameters. However, optimised code would replace the "imul" instruction with a simple bit shift by the exponent of that power of two value (logarithm with base 2). So in this node editor, all that needs to be done is add a condition that checks the input value - and if it's a power of two, output the bytes of a bitshift instruction instead of the bytes of a multiplication instruction. This only has to be done once, most likely by me years before this tool releases publicly. Any user after that doesn't have to touch any of these low level details to get more performant multiplication. In general, a lot of those optimisations you mention are required simply due to the source implementation being written in a human readable language. It's easier for us to split up complicated equations into separate variables for example, which is completely suboptimal for the final machine code. That intermediary translation is completely removed from this compilation process.
That is some cool stuff, visual programing needs some proper love. BUT But it seems lacking at the 12:30 register is set manually, it would be cool if there was wrapping node around 'add modrm' that contains logic that helps to fill data. So 'reg' can be chosen by name of 'RAX' or 'RDX' instead of writing binary values to 'reg' directly. Even if value '010' comes from external node it seems as a chore to drag that node manually. And if possible the other fields should be filled based on the choice that was already made if it is deciding factor. I'm not sure but if choosing 'reg 010' guarantees that 'mod is direct register' then 'direct register' should be filled automatically. So some sort of Abstraction Layer on top of nodes to handle Explicit and Conditional logic that helps with filling data in those nodes? I personally wouldn't want to remember by rote the op code in binary not to look at the documentation. I guess it would be wrapped in by some compiling nodes so it still will be hidden from the user with is nice.
I cut out the part which explained that the register assignment problem and any register abstraction such as a mapping of their mnemonics to their binary representation will be a topic for another video... this video was already long enough ^^ You will eventually not specify registers at all, we'll implement automatic register assignment using a global state. But that's two abstraction levels higher than we're currently working at, so it's something we'll do a bit later on. If reg is referencing a register directly within modr/m, mod can still not be 0b11 (direct register addressing). The addressing mode of mod only controls the interpretation of the last 3 bits in rm. the value of reg is interpreted implicitly by your choice of opcode. So automating the input in other sockets based on a given socket (i.e. set mod to "direct register") would actually make us lose control over the output in this situation. We're currently working at 2 abstraction layers below standard assembly code - the vast majority of users won't or won't want to touch anything "down" here. Luckily the vast majority of this complexity will be hidden behind multiple layers of abstraction once we get to standard high level practices.
@@Kronark The conditionals on the "direct register" I mentioned was an example 'if it is possible' as I have not enough understanding of the Assembly to know that such condition makes sense. And as you explained, it does not. As there is some form of abstraction latter then all is good. I can't wait to see how you will solve it, it is nice to watch. Still I wonder how the high level case of let say of providing Keys for the Interaction with keyboard will be made. Naturally the simplest way is to treat Key Codes as Integers they are and just provide them Manually. Then the next step of abstraction will be to use 'mappings to key names' and that is equal to what is provided in txt programming by enums or or other named static variables, albeit potentially faster depending on the compilation results. What makes me biased is probably the fact that I'm fan of the 'Widget Approach' for the visual programming with allows to naturally get better abstraction than text programming. In the Key example it would be fully rendered Keyboard from with user can select Key that will be used. properly done User would be able to specify if selected is the Key for it's value or because of it's position on keyboard with would resolve the problem of 'QERTY/QWERTZ'. Or multiple key need to be pressed conditionally. But that is just my personal preference. I'm fully aware that widget creation is not only complicated but also time consuming, not to mention the baggage that would need to be carried over all the implementations during prototyping phase.
Yeah, with most topics / features you can expect that we‘ll take the route most programming languages used e.g. „enums“ for the key codes. We‘ll add more and more abstraction layers over time, which may result in such a key selection interface eventually :)
There really isn't much else besides your standard wiki articles + the official specification to be honest. I'll see if I find the time to add links soon.
It'd be an assembler if I were limited to writing assembly. However I neither have to compile into object files, nor do I have to build assembly instructions with this software. So no, not an assembler. It is a compiler by definition. It transforms a source representation into one of various target representations. Assemblers are limited to assembly by definition. :) And thank you! :D
@@Kronark Ah... very cool! Yeah, this project is super interesting. Something that would make me lose by breath with amazement as a Linux user is if you adapted this to also work in a terminal. XD
Hey i realized you have three different playlists on your channel each one video long for now, i think it would be useful especially when your channel will have more uploads to make a playlist with all your videos for convenience :)
True! Was focusing so much on staying organised, I never really looked at the channel page - i‘ll make a general playlist with all videos in order of upload
The video is cool, but I have some questions about the project: Which programming language/framework did you use? What are the base nodes for the project (for example: add, rest, and, not)?
Hey, thanks! This prototype was built using electron.js, so javascript is the main language. There are no built-in nodes that provide any sort of programming functionality. There are only nodes that tell the compiler what to do with any given string of bytes. All programming functionality will be emergent using the built-in nodes. You can find a list of the current built-in nodes in my introductory video uploaded last week (the one with „this is machine code“ on the thumbnail)
It‘s understandable, I think you have to already be thinking about low level stuff like this to sort of see what I‘m trying to do - it‘ll definitely become more comprehensible later on! :) Thanks! :)
Hey are you willing to open source your data creation suite? Or at least allow people to download and play with it? I’m having trouble following as I’m not familiar with the software, however this seems super cool to play around with and wrap my head around.
@@chrisjones9132 Hey man, unfortunately that won't be possible for the time being, no. But the next videos I'll be uploading will explain how the software works, so maybe that will help! We'll be covering all the different nodes and the compilation algorithm itself. :)
The software may or may not become open source eventually, it will be free though. The node networks will be open sourced via a public node sharing plattform sometime in the future :) i‘ll probably ship the fundamental nodes like the ones in this video with the release build.
@@Kronark I already want to use it! Thanks for planning to provide it for free. I feel like it could be a very good entry point for teenagers interested in learning programming 🎉
12:36 Actually, that is only in Intel syntax. In AT&T syntax the target goes last. Intel: operation destination, source AT&T: operation source, destination
Great video(s)! One question: is it possible to make the system in a way so that it can output a multi-os executable/executables for multiple OS-s at the same time? Sort of like the Cosmopolitan project for C++.
Thankyou! Yes, that‘s actually pretty straightforward. If you add multiple file nodes, they‘ll all be compiled. You can channel the same byte sequence into different files natively with this tool. We will need to add a level of abstraction to actually make the same algorithms be output for different architectures / OSs e.g. that one outputs ARM bytes and the other x64 - but it will be possible natively using a state system. Still need to fix some bugs with that system currently though - good thing we‘re only going to have a single target architecture and OS for a bit.
that is exactly what I've been feeling like for the past 2-3 years - using python and java in uni was an absolute pain when I knew I could do so much better if only I had this tool :D
amazing! although I understood very little. The process of building such a low level design is really interesting . ps: ur style of editing also feels nice :)
@@Kronark In all seriousness, great work. As someone who is interested in the general idea of "programming" but hates "coding", this visual approach is just attractive to me. Even if I never personally use this, I'm excited that this may now be possible.
How exactly do you create applications with this software? I understand that each node is a command, and many commands create an application, but since you are using a high-level language (JavaScript if I remember correctly), won't you have to interpret the nodes some way to create an executable? I believe that you said in the last video that the program is able to decide which nodes to interpret through perceived changes since the last compiled version, does the program translate the nodes into assembly, and then compare each line of both versions to find differences, or do you have another way of doing that? It's a bit confusing since I do not know the underlying structure, so I can explain my question in more detail if you want.
In a earlier video he mentioned this project can be used to generate any type of file from png to bmp to exe. My guess is that in the final step of compilation it prints some bytes to a file which results in an executable.
While the prototype is written in javascript, the nodes don‘t output javascript to be interpreted. Each node is a simple object on the heap with an array of sockets associated to it. To detect if a node needs to be recompiled / the input has changed, a simple flag in the socket data structure is checked -> if it‘s state doesn‘t match the compilation iteration, it needs to be recompiled. So no expensive string comparisons needed! But you can think of these nodes as a weird ui for building strings of text. To make an executable, some specific text is put into a file - but like you said, we‘ll get to that soon. We‘ll probably not get around to actually executing our first exe next week, since a core component will be split into a separate video - but we‘ll get there eventually!
This one gets it! Exactly. All these nodes do is create strings of bytes in different ways. And we can combine them to build a byte sequence resembling an executable!
He's got the plan to build an "if" node that takes a condition and two code blocks as arguments, which it then will use to construct jmp wrapper around the code blocks based on the condition - that's still a bit in the future rn.
I‘m tending towards not making this software open source. But there will be a release version eventually, once all bugs have been fixed and everything has been optimised. We‘ll develop that release build using this prototype during a dev log series on this channel. In short, you‘ll be able to use it eventually, just not any time soon.
I hope so too, but it looks like the PE file format is gonna be a lot of stuff again. It‘s gonna be a bit simpler in terms of what each byte means tho. It‘ll all be way more digestable once we get to single operations per video like addition and printing to console instead of such abstract formatting stuff!
How will compile time optimizations be done in kronark? This looks like each operation will simply be deconstructed into its primitives and written to a binary file. Is there going to be any kind of optimization?
The neat part is, with this tool we‘re moving all compile time optimisation into development time. All optimisation is completely in user control. Since we‘re designing our own abstraction layers, we can decide which actual machine code is emitted based on certain inputs to a node. So for example we can control to which degree a loop is unrolled based on the amount of instructions inside the loop body. We can emit a bitshift instruction instead of a multiplication instruction if the factor is a power of 2. And so on…. It‘s all at our fingertips, nothing is hidden behind a black box implementation. This will be tedious in the beginning, but we‘ll eventually get on par with modern compilers and then have all the freedom to grow beyond that way more easily.
Exactly! With the bonus that you can compile to anything and also add new compilation targets on the fly :) Kronark is the name of the overarching „business“ i want to build btw, this is just the „Kronark Compiler“ - name not found yet, will tackle on release build ^^
Probably not, or at the very least only with a restrictive licence. You'll be able to follow along on development of the release build though, in case you plan on making your own!
I can, but i won‘t in this prototype. To much work for something that‘s only aesthetic. Visual customisation will be in the release build though, including connection shapes :)
DLLs still need an exe to actually run as far as I know :P DLLs don't provide an entry point which tells the OS where to start executing. Besides that, they both use the same overall file format anyway, so why bother with DLLs right now? :D
@@Kronark Ok, you got me xD I thought of this as a way to provide fast tools or libraries. My first thought was to make a dll that could be executed by node-red for example. EXE just felt so limiting, because you can only use it on windows. Can't this produce a code that can be interpreted on any system? But then - it doesn't produce code... So it will always need to pack the output for the target operating system. I don't think I fully grasp the concept yet. I'll surely watch more to fill that gap.
The neat thing is, it CAN produce source code as well. But other than that, we‘ll obviously eventually add support for ELF and maybe even Mach-O files, so other OSs will be targetted as well. Maybe the next video will clear some things up!
Sorry for the delay! If you didn't see my community post, my editing software of choice (DaVinci Resolve) unpredictably took almost 9 hours to render this video for some reason.
This topic is the foundation for everything else we're going to do, so I hope you can excuse this extremely dry explanation which definitely got way more technical than I anticipated... Once we're done with all the ground work, some real exciting sh*t will wait for us in the end!
Don't care too much about the upload time the first thing in your list should be to be in time with your code!
Yeah davinci does that sometimes, try switching out the renderer to voukoder
Looks interesting, I‘ll try it out!
I will follow this series with great interest. Thanks for this content.
Please tell me how I can mess with this program rn. IDC if it's an unusable beta
this is what i though computer science was like before i got in. im in awe
As a small time graphics TH-camr myself, it’s great to see this space get more attention. I really like the systems perspective you bring with your videos!
Thank you! :)
This is actually insane. I feel stupid af, good job!
Hahaha It's just my special interest, I've been doing this for years, don't feel bad!
But thanks, I guess? :D
Regardless of whether you think this way of going about things is effective/efficient or whatever else metric you could think of(I'm just fascinated, and feel like there will be a lot of value, and reusability, in larger blocks, once assembled), the breakdown of subject matter is *invaluable* .
Thrilled for more!
Thank you! :)
I don't really understand the reusability argument, how are nodes more reusable than functions/methods?
I'll keep watching because I still think it's interesting but I don't think it's a really useful concept, at least as far as making things more reusable goes.
It's worth exploring though I guess that's the point, even if not for that there may be other benefits :D
You can think of a node as one complex compiler macro. Instead of moving functionality to different functions, which may come with additional runtime overhead, we're moving them to building blocks outside of a programming paradigm. If everything is put into functions within functions, even the best compiler will eventually stumble and your final executable will have a bunch of stack and register cleanup code + jump / call / return code in it, not even mentioning the likely instruction cache invalidation that would cause. Yes, some languages allow you to explicitly make a function inlined - but not all of them, and we're trying to target all of them. By essentially using macros, all these function related code pieces don't exist, if we don't explicitly want them.
Besides potential performance benefits, it's also a much more readable visualisation once we get into multiple targets. Imagine mixing C++ source code and Javascript in some text-based file format...
We're using a completely separate form of visualisation, outside of the standard programming interfaces, so we can combine different forms of byte sequences under the same hood. :D
@@Kronark Mm I see, my intuition was that at the end you'd just end up with a lot of unoptimized code if you just combine nodes that way. Or if you've addressed that maybe I didn't understand, I'll try rewatching these later ^^
But apparently you have a nice plan laid out so I'm excited to see how you go about this concretely then 👍
Best of lucks with the thesis too 😄
Haven‘t addressed optimisation yet, don‘t worry :) there‘ll be separate videos wjere we optimise stuff :D
Optimisation will work the same as umoptimised output - instead of naiv approaches, we‘ll simply output optimised bytes. For example, if we multiply something by a power of two, we‘ll output a bitshift instead of multiplication instruction inside our multiplication abstraction!
Thanks, I‘ll need it! 😅
you: "man, programming languages are SHIT"
also you: "fine, I'll do it myself..."
anyway, can't wait for the 3rd devlog...
Definitely gotta get started on that 3rd one asap - gonna be another longer one :s
@@Kronark yeah good luck, hope you finish that thesis early, so you'd be able to relax a bit at the end, and so we could have that cool looking language
Thank you! :) i hope so too ^^
Maybe you should give each line/connection a different color to make the whole project a little clearer. Or use different colored backgrounds for different topics in the video! Love your videos! Keep going! 😊
Connections aren‘t colorable in this prototype unfortunately, and adding that would be to much of a detour i‘m afraid!
But the release build will have ui customisation ☺️
I‘ll try adding some highlighting effects for the dev-logs coming up!
And thank you! :)
bro is posting banger videos like that thinking that i wouldn't notice
Who gave me away?!
@@Kronark I am here very greatful this time to youtube algorithm. A rare sight
I‘m also very positively suprised by the algorithm… didn‘t think I‘d be even remotely in the ball park i am right now
I'm very glad I found this channel! Can't wait for the next episode, because executable formats are pretty difficult for me :D
Glad to hear it!
I think I've gone over the PE format so many times now, I'm really unsure how well it will go until next week. I will probably split one hefty part into a separate video though: the checksum calculation - simply because there are no good sources on that algorithm and It'd be nice to have that separate for easier visibility on search engines... Hope the remainder doesn't result in another 13 minute video ^^
Been waiting for these from that raymarching video, very cool stuff
Thanks for waiting man!
Lovely tea break material, a brilliant rethinking of assembly and information in general
It‘s my pleasure being your tea break entertainment of choice! :)
This is the coolest thing on YT 2024
Thanks man!
I know it's too early for this kind of feedback... but a concept that can help to master the complexity of the network, when it starts to get overloaded, is to adopt balloons that abstract functions, with inputs and outputs, which have an open mode (to visualize the internal network) and a hidden-spam mode to reduce visual complexity... it's just an aesthetic tool, but it can significantly increase expressive power.
Another very useful and widely used functionality is 'while', for me it could be simplified and would avoid programmer errors, if it were based on only 2 types, the range type n (if n
I think your balloons are the core mechanic of this tool... isn't that exactly what we're doing in these videos? Grouping nodes together to form new nodes?
Regarding array implementation, there won't be dedicated graphics for it. We will be using something called a repetitive socket though, which allows us to provide multiple values for a single input.
babe wake up, new Kronark just dropped
hey I love the video but the flicker of things entering exiting is so jarring for me lol
Thanks!
and sorry about that :/
unfortunately those will probably not be replaced any time soon... don't have time to redesign the video aesthetic right now... maybe next year!
So glad I found this, what an amazing project
It’s amazing how well edited your videos are, learning a lot, still easy to follow. You are a rockstar!
Thanks mate! Appreciate it :)
Glad you‘re here :D
babe wake up kronark posted
I've been thinking about visual programming languages a lot lately, particularly how in many respects it could actually simplify language development, i can't wait for you to explain in more detail how you compile these nodes
gonna be a while until we get into that stuff, will focus on my masters thesis until February / March, probably do some more graphics stuff after that and then eventually do compiler implementation again - work on the release product. definitely need a break from compiler design after the prototype is done
@@Kronark I will patiently await this :) good luck on the thesis
I'm a 3D art enthusiast, I use blender and I'm familiar with Nodes.
I'm fascinated by programming and I'm seeing your videos that are very interesting and well done, but I can't understand much about them, what do you recommend me to read or see in order to understand more of what you're doing?
I know a little bit of programming, but I don't know anything about low-level languages.
If you have a grasp on general programming concepts you‘ll find in every language (ifs, loops, math, etc.) and you‘re really interested, i‘d recommend looking into how interpreted and compiled languages work i.e. how your text actually gets turned into something a computer can use. So stuff like lexing, parsing, abstract syntax trees… It all really depends on what you already know to be honest. If it‘s too heavy stuff, I‘d just start with how some data structures work, because you usually have to think about memory and operations etc when tackling those.
Once you understand that pipeline you should somewhat understand what i‘m trying to do here :)
In this video he is encoding x84 instructions. Learn assembly. It is not that hard to write for small projects. Then look at the output in a hex editor.
Do not confuse "low level languages" with "low level languages". C, C++, Rust are "low level languages" that compile to "low level languages" like assembly.
But assembly is more of an intermediate language these days. Frankly. You only use it for a few lines to write a boot loader for operating systems that then are written in C or Rust. Or to accesses hardware functions that then abstracted by C or Rust anyways.
So unless you want to relive the glory days of Wolfenstein 3D or classic Quake and Doom that had in addition to C code some assembly in it in performance critical places because you are John Carmack. Other then thay assembly is not a "low level language" but just another intermediate language that low level languages like C and Rust compile to before they become final machine code.
@@techpriest4787 There are nearly no languages that compile to assembly. Assembly is machine code with direct translation to mnemonics for readability. Compiled languages just compile directly to machine code.
Video quality is amazing, really would like to watch more videos like this
Well you'll be glad to hear that I'm planning on making more! :D
Is your idea that you should be able to build abstractions upon abstractions to effectively make it so you can write high level code that is built up by lower and lower levels of abstractions?
yes :)
This is the coolest channel ever
I'm humbled you think so! :D
Im not sure if this is happening to anyone else but the graphics have this flickering effect sometimes.
Great video im wondering if the format could be changed slightly, i felt like i got lost at the begining/middle and then when you tied everything up with a goal and an example it made more sense. Maybe a graphic that kind of breaks down each concept and we can see where it will be used in relation to the end goal is needed. Its hard to keep many things in my head when i dont know how/where they will be used
Hiya mate, yeah I‘m aware of some weird rendering artifacts - I currently just don‘t know how to prevent them. I may experiment with render settings if I have enough time for the video next week.
Speaking of the video next week, it seams like it‘s going to be similar in complexity to the video this week. So I‘ll definitely try to implement your feedback there - I was in a bit of a rush for the one this week.
Thanks!
@@Kronark Thanks for taking my feedback on board, looking forward to the next video you have my sub :)
Thankyou! :)
Hope I have enough time to actually consider it the next week, I‘ll try my best!
Love the music and intro
Thanks! Really like how it turned out :)
Ghost of Grace Hopper: “Yes… Hahaha… YES!
I do hope so... :)
This is a nice overview of x86!
Glad you think so :D
Really interesting project, at this point I think I'm even able to follow you but im really interested in how you will make the step to compiling actual programs because I don't get how you will be dealing with register usage and not having everything being inline. Im kindof familiar with low level programming/assembly but I have never worked with node based programming. Maybe thats what hinders me right now. I don't want to criticize your way of explaining everything from the ground up but at least for me the bigger picture is really not clear at all.
Looking forward to more videos:)
We‘ll use a global state system built into the compiler for register assignment and stuff like that! It‘ll all become clearer eventually I hope :D
The bigger picture will be more apparent eventually as well ;)
the sound design is impressive
Thankyou! :)
Watching your videos, despite knowing some assembly for years, I realize that I don't belong in computer science, dev or IT of any form... Just wow
Don‘t be so hard to yourself man :) i‘m pretty sure the vast majority of programmers have no idea about what‘s going on here ^^
@@Kronark Too kind.
I barely get it too, but then I'm old enough to know assembly and co. Yet, not skilled enough to do anything useful - not even with high level languages XD
But I'm curious to see where this goes. It can be ANYthing, the potential is almost endless.
Question: GCC and other compilers perform a LOT of optimisations on C and C++ Code when converting it to assembly, which improves efficiency, However, your solution requires the user to effectively use assembly directly. So how would you implement these compiler optimisations?
Users won't be required to use assembly directly for optimisation or anything else. It will be hidden behind multiple layers of abstraction just like in a traditional programming language. However, the difference in this tool is, that you always have access to the lower levels of any given building block - so you *can* use the underlying assembly directly, but you don't have to.
We'll get into optimisations further down the road, but the vast majority of optimisations essentially boil down to "replace x with y, because y is faster". A simple example would be the multiplication of some value by a power of two. Unoptimised code might simply compile that down to an "imul" instruction with a register and an immediate value as parameters. However, optimised code would replace the "imul" instruction with a simple bit shift by the exponent of that power of two value (logarithm with base 2). So in this node editor, all that needs to be done is add a condition that checks the input value - and if it's a power of two, output the bytes of a bitshift instruction instead of the bytes of a multiplication instruction.
This only has to be done once, most likely by me years before this tool releases publicly. Any user after that doesn't have to touch any of these low level details to get more performant multiplication.
In general, a lot of those optimisations you mention are required simply due to the source implementation being written in a human readable language. It's easier for us to split up complicated equations into separate variables for example, which is completely suboptimal for the final machine code. That intermediary translation is completely removed from this compilation process.
That is some cool stuff, visual programing needs some proper love. BUT
But it seems lacking at the 12:30 register is set manually, it would be cool if there was wrapping node around 'add modrm' that contains logic that helps to fill data. So 'reg' can be chosen by name of 'RAX' or 'RDX' instead of writing binary values to 'reg' directly. Even if value '010' comes from external node it seems as a chore to drag that node manually. And if possible the other fields should be filled based on the choice that was already made if it is deciding factor. I'm not sure but if choosing 'reg 010' guarantees that 'mod is direct register' then 'direct register' should be filled automatically.
So some sort of Abstraction Layer on top of nodes to handle Explicit and Conditional logic that helps with filling data in those nodes? I personally wouldn't want to remember by rote the op code in binary not to look at the documentation. I guess it would be wrapped in by some compiling nodes so it still will be hidden from the user with is nice.
I cut out the part which explained that the register assignment problem and any register abstraction such as a mapping of their mnemonics to their binary representation will be a topic for another video... this video was already long enough ^^
You will eventually not specify registers at all, we'll implement automatic register assignment using a global state. But that's two abstraction levels higher than we're currently working at, so it's something we'll do a bit later on.
If reg is referencing a register directly within modr/m, mod can still not be 0b11 (direct register addressing). The addressing mode of mod only controls the interpretation of the last 3 bits in rm. the value of reg is interpreted implicitly by your choice of opcode. So automating the input in other sockets based on a given socket (i.e. set mod to "direct register") would actually make us lose control over the output in this situation.
We're currently working at 2 abstraction layers below standard assembly code - the vast majority of users won't or won't want to touch anything "down" here. Luckily the vast majority of this complexity will be hidden behind multiple layers of abstraction once we get to standard high level practices.
@@Kronark The conditionals on the "direct register" I mentioned was an example 'if it is possible' as I have not enough understanding of the Assembly to know that such condition makes sense. And as you explained, it does not.
As there is some form of abstraction latter then all is good. I can't wait to see how you will solve it, it is nice to watch.
Still I wonder how the high level case of let say of providing Keys for the Interaction with keyboard will be made. Naturally the simplest way is to treat Key Codes as Integers they are and just provide them Manually. Then the next step of abstraction will be to use 'mappings to key names' and that is equal to what is provided in txt programming by enums or or other named static variables, albeit potentially faster depending on the compilation results.
What makes me biased is probably the fact that I'm fan of the 'Widget Approach' for the visual programming with allows to naturally get better abstraction than text programming. In the Key example it would be fully rendered Keyboard from with user can select Key that will be used. properly done User would be able to specify if selected is the Key for it's value or because of it's position on keyboard with would resolve the problem of 'QERTY/QWERTZ'. Or multiple key need to be pressed conditionally. But that is just my personal preference. I'm fully aware that widget creation is not only complicated but also time consuming, not to mention the baggage that would need to be carried over all the implementations during prototyping phase.
Yeah, with most topics / features you can expect that we‘ll take the route most programming languages used e.g. „enums“ for the key codes. We‘ll add more and more abstraction layers over time, which may result in such a key selection interface eventually :)
thank you..
woukd really appreciate if you attach links for additional resources or so...🙏
There really isn't much else besides your standard wiki articles + the official specification to be honest.
I'll see if I find the time to add links soon.
Can't wait to get my hands on this!
The wait will be over sooner than later I hope :)
Could this be called a compiler? Seems more like a node-based assembler. Great work either way! :D
It'd be an assembler if I were limited to writing assembly. However I neither have to compile into object files, nor do I have to build assembly instructions with this software. So no, not an assembler.
It is a compiler by definition. It transforms a source representation into one of various target representations. Assemblers are limited to assembly by definition. :)
And thank you! :D
@@Kronark Ah... very cool! Yeah, this project is super interesting. Something that would make me lose by breath with amazement as a Linux user is if you adapted this to also work in a terminal. XD
Glad to hear it!
How would a node editor work in a terminal though? :P
@@Kronark Uh... good question... maybe similar to how it works as a GUI? It would just be sharper and a lot more grid snappy.
Hey i realized you have three different playlists on your channel each one video long for now, i think it would be useful especially when your channel will have more uploads to make a playlist with all your videos for convenience :)
True! Was focusing so much on staying organised, I never really looked at the channel page - i‘ll make a general playlist with all videos in order of upload
@@Kronark nice 👍🙂 i love watching these videos
The video is cool, but I have some questions about the project:
Which programming language/framework did you use?
What are the base nodes for the project (for example: add, rest, and, not)?
Hey, thanks!
This prototype was built using electron.js, so javascript is the main language.
There are no built-in nodes that provide any sort of programming functionality. There are only nodes that tell the compiler what to do with any given string of bytes. All programming functionality will be emergent using the built-in nodes.
You can find a list of the current built-in nodes in my introductory video uploaded last week (the one with „this is machine code“ on the thumbnail)
Is it bad if I do not understand what is going on at all 😅 But I think you said it will become higher level eventually. I still love these videos!
It‘s understandable, I think you have to already be thinking about low level stuff like this to sort of see what I‘m trying to do - it‘ll definitely become more comprehensible later on! :)
Thanks! :)
Hey are you willing to open source your data creation suite? Or at least allow people to download and play with it?
I’m having trouble following as I’m not familiar with the software, however this seems super cool to play around with and wrap my head around.
In fact it might be cool to follow along with your implementation to get the hang of this.
@@chrisjones9132 Hey man, unfortunately that won't be possible for the time being, no. But the next videos I'll be uploading will explain how the software works, so maybe that will help! We'll be covering all the different nodes and the compilation algorithm itself. :)
This is pure gold. I suppose this will be an open source programming language right?
The software may or may not become open source eventually, it will be free though. The node networks will be open sourced via a public node sharing plattform sometime in the future :) i‘ll probably ship the fundamental nodes like the ones in this video with the release build.
@@Kronark I already want to use it! Thanks for planning to provide it for free. I feel like it could be a very good entry point for teenagers interested in learning programming 🎉
Definitely! There would be a lot more programmers if it wasn't so intimidating like it is now :)
12:36 Actually, that is only in Intel syntax. In AT&T syntax the target goes last.
Intel: operation destination, source
AT&T: operation source, destination
Sh*t, forgot about that. But at least it‘s in line with what‘s shown on screen - would‘ve been worse if I had used AT&T and said the intel order…
very cool information, the only criticism would be the background music.
What‘s wrong with ma music? :D
@@Kronark The music itself is fine, it is distracting for a technical video in my opinion, though others might not notice :D
Ahhhh yeah I get that, makes things a little less dry in my opinion though :)
@@Kronark whatever you fine with, I'm here for the content anyway.
Are you eventually going to put something up on github or something like that?
Probably, but that‘s still a bit in the future.
Great video(s)! One question: is it possible to make the system in a way so that it can output a multi-os executable/executables for multiple OS-s at the same time? Sort of like the Cosmopolitan project for C++.
Thankyou!
Yes, that‘s actually pretty straightforward. If you add multiple file nodes, they‘ll all be compiled. You can channel the same byte sequence into different files natively with this tool.
We will need to add a level of abstraction to actually make the same algorithms be output for different architectures / OSs e.g. that one outputs ARM bytes and the other x64 - but it will be possible natively using a state system.
Still need to fix some bugs with that system currently though - good thing we‘re only going to have a single target architecture and OS for a bit.
Brilliant
Thankyou! :)
rex
Yes
Darn! I want this NOW!! :|
feels kinda stupid going back hacking Python 24/7
that is exactly what I've been feeling like for the past 2-3 years - using python and java in uni was an absolute pain when I knew I could do so much better if only I had this tool :D
amazing! although I understood very little. The process of building such a low level design is really interesting
.
ps: ur style of editing also feels nice :)
I‘m sure the topics later on will become increasingly familiar, once we reach higher levels of abstraction :D
Thank you! :)
Oh my god. How many hours do you spend on this project per day?
not to many actually... the programming is done in a day usually, the video editing takes a bit longer rn, especially if its so long like this one
I don't understand anything in the video. I'm not even a programmer. I'm subscribed.
Welcome aboard either way! :D
@@Kronark In all seriousness, great work. As someone who is interested in the general idea of "programming" but hates "coding", this visual approach is just attractive to me. Even if I never personally use this, I'm excited that this may now be possible.
How exactly do you create applications with this software? I understand that each node is a command, and many commands create an application, but since you are using a high-level language (JavaScript if I remember correctly), won't you have to interpret the nodes some way to create an executable? I believe that you said in the last video that the program is able to decide which nodes to interpret through perceived changes since the last compiled version, does the program translate the nodes into assembly, and then compare each line of both versions to find differences, or do you have another way of doing that? It's a bit confusing since I do not know the underlying structure, so I can explain my question in more detail if you want.
Just got to the end of the video, I guess I will have to wait a week for the answer 😅
In a earlier video he mentioned this project can be used to generate any type of file from png to bmp to exe. My guess is that in the final step of compilation it prints some bytes to a file which results in an executable.
While the prototype is written in javascript, the nodes don‘t output javascript to be interpreted. Each node is a simple object on the heap with an array of sockets associated to it. To detect if a node needs to be recompiled / the input has changed, a simple flag in the socket data structure is checked -> if it‘s state doesn‘t match the compilation iteration, it needs to be recompiled.
So no expensive string comparisons needed! But you can think of these nodes as a weird ui for building strings of text. To make an executable, some specific text is put into a file - but like you said, we‘ll get to that soon.
We‘ll probably not get around to actually executing our first exe next week, since a core component will be split into a separate video - but we‘ll get there eventually!
This one gets it!
Exactly. All these nodes do is create strings of bytes in different ways. And we can combine them to build a byte sequence resembling an executable!
Alpha
Are you calling me a buggy, unfinished version of what I'm supposed to become?!
@@Kronark No, I'm saying that i made the first comment. Great videos btw, love the aesthetics and presentation
I was joking mate! :D
Thankyou :)
Sen var ya sen... Adam adam
Bro what? :D
@@Kronark xD. Good content, good presentation. Can't wait to read your thesis
Thanks! :D
👀 how's he gonna implement control flow? Or is it gonna be branchless...?
He's got the plan to build an "if" node that takes a condition and two code blocks as arguments, which it then will use to construct jmp wrapper around the code blocks based on the condition - that's still a bit in the future rn.
Dude I just finished my undergrad. Don't make me feel uneducated.
Greate work
Thanks!
Isn't it known as x86-64
it's weird hearing someone call it x64
X64 is the short form :)
What's the music called?
There's a music note at the beginning of every video, directly after the title sequence :) The link to the artist's channel is in the description
Will this ever become an open source project or something we can use?
I‘m tending towards not making this software open source. But there will be a release version eventually, once all bugs have been fixed and everything has been optimised. We‘ll develop that release build using this prototype during a dev log series on this channel.
In short, you‘ll be able to use it eventually, just not any time soon.
les go
i hope the next video is a little more digestible lol
I hope so too, but it looks like the PE file format is gonna be a lot of stuff again. It‘s gonna be a bit simpler in terms of what each byte means tho.
It‘ll all be way more digestable once we get to single operations per video like addition and printing to console instead of such abstract formatting stuff!
How will compile time optimizations be done in kronark? This looks like each operation will simply be deconstructed into its primitives and written to a binary file. Is there going to be any kind of optimization?
The neat part is, with this tool we‘re moving all compile time optimisation into development time. All optimisation is completely in user control.
Since we‘re designing our own abstraction layers, we can decide which actual machine code is emitted based on certain inputs to a node.
So for example we can control to which degree a loop is unrolled based on the amount of instructions inside the loop body. We can emit a bitshift instruction instead of a multiplication instruction if the factor is a power of 2. And so on….
It‘s all at our fingertips, nothing is hidden behind a black box implementation. This will be tedious in the beginning, but we‘ll eventually get on par with modern compilers and then have all the freedom to grow beyond that way more easily.
Wouldn't this make it super hard to build reusable abstractions?
This makes kronark basically a very fancy assembler
I honestly can‘t think of any abstractions that can‘t be made reusable using a separate node definition :)
Exactly! With the bonus that you can compile to anything and also add new compilation targets on the fly :)
Kronark is the name of the overarching „business“ i want to build btw, this is just the „Kronark Compiler“ - name not found yet, will tackle on release build ^^
Is there a way I can contribute?
Check your other comment :P
will this be open sources when its completed?
Probably not, or at the very least only with a restrictive licence. You'll be able to follow along on development of the release build though, in case you plan on making your own!
Can u add some bezier curve or some thing la that for the link / wire?
I can, but i won‘t in this prototype. To much work for something that‘s only aesthetic. Visual customisation will be in the release build though, including connection shapes :)
This looks soo cool! I only wonder why bother with exe instead just making dll's?
DLLs still need an exe to actually run as far as I know :P
DLLs don't provide an entry point which tells the OS where to start executing. Besides that, they both use the same overall file format anyway, so why bother with DLLs right now? :D
@@Kronark Ok, you got me xD I thought of this as a way to provide fast tools or libraries. My first thought was to make a dll that could be executed by node-red for example. EXE just felt so limiting, because you can only use it on windows. Can't this produce a code that can be interpreted on any system? But then - it doesn't produce code... So it will always need to pack the output for the target operating system. I don't think I fully grasp the concept yet. I'll surely watch more to fill that gap.
The neat thing is, it CAN produce source code as well. But other than that, we‘ll obviously eventually add support for ELF and maybe even Mach-O files, so other OSs will be targetted as well.
Maybe the next video will clear some things up!
I keep blacking out 'cause I don't understand any of this shit but it should get better as we go to higher levels, right?
don't worry, absolutely! The goal is to reach a level of abstraction akin to python, just without the performance deficits.
@@Kronark Fast Python?! Sign me up!
I just want to make a chess game using Unity, now I'm here
I hope that‘s a good thing ^^
i have an x86 processor
As in, 32-bit?
@@Kronark No it is 64-bit
Then you‘ll be able to run this stuff :)