The Case for MIDI Music in Games
ฝัง
- เผยแพร่เมื่อ 26 มิ.ย. 2024
- In the years since MIDI music was officially deemed "obsolete" by the games industry, the technology has quietly continued to exist as an essential part of music production. In this video, I ask the big question: is it time for developers to re-adopt MIDI technology in games?
- เกม
Hello everyone! Thanks for checking out this video on MIDI music in games. I've gotten quite a few comments taking issue with the way I discussed SNES music, so I'd like to append what I said in the video. At one point I claim that SNES games stored their music "in the SPC format", which is perhaps questionably-worded, but it's still a stretch to call this false.
The .spc files you'll find on Nintendo fansites are a dump of an emulator's RAM data that captures the notes being sequenced. Those are user-generated files based on running the final, compiled release cartridge, which is obviously a different set of information than the developers would have used.
However, the larger issue with my phrasing is the implication that SNES games stored their data in discrete files, which is incorrect. According to the official NOA SNES Development Manual (section 3-7-9 or page 176), note sequencing information could be written directly onto the game's code, and applied to "voices" much like the MIDI format. Developers could either work with Nintendo's own audio tools or create their own playback engine. The music isn't stored in a separate file, which is why fans have resorted to capturing RAM dumps through an emulator rather than just grabbing the information from the ROM (which you can *somewhat* do with a few N64 ROMs).
As well, a few people have taken issue with my apparent dismissal of dynamic audio in modern games. There are absolutely some effective examples of devs having used rendered/flattened/bounced audio stems from a DAW and using intricate event triggers to sequence the music based on gameplay events. While this technique has led to fantastic game audio, the purpose of my video is to describe the benefits of rendering all the audio directly in a game's engine, by writing the notes into some variant of the MIDI format and using virtual instruments that run on the end-user's hardware (UE5's MIDI plugin and sine wave generator being an interesting example). It's not the way the industry currently works, but hey, that's why I made such a long-winded video full of suggestions.
Anyway, thanks for checking out the video!
Video games still mostly rely on sequenced audio to make the music to bounce it down. And the processing required to process large sample libraries is yeah please for the life of Christ don’t bring this back
super pedantic of them to say that. doesn't matter if the language is MIDI or something else or how exactly the various notation languages work. they all just make sure games can synthesize sound in cool ways and your point came across loud and clear
Wait I don't get it
So few games put sequence data in the Sample RAM and others didn't?
The thing is,midi…was never in consoles.midi use external proprietary sound database (Yamaha ,Roland …) .games in the other hand was loading those database from the game and ram.
Making console music closer to tracker music… this is why console and pc music sound so different. It you want music close to the snes,the sega cd or n64 on pc you need a amiga pc,or a Gus sound card.
And for ff7 the game actually uses sequenced music pretty much all the time since the game use the cd player to load backgrounds and video files. The only red book audio the game use is when it read a video and one winged angel…
@@snesmocha It could still use soundfonts, which are very much in the realm of possibility for modern PCs. Either that, or Unreal style Tracker music.
I have an obsession of downloading videogame music as MIDI and playing it back and hearing each instrument separated.
how does one do this?👀
@@chanotv1025 I also must know this
Oscilloscope Deconstructions are Sick
You're not alone my friend.
@@chanotv1025 You can do this in any kind of MIDI-making program. Anvil Studio has worked really well for me. It allows you to mute and isolate individual tracks in a MIDI. It's easy to use and free to download!
5:50 You are MASSIVELY underselling the prevalence of sampling in cartridge games. Several NES games used sampling, for example Kirby's Adventure and Super Mario Bros 3 used sampled drums to aid the Noise channel's percussion, and basically every Genesis game used the DAC channel to play percussion samples
You’re correct about those games, but I’m not sure this particular distinction is all that meaningful. Sampled drums and midi drums are the exact same thing, it’s just incredibly short clips of audio being stored as different notes, and then the software playing them back in a sequence. MIDI samples always have come from somewhere, unless they’re just raw square waves.
@@ComputerLabHighjinks not for consoles with poor sampling capabilities like the MegaDrive or NES. A lot of games on the MegaDrive DID use sampled drums, but an equal amount used noise, or FM drums. Also I don't like how you referred to most sequenced music as MIDI in the video lol. I get that you did that out of convenience but it annoys the crap out of me as someone interested in sequenced music.
Mega Drive*
I'm gonna give few extra examples
There was Batman: Joker Returns, Mr Gimmick & Hebereke from Sunsoft that used DPCM for a Better and More Complex Bass
And there was also Earthworm Jim & Toy Story which used Software Mixing on DAC
Uh there are plenty of songs on the NES that sampled more than "just drums." People added bass lines with the channel as well. There are a lot of people that also still do really cool things with samples in Famistudio, famitracker, etc. You can still do this today, and also keep things within the constraints. So while vocal hits would not be the best on NES but limitations breed innovation
As a musician and game dev I have to point out a few things. We haven't stopped using sequencing in games, far from it! We use it all the time, tools similar to what UE5 now does have been around for a while, the standard begin wwise, FMOD being another popular option. They allows us to implement all sorts of events and logic to react to the game in real time, much like what midi would allow you to do. The thing to keep in mind about these wonderful VST libraries is that they're HUGE. The reason they sound so good is because they encapsulate samples for different instruments, notes, intensities, playing style and so on... which adds up to be usually in the tens or even hundreds of gigabytes. At that point it makes much more sense to combine the sequencing capabilities of the tools I just mentioned with pre recorded audio. You could achieve anything you want, it's just a matter of how far you're willing to push it really.
I'd be interested in your feedback given your experience. Are the MIDI capabilities of tools like Wwise, FMOD or Elias frequently used?
Because from what I've seen by looking into them, they don't seem to fully take advantage of MIDI & live synthesis/sampling. The big advantage of MIDI is that notes can be generated or edited on the fly, depending on game events and data (which goes into algorithmic composition, and thus quite farther than a dynamic layering and sequencing of pre-rendered audio stems, which seems to be one of the main use cases of FMOD for instance). Yet what these audio middlewares do in terms of MIDI is just playback. To be fair, this already presents some value, I can notably think of two reasons:
- better control over playback speed (change the BPM without having to do pitch shifting)
- smoother transitions (eg. let a note release or a reverb tail flow over a transition between two clips, instead of having to crossfade)
but besides that, if you are planning on only using layered "static" clips, then indeed for all the reasons you gave I don't see the point of going through the trouble of using live MIDI instead of pre-rendered audio in a game nowadays, even if that means pre-rendering the same MIDI clip several times with various instruments. (Though sample libraries' size might not be a problem for every game: not every game soundtrack needs a full-fledged realistic-sounding orchestra, a lot of recent indie games just rely on pretty SNES-y sounds anyway and still get a great mileage out of them)
Am I missing something regarding the MIDI sequencing capabilities of the tools you are mentioning? (I don't know anything about UE5 for instance)
You're right when noting middlewares don't take full advantage of what MIDI has to offer. The old games running on Atari and what not didn't either!
Wwise and FMOD do allow you to create and set parameters for the events you trigger, so in a sense if would be perfectly possible to create virtual instruments and generate music dynamically, similarly to what Ocarina Of Time did, but in practice if you're looking for a fully fledged instrument that would prove difficult, for a very minor benefit.
That would mean creating a full sampler or synthesizer inside your game, and MIDI wouldn't solve that problem: you'd still be left with all the work ahead of you, as MIDI doesn't have a sound engine, as opposed to the tools I mentioned. Which is perfectly fine, because as you noted no game needs a full virtual orchestra, so usually the use of stems is perfectly acceptable, and the tools at your disposal will allow you to go very far. It's worth noting that wwise does have synth plugins you can use, so procedural sound generation is totally possible.
Then again, it's all a matter of what you need. And also performance, because as you can imagine your CPU is also busy running the game as a whole, and synthesizers can become very computation heavy if you're not careful, so even with the hardware we have today, using stems wouldn't be a bad idea unless you REALLY need that kind of granularity. In my experience, very few games do. Think of a game like Celeste: chip tune, 8-bit style soundtrack. Seems simple enough, but there's a lot going on in Lena Raine's music, and generating all of that on the fly would be un reasonable, not to mention useless because the game doesn't do anything music wise that requires that kind of control.
There are physically simulated instruments now that don’t take up a lot of space at all though
What you gain in memory usage you lose in processing. If you really need a full physical modelling engine for your game and you can afford it, then that's great. Most games do not, and it would simply be a waste of performance.
What usually takes up more storage? Sequenced music split into multiple stems? Or a Sequencer Framework along with Complex Orchestra Sequence Data?
This video needed to be 10% about MIDI and 90% about tracker audio.
Hoping to make a follow-up video at some point! Although, in fairness, I feel like the title matches the subject. Instrument stems and shorter clips have tremendous utility, but this video is more focused on generating audio from within a game’s engine.
@@ComputerLabHighjinks : "...this video is more focused on generating audio from a game's engine."
Wavetables, transwaves, and some other technologies used in professional instruments are essentially specialized samplers, so if you want to "generate audio from a game's engine" you could absolutely get a lot more out of music trackers provided certain features were implemented.
*Edit:* Also, music trackers like Deflemask demonstrate that even when sticking to traditional feature sets there's still a lot that can be done with a music tracker.
None of these old consoles used the MIDI protocol.The MIDI protocol was used (and is still used) for musical instruments and sometimes on PC, but NEVER on consoles, which relied on either sample based (SNES/PS1) or synth based (NES/Genesis) sequenced music.
Both sample based and synth music is composed by MIDI.
@@mdjey2 no, you are confusing sequencing with a protocol :) MIDI is a specific protocol and not used in old consoles. :)
@@cherrystarscollide No I am not confusing. You are making argument that whole music is just composed for video games, which is not. Midi is the standard for sequencing synthesiser and sample music.
@@mdjey2 MIDI is a specific form of sequenced music, which is indeed not used by things like the SNES or NES.. However, converting sequencer data from one format to another is usually trivial. The uploader used MIDI as shorthand for all sequenced music, which is incorrect, but a fairly common colloquialism too..
That's why 3:02...
Im a producer and im confused, everyone uses "midi" in music now, we all use ableton, fl studio, bitwig etc etc... so why are people acting like midi isnt the norm? Are they just talking about the way games USE the audio?
Note sequences is not equal to midi in all cases. Seems that the author of the video is confusing midi with mod music made with sound trackers. This is why people complain.
FL studio doesn't use midi directly using native generators like samplers and synths.
I think they mean the game playing back the actual module/ project file/ .mid sequence in real time, not just as like a audio recording
Music made for games uses MIDI very often, but the games themselves only play the resulting recordings.
@@Eichro also HEAVILY depends on what was playing your mid files, It could drastically different from midi card to midi card (maybe even module like an sc 55)
@3:15
This is incorrect. The SNES does not store it's music in .spc format. Developers made music on the SNES by programming custom engines for audio sample playback.
SPC files are just partial captures of the contents of the SNES ARAM given a container for playback on modern programs. The SNES is incapable of processing these files unless some external program or hardware decodes them back into usable data.
Also, in general, pre-2000's consoles did NOT used MIDI files for music playback. As mentioned before, they stored audio engines, which communicated internally with the audio chips and data within the cartridge or disc.
While it is *technically* possible to play MIDI on old systems, you need both external hardware and software to do so (sometimes not even official), as no old console counts with the necessary program or hardware to process such files internally. Not to mention, MIDI file sizes far surpassed the available memory at the times.
MIDI files for gaming were mostly a PC thing, as they do have the programs and memory to process them.
though techically correct i feel like calling all the pre 2000s consoles sequence methods "midi" is a pretty darn good short hand.
@@charlesgreenberg6956 if MIDIs had never been used in video game hardware, i would just default to midi but like op says- they have. I just call it 'sequenced music' or 'sequenced audio'.
@@SianaGearz Pretty much. MIDI was indeed used using the authoring process to facilitate music sequencing with the console, but it should be noted that it always needed an external device to do so. There's a distinction between using external devices to send playback data from MIDI into the console, and the console being actually able to process MIDI data on it's own.
One curious case of this would be Castlevania Chronicles for the PS1. The original release on the X68000 did support Roland MIDI modules, thanks to the PC platform it was released on. The PSX release contains no MIDI data due to the console being unable to communicate with a MIDI device (without a third-party converter), and therefore the game emulates the Roland MIDI tracks through it's own custom audio engine (along with the new redbook music).
@@charlesgreenberg6956 On 8bit consoles, space and CPU cycles were too short for MIDI per-se. Games had weird custom text-base input formats, some based on BASIC music functions (MML, typically used by Japanese game companies), some based on assembly language (most British tooling). This is also true on SNES (all the music and sfx had to squeeze into 64kb on a 8bit sub-cpu), NEO GEO (lots of MML), and sometimes true on Genesis (ex: Streets of Rage is MML, but western devs often used GEMS which is MIDI-based). Obviously there's also the whole family Tracker-based music systems which comes from the Amiga but showed up in a lot of PC and GBA games, and is rather incompatible with MIDI just because it does everything differently.
@@boptillyouflop I’m aware of the details it’s just when your explaining a broad concept like sequenced music on a basic level it’s abit clumsy to specify what the nes is doing compared to the n64 when what matters for explanation is that the sound is being generated at run time and can thus be altered at run time
Midi is a good understandable short hand for music being generated at run time even if we’re glossing over some system details
i don't know if you knew this, but the entire Plants VS Zombies soundtrack is a sequenced tracker module. The file contains the entire soundtrack, and only takes up 2.12 MB.
It sounds like that on the DS version, but I'm not sure about the moblie/PC version
@@Kuba-xw1mw its true for the PC version! It's a file called mainmusic.mo3, the game is all sequenced music
@@Kuba-xw1mw It actually is a tracked module on PC, but on mobile, I'm not sure it is. I've extracted and messed around with the file on OpenMPT some time ago by myself, tho I don't remember the name of the file.
@@RandomHomoSapiens probably XM or MOD
ogg audio format can get very small too, i can put an entire orchestra track into like a few mb and still have super good quality sound. That's what fmod renders the audio into for the Audio build files for your games.
I remember being a kid and finding it fascinating that when Mario mounted Yoshi, the song would have additional bongos on top. I work as a game composer today haha.
Just curious what kind of education did you need to get into that field?
this video is so full of inaccuracies that it's actually upsetting
Sequenced music, while rarer than before, is definitely still used in modern games (really only by nintendo, but that's obvious). For example, very WarioWare game has sequenced music, even the ones on the switch. This is so the speed ups can be done properly without changing the pitch of the music. These songs can range from very obviously sequenced like the Mario 64 slide minigame music from Move It, to practically unnoticeable like Penny's Theme from Get It Together.
That's just fully sequenced songs though. Pretty much any time there's sound effects harmonizing with the music in modern games there is usually some sort of sequence data following behind.
Wait, how come Trackers/Modules never got mentioned once?
I was thinking the same thing... 🤔
This is common for folks who overuse the term MIDI
The people who feel nostalgic about old games today seem to be mostly form America and played with consoles instead of personal computers.
@@tommj4365 So don't some trackers use MIDI? What does Renoise uses then? What does Polyend tracker uses?
@@mdjey2 some modern trackers may use midi, as an optional feature, but typically trackers don't rely on midi for anything
Just from barely researching, I found that Final Fantasy VII on PS1 is not audio files, it is indeed sequenced, but it uses a different standard from midi, AKAO. There is a lot of misinformation in this video, I don't know what your sources are, I would like to see them. No one ever really stopped using midi/tracked/sequenced audio, though many games use dynamic audio and sequencing that isn't the midi standard. Midi 1.0 has lots of limitations such as only allowing 16 channels, 7 bit values (0-127) for all parameters.
There's lots of other misinformation in this video. There is so much fascinating information out there about sequenced and dynamic audio, but very little of it is in this video. I fear that if the algorithm takes hold of this video it may in fact cause more people to be less-informed of what you're trying to vouch for in this video.
I would suggest you look into the MIDI 2.0 Standard and Wwise by audiokinetic.
Thank you so fucking much I thought I was going insane with that ff7 thing
Also fmod is cool too :) that’s what pizza tower uses along with a lot of cool indie games!
MIDI 2.0 is brand new and barely used anywhere yet, no need for anyone to look into that for such a video.
good chance his sources are AI since a lot of the stuff he mentions is extremely wrong
@baggagelizard He does warn right at the beginning of the video that he is using the term MIDI loosely for "sequenced music tracks in the broader sense".
When he says "MIDI should be back in games" he is actually saying "more games should start again to rely on sequenced, live synthesized/sampled audio rather than pre-rendered audio", the former having indeed undoubtedly declined a lot since the N64 era.
Also even if current tools like FMOD can deal with MIDI data and do live sampling via soundfonts, it doesn't mean that a lot of games actually use these features (virtually no FMOD tutorial here on youtube touches on them)
> I would suggest you look into the MIDI 2.0 Standard and Wwise by audiokinetic.
I don't think Wwise supports MIDI 2.0 (very few software/hardware support MIDI 2.0). Also most of MIDI 2.0 features make little sense in the context of a game, they are about easier interconnection between different hardware devices.
> Midi 1.0 has lots of limitations such as only allowing 16 channels, 7 bit values (0-127) for all parameters.
Higher resolutions for MIDI parameters have been around as MIDI 1.0 extensions for a while (NRPN, 14bit CCs, etc). The 16 channel limitation is irrelevant in the context of modern MIDI software which don't use one single MIDI bus (eg. in any DAW, each track is its own MIDI bus, which means you actually have 16 channels _per instrument_). MIDI limitations are not really a problem, but again MIDI itself is not really the point anyway.
The PC port of FF7 from 1998 used MIDI files and a SoundFont so I'd be willing to bet the AKAO sequence format used by Square on PS1 games was probably similar enough for them to convert between the two formats with ease.
These techniques are used in modern games all the time, often using audio middleware like FMOD or WWise. For a modern example, if you listen to the sound that plays in Mario Odyssey when you travel through a wire, you'll hear the arpeggio sound changes with the chords being played in the soundtrack. These are really fun things to do with dynamic music, but midi is not particularly necessary for any of them (other than maybe the cute band in Conker's Bad Fur Day.) I think it's hard in most cases to justify the performance cost of using modern software instruments over just rendering out the parts in layers and manipulating those. (Edit: didn't see the pinned comment. I still think the performance cost is hard to justify unless you have something you really just can't achieve without building software instruments into the game)
Fmod mentioned
There is one problem I can see with the pure MIDI based implementation with VSTs. While there are plenty of good free ones out there, and the ones built into some game engines, most of them have a license fee attached to them. You want to use the VST to make music in your DAW, you have to pay. Problem is, if you want to have that VST built into the game file, you’d have to pay for a copy of the license for every copy of the game sold. And it’s pretty self-evident how infeasible that is. So if the composer has a particular VST they like to use, then they have to record it to audio.
I know of a somewhat modern game that has dynamic audio, midi sequenced stuff, and streamed tunes for space/storage reasons that actually use their own studio recorded sample library
the audio director mentioned licensing issues using samples from say east west or something by having raw samples from those companies in a game.
the game has live music tracks, so they were able to get the band to record samples for midi sequencing, and it managed to work out actually pretty well.
@@goatsoup Yeah that’s probably the best approach, sample the professional VSTs or live musicians and then use the in-engine midi sequencer to dynamically play back the samples.
@alexgrunde6682 i mean in those cases that was done for memory/storage limitations, i imagine most games would just stream it.
The reason games mostly stopped using "MIDI" (aka:sequenced music) is because you need exponentially more CPU and memory resources to match the quality of a pre-recorded piece. Yes, most non-live music is authored using "MIDI", but they use humongous sound fonts which can go into gigabytes and audio filters which make the Nintendo Switch CPU cry.
Ironically, music in modern games is usually far more dynamic than when it was sequenced on the fly. Things like several variations of the same track which are played in sync and transition into each other like in Hades and tracks that are cut into several pieces that can switched around seamlessly like in Doom Eternal.
Also, 32-bit consoles were great at sequenced music. For example, Chrono Cross' amazing sound track is fully sequenced by the PS1 sound hardware.
This was definitely true historically: not only was it about competition for compute resources, but "consumer audio" on Windows and Android was, for a long time, exposed with some shoddy, high-latency APIs that inhibited using those platforms like a professional DAW. The industry was not investing in the "audio guy"(and studios of yesteryear usually wanted one "audio guy" doing everything).
The argument for not doing "live" sequencing now has diminished because we really have a lot of hardware to play with, applying it all to graphical fidelity isn't doing what it used to, and there's more interest now in getting a low latency experience. If we imagined designing a hobby game console around cheap microcontrollers, as is being explored in the retro space, it's reasonable to think of adding a single board audio device like a Daisy Seed as the audio processor. The BOM is lower than what old sound cards used to cost, and there's no operating system getting in the way on those little MCUs.
@@JH-pe3ro it still feels like a waste of CPU time mixing and filtering several audio voices to produce music that could be done using a single stream, when you could be using all those channels for sound FX and environmental effects instead.
@@manoelBneto Well, decoding MP3 or OGG eats as much cpu as mixing something like 32 channels of music. It's not as much a cpu-use problem as an integration problem.
Although I’d agree that is the main reason why midi (sequenced music) died out it is kinda hell to try to recreate even the systems from the n64 using n64 samples in wwise generally speaking wwise’s midi implementation is kinda lacking.
Wwise as an audio tool is fucking awesome and I’m very grateful I’m getting into the industry now where it and fmod exist but if I was to try to do what banjo kazooie does. I COULD do it but it would be much harder then if I had the tools in Wwise to have midi play at run time
The reason being is that audio files don’t gracefully end. If you have reverb baked into your track you have to bounce the score a bar at a time. And even then that limits your transitions to a bar at a time (if you cut off the reverb it clicks and sounds generally bad)
Where as with midi you can cut and change and go up key and do all that because your generating your music in real time. If Wwise had a more mature midi implementation we could be doing some really cool stuff with it (considering that the inbuilt Wwise reverbs are already used on switch and arnt heavy on cpu and some games do use the basic Wwise midi implementation )
I’m sorry for the super long reply this has been my fucking life as I’ve gone through school and learnt Wwise shit bothers me abit. The rest of the program great but damn if the midi implementation isn’t abit half baked
@@boptillyouflop the 32 channels of music isn’t the problem. If you were using “realistic” sound packs like the strings he mentioned in the video you would be wielding super high resolution strings with multiple velocities,expressions etc you would have to have afew gigs of ram ready just to load everything. That string pack is like 40 fucking gigs aka the size of a moderate triple a title
8:31 THAT'S INCORRECT!
FINAL FANTASY 7 DOES USE SEQUENCED MUSIC!
Tho STREAMED audio was quite popular on the PSX. The PSX could also do SEQUENCED music.
FF7 use sequenced music because of the sheer amount of tracks.
Not only that, but Squaresoft continued to use sequenced music into the PS2 era due to capacity concerns.
Was about to say the same thing. I think one of the re-releases of FF7 used recordings from the original PSX release.
The CPU power and storage needed is way beyond what we would need to play a simple MP3 file now. Yes, sample libraries like East West can sound convincing, but these complex DAWs that are used to create this music require beefy machines with huge amounts of RAM and fast storage.
The environmental examples like you see with Banjo Kazooie can easily be accomplished with audio crossfading, without needing to rely on MIDI.
Pianoteq synthesizer sounds exactly the same as a real piano and doesn't reach 100MB in size, but you need an UFO CPU to run it.
I basically don't use samplers, my CPU can generate a realistic synthesized orchestra, it's a luxury that a lot don't have.
Yeah about every game today has complex systems to make the music follow dynamically what's happening, it really has nothing to do with midi at all
"MIDI"/Sequenced audio is still very much part of game audio to this day, it's just now a much smaller part of the pie because it's not really needed as much as it was back then. Game audio has evolved to sport so many different kinds of ways to do dynamic audio.
Also, your idea of the weaknesses of streamed audio would've made sense in the 1990's. Back then, very early CD-ROM hardware was very slow, even some cases where looping a track wasn't even feasible (most notable PC ENGINE/TG16 comes to mind), things like increasing tempo, dynamic layers, anything like that was just not possible (or not easy at least) then.
But now? Streamed audio is just as malleable, if even more than sequenced music ever was, and game developers are 100% taking advantage of the processing power of newer hardware with sound.
In a sense, games are still at times basically sequenced!
just now with dynamic layers, sections, with realtime effects (which you touched upon) and much more .. which unfortunately, doesn't get enough love/coverage, it's really a shame wwise didn't make it in this video despite being such a popular middleware tool, that *can* do sequenced music and tons of dynamic game audio stuff and there's a ton of examples of games that use it- I think that's the 'new tool' you were kind of yearning for but weren't aware existed ^^
I think it's really unfortunate that you've phrased that musicians aren't open to adopting new tools when with wwise (and other audio middleware), I can think of countless musicians and people working in game audio that have been at the forefront, been interviewed, done GDCs showcasing some really adaptive, dynamic, and just cool. audio tech to audiences, even some still using MIDI/sequenced audio as well!
Also a lot of those examples you brought for MIDI are.. still done in games today!
Mario Odyssey has internal sequence data for every major area song to make sure sound effects harmonize with the music/helps the game keep track of that, and then obviously the town tune is still in newer AC games. New Super Mario Bros. on Wii/3DS/Switch also stores game sequences in the files to make sure the game responds to the "bah" sounds. Modern game audio tools can easily fade songs in and out, speed up music, even internal midi/sequence-based system working under a streamed piece of audio would totally work for trying to do mother 3's audio too!
Yooka Laylee, the spiritual successor to banjo kazooie could also achieve the effect in BK without needing midi at all!
All of that is the beauty of modern game audio, we can achieve the same things as these old games, but without sacrificing audio quality or being too costly on the game!
So to put it again- Game audio today isn't just midi or streamed- it's both, and so much more. Game audio is a lot more complex than it was 25 years ago and we have so many more options and tools now, sequenced audio is just one of them! ^^
Minor notes-
MIDI, the protocol, not sequenced music, was made for musicians/music hardware so it is inherently always been from its inception, tied to music production and is very different from sequenced game audio. It's not 'quietly used' like your description says haha
And trying to do sample library playback in a video game is just incredibly unfeasible and expensive without optimization, speaking from experience and most notoriously associated with orchestral libraries!
I appreciate you flashing hollywood orchestra there but I suggest you check the "system requirements" on that page.. and that's just for PLAYING the music in a DAW, not considering an entire game running beside it!
In general- The MIDI Protocol, sample libraries and such were made and optimized for a music producer/musician's use. Game audio, both in 1990, and today, are optimized well- for a video game!
Has anyone ever thought of attempting to convert the sequence data in mario odyssey and convert it to a module file or something
@@ssg-eggunner im unsure, but the sequence data from what i heard wouldn't be of like the whole song, it'd just have basically like the chords/the necessary stuff needed.
8:13 the FF7 soundtrack is entirely sequenced on PS1 too, actually.
One example of sequenced music that really is pushed to crazy extents that I really like is a lot of the level music in Wario Land 4. Each transformation (or certain statuses) Wario can go through can change up the music by slowing it down or speeding it up, raising or lowering the pitch, and/or adding varying speeds of vibrato to some of the instruments depending on what is happening to him.
The impressive use of sequenced music extends beyond this too. All of the sounds and voice clips are also used in many untraditional ways. Some of Wario's voice clips sometimes pitch bend for things like laughing (which only consists one sample of him going "Ha" repeated with the pitch sliding down for each), some of them are split up to make various different phrases (Like "Here I go!", "Here we go", "Wow, Go!", etc.), and some just do something weird like giving them stutter effects (like the infamous "H-H-H-Hurry Up!"). The sound effects in general often seem somewhat random at times as they use sequences borrowing from several samples stored in the game, as well as strange uses of the original Game Boy sound generator built into the GBA's sound system.
Of course, one last thing I'd like to mention is that there are a few tracks in the game that actually have lyrics, a rarity for cartridge-based games (the only other I can remember is Klonoa Heroes's main theme, Sign of Hero). The title theme uses various stock vocals from CD libraries, while the theme for Palm Tree Paradise (a.k.a. Medamayaki) and the English and Japanese versions of the ending theme have completely original lyrics made for them. For Medamayaki, almost every syllable was chopped up and compressed to fit into the game, so they could play in a sequence of the music file. However, for the latter two, they still play in sequences, but the lyrics are divided into short phrases instead. There's also an English-only reprise of the ending theme that seems to only play in the bad ending when you don't any treasures after beating all the bosses, but it's also divided into phrases.
Either way, sequenced music is great for the adaptivity of it as well as the simple fun of seeing what you can do with it. I've been fascinated with both that and streamed music just to see what was used to create each and it's nice to know what they are.
Super Mario Galaxy is proof that it’s very much possible to sync MIDI to audio files. Everytime you use a launch star a harp arpeggio plays, and it’s always harmonizing with the current chord that’s playing. Same goes for those blue-colored platforms that you have to turn yellow, the sound is always in-tune. And that was in 2007. Just wish more developers made that same effort.
Fantastic example!! Gamecube - Wii era had such great event / timing unique audio.
My personal favorite example would be Twilight Princess triggering different parts of the boss music depending on battle phases and advantageous moments. And the orchestration on this and Mario Galaxy sound amazing.
If you want another neat example of sequenced music being used for gameplay: I worked on a few games for the Nintendo DS (which still tended to use sequenced music in order to fit things onto small cartridge size limitations) and for one such project we were doing a music game, where the gameplay needed to sync perfectly to the music as every level was a song that the players were interacting with and playing along to. We had realized the music format the game used supported audio FX that the game engine itself didn't support when playing back the audio... but the game could still detect that those FX were being triggered even if they didn't actually do anything when the audio was played. So, we set up the game's code to detect those audio FX triggers to trigger stuff in the game's code instead, and thus could use the music editing tools as a level editor and literally script gameplay directly into the music files itself-- the music gameplay of the level always stayed in sync with the music because it was effectively scripted as inaudible parts of the music.
As someone who makes music..
MIDI hasn't gone anywhere.
Your choice of SPC files as a comparison with MIDI is an unfortunate one. They do not operate on the same principal at all. MIDI files contain note sequence data. SPC files are actually RAM dumps of code and data that run on the Sony SPC700 chip. They're so different that accurate SPC-to-MIDI conversion is practically impossible (although I suppose with some type of AI technique you might be able to do it).
It's totally possible. I used to convert SPC files to Impulse Tracker format so I could rip the instruments from them with the loop points intact. This was back around 2005.
alright Sheldon Cooper
@@Guacamole1000 bazinga
@@PlasticCogLiquid that's awesome. What tool did you use?
@@suitandtieguy Can't say what tool he used, but if I ever needed notation or just midi conversions of spc-files, I used spctool
I remember Mario Galaxy being a big deal for having a "fully orchestrated" soundtrack, only to find later that only a handful of songs are digitally streamed, let alone recorded by a real orchestra. It's crazy that you can just extract dozens of instrument samples from that game with little effort.
They did that to cheap production costs.
I'm not much of a gamer, but as a musician, I'd love to see MIDI implemented back into games. I remember playing Doom 2 once when I accidentally left one of my keyboards plugged in and turned on. The game automatically sent the music to the instrument, and the whole experience was transformed.
Hey I love your video, but I just want you to know that the original PSX release of Final Fantasy 7 does in fact use a soundfont (not the technical term, but you understand). The entirety of the Final Fantasy 7 OST on PlayStation is under 1MB!
That said, fantastic video! I hope that game developers start using MIDI again!
Thanks for the comment! I may have to look more into the FF7 situation, perhaps I'll do some sort of follow-up video. The nature of One Winged Angel's vocals as apparently just being one extremely long midi note are quite interesting. Also, I love your series on OOT with the Chrono Cross soundfont! Really great stuff.
@@ComputerLabHighjinks Thanks for the compliment, and the thoughtful response! Yes, the vocals in One Winged Angel are just a few samples triggered by midi notes! I realize people are correcting you a lot in the comments, sorry to be a contributor to that bombardment. I'd love to see a follow-up video, Keep doing what you're doing!
Lucasarts iMuse system is a great argument for MIDI. I was spacing off at work the other day and thinking about how well it worked in TIE Fighter to change music smoothly and dynamically with what was going on in the game (even down to a friendly ship being destroyed).
Also, I love the FFVII PC score for Yamaha XG. I have to admit that I amassed a small arsenal of Yamaha MIDI devices just to hear the music across different levels of hardware. MIDI was just fun on PCs! Before the early days of 3D cards, it was the one thing that could make a game seem very different from computer to computer.
monkey island 2's woodtick theme system is still one of my favourite pieces of dynamic music. these people really layed the ground for the future. BUT systems like these are perfectly possible on mainstream audio solutions without MIDI or MIDI-like interfaces. the limiting factor is not the tech or reluctancy to use other tools, but money, communications and the low priority soundtrack often gets inside productions.
iMUSE was used to great effect in all SCUMM LucasArts adventure games where it created an incredible cinematic feeling. Especially Indiana Jones and the Fate of Atlantis with its very atmospheric soundtrack that seamlessly blends musical cues one to another. People rank Monkey Island as the greatest Lucas adventure game, I strongly disagree, I find Fate of Atlantis has much better writing and has a much more epic feel.
I prefer tracker music, such as what Unreal and Deus Ex used. The composer gets to choose their own instruments, and the listener will always hear *those* instruments regardless of what sound device they have.
That's how all sequenced music works. It's not just trackers.
@@codahighland Technically, MIDI is sequenced...
@@AmaroqStarwind Yes, exactly, that's my point.
@@AmaroqStarwind To be more specific, I suppose I should have said that you CAN do that with any sequenced music, and I should have clarified that the target hardware defines what instruments the composer can choose (e.g. the NES's channels). It was really only a fairly narrow part of history where people were playing back MIDI music on arbitrary hardware without composer-supplied samples.
Wwise (one of the most popular middleware for music composers working with game engines) makes it easy to work with midi. Hoping that indie devs take advantage of this too
One great example of the advantages and disadvantages of MIDI implementation is Guitar Hero vs Rock Band animations. Guitar Hero 3 and onwards tended to use a mix of motion capture and manual animation on its character models when they're performing songs. The pinnacle of which is Bohemian Rhapsody in GH Warriors of Rock - which beautifully recreates Queen's original music video with handcrafted staging and lighting angles. And I remember reading that one song took over a month of dev time - Guitar Hero songs were incredibly laborious to produce. And that results in a lot of shortcuts elsewhere - the camera hardly ever zooms in on what the guitar player is actually playing (because it's not that accurate), and you'll see certain canned animations like Slash dancing in a circle multiple times. And if you export a song to another game or get DLC, it's likely that song will have gimped animations due to filesize limits.
Rock Band by contrast controls almost all animations with a midi track - everything from where a guitarist is fretting to venue lighting and camera work. And the end result can certainly look janky or robotic at times, but the quality is way more consistent and Harmonix had a cheaper and easier time publishing DLC tracks - to the point where over 4000 songs were officially released in Rock Band's lifespan compared to several hundred Guitar Hero songs produced by Neversoft.
Anyone who's delved into editing rhythm game tracks can tell you how much easier a MIDI standard is to work with than some awful custom file format solution. Beat Saber of all games uses freaking JSON text files to describe note positions, which is hell if you want to map a song with any BPM changes (which is pretty much any song that's played by real musicians not using a click track) or if your song goes longer than 10 minutes (since the note positions use floating point values, which become less accurate the further you get from the origin). It's so painful seeing devs reinvent the wheel when the MIDI standard has existed since 1983 and is STILL on v1.0 because it's so well designed.
was always impressed with how all the diddy kong racing zones had their theme music crossfade as you went around and was basically a differently instrumented version of the same song with different stylistic bits added in. was always shocked at how well it was done and would end up getting sidetracked by messing with 'playing it' like an instrument lol
Quick Time still supports MIDI
Rick Astley will be on repeat in my head tonight
If I'm not wrong, the GameCube, Wii, and Wii U all still used midi-type stuff for some games, so at least for Nintendo, it depends on the game but they're still doing it (idk about switch, don't own one myself)
Yeah, all first party Nintendo games (Intelligent Systems excluded) used MIDI for the in-game music on the GameCube. Some Wii games as well, and of course GBA, DS and 3DS (the system stuff, Mii Plaza, Mii games etc.)
Man, I really have a hard time describing how much I love this channel! Your videos hit that perfect nostalgic spot, while still keeping everything relevant. This is my new favorite "comfort" viewing channel and I love it. Greetings from Sweden!
i’m a multi genre music producer currently learning game composing who recently got in contact with a game dev to do work and i mostly use just midi for almost all my tracks. i love the creative simplicity of it plus i work with orchestra a lot and i can not afford to hire an orchestra to play my music for me. then u have my beats which is also always midi. my electronic music. always midi. it’s only my guitar i use for some rock that’s not midi
this channel is a hidden gem of videos, im gonna binge them all.
A modern game that uses midis is My singing Monsters
Itll have a long midi track just for a monster's idle and playing anim, If theres no note being pressed, it'll do the idle, if theres a note being pressed, itll do the playing anim
The problem with using MIDI with modern virtual instruments in video games is that they take up a lot of processing power and RAM. I produce music using a gaming laptop and there is a point in every project where I will have to start freezing tracks to audio in order for the computer to handle it. As video games are generally also quite heavy developers won't want to use the majority of available cpu on background music. Then there is also the issue of disk space. While midi itself does not take up a lot of disk space, some virtual instruments do. For example Hollywood Strings mentioned in this video takes up around 300GB of disk space alone(and you'd probably want more instruments in the game than just strings). So while these amazing virtual instruments exist, in realtity a video game would have to use significantly less amazing ones, and the quality of music would suffer as a result
I've been a composer, sound designer, and tech sound designer for over 10 years. Most stuff here is absolutely spot on, but meta sounds in UE just isn't ready for most audio applications yet. Wwise, for all of its draw backs, is still the gold standard. It even allows you to set up MIDI and make the music dynamic, and combine MIDI with normal audio playback. Check out the GDC talks for Peggle 2 from Becky and Guy.
the things you claimed being only possible with sequenced music are still done with streamed music today. mario galaxy had a different mix for underwater (beach bowl galaxy) and botw whole soundtrack is all streamed and insanely responsive in such an immersive way, people don’t even realize. hi fi rush has the combat system intertwined with the streamed music to a super high degree. there are cases for sequenced music in games - but these were not them. having a game handle on the fly key changes, a rubato tempo that subtly shifts with the player with no loss of sample rate are both things i’d be super interesting to see. i’m fairly certain mario galaxy’s monkey ball-esque levels actually use sequenced music just so they could get that super dynamic tempo that responds to player speed.
the other thing that would be fun to see on pc specifically is having you game composed in standard midi and allow for users to either change the soundfont being used, or the ability to use external midi devices like the roland MT and SC-55
The case for midi files is Bug Fables' ost. That's the entire case. Thats all it needs.
I agree. Anothee good example of MIDI tech in games was iMUSE, which built up songs dynamically out of smaller elements. There is a new MIDI 2.0 standard that allows for things like vibrato, but as typical, it is only used by music producers, not games. The sound libraries required to make MIDI sound good are stipidly expensive. For MIDI to be used in games nowadays, there would need to be modern, standardized libraries that come free with platforms like consoles or Windows. I think more simulation of notes could be helpful in bringing cost and memory usage down.
Agreed, I wish more games used it. WarioWare is a franchise that always uses sequenced music due to the nature of the microgames and how they speed up
You have implied that "MIDI once sounded bad and now sounds good due to high quality VSTs". Well, that's pretty much a half-truth and either deliberately omitting a lot of important information or not being aware about it. MIDI isn't and never was a protocol solely used within software environments on general-purpose computers and game consoles. It is completely device-agnostic and initiallly it was developed to connect together sequencers/midi controlling keyboards with physical hardware - a lot of professional music synths from the 80s and the 90s had MIDI in/out support and a lot of music from that time had been sequenced using MIDI. Atari ST computers used in professional studios, controlling a wall of rack-mount pro-grade synths weren't uncommon. A lot of mainstream, 80s and 90s music used this technology already. For example Michael Cretu, the composer of songs performed by Sandra programmed a lot of his compositions using an Atari ST computer.
So. You can kinda just do all of these with pre-recorded audio perfectly well. The main advantage of midi remains file size... unless you're using those super realistic VSTs you were talking about where the digital instrument is 1TB of samples.
For one, you can totally just provide the sheet music in a midi like format alongside the audio track to get all those "game responds to the beat" party tricks you were showing off, without having to actually synthesize the music on the fly. Half the time you don't even need that, a "this is the tempo" annotation would be plenty for a lot of effects.
The banjo kazooie thing where the audio swaps out instruments underwater is also very simple to do and even common in modern games - you just record two audio files and switch which one you're playing. A great example of that is the way Nier Automata seamlessly swaps to a chiptune rendition of the soundtrack whenever you start the hacking minigame. It's even simpler if all you want is the "low pass filter" sound - now we can just have the game run the audio through an actual low pass filter.
The hand pushing harmonic piano keys is a little tricky, but still doable. Again you can provide a midi file as metadata alongside pre-recorded audio. Or you could annotate the track with key/chord information. A very similar problem that I've seen solved multiple times (Gris, Transistor, It Takes Two) has a much, much simpler solution - I'm talking about making a 'press X to hum' button. You want the character to hum and harmonize with the soundtrack playing when the player presses a button. Gris just has you hum a little chunk of sound that does the music theory to fit in with any part of the soundtrack in the area you can use it. It Takes Two tried to be super fancy and make a crazy modular synth thing using midi data to choose a note in the right key and chord based on what was playing... until they scrapped that and ultimately used the solution I'm about to talk about. Transistor (and It Takes Two in the version they shipped) made the hum button super easily. They want the character to hum and harmonize with the current soundtrack. So... they just recorded someone humming along to the whole track, and brought that in as a separate audio track. When you press the hum button, the humming layer of whatever song is playing fades in.
Sooo. With that in mind, how would I actually make the piano effect you showed off? I'd compose a piano layer into my soundtrack. And all it would be is a little annotation saying "Play this note at this part", no fancy music theory aware programming, just check the midi file for the current note and play that. So the piano itself would be kinda a super simplified pseudo midi, but the music track itself would probably still just be a normal audio file.
If you wanna make an Animal Well type game and stuff it all into 30 MB then I'd seriously consider midi. But otherwise, there's actually not that playing midi files on the fly offers that I can't easily do with regular audio, and less CPU power besides.
Yeah. Using the term "MIDI" nowadays is a bit hard because for example Final Fantasy VII Remake's OST was composed on virtual instruments with MIDI and not a live orchestra but sounds incredibly real, that's just the level virtual instruments are at nowadays so when people say "MIDI music" as a way to describe old sounding music it's kind of wrong.
MIDI is just data, it has no sound.
MIDI can trigger samples ranging from blip-blops to real life orchestral, drum or any other instrument samples that you would have hard time knowing it was recorded live or sequenced
If "MIDI" does become a big thing in games again I really hope it's in the form of more modern standards, such as MPE, or even OSC. I also find the idea of a game having it's own built in version of something like Vital as a modern version of sound fonts a bit amusing.
MIDI is just data.
It has no inherent sound.
People think "MIDI" is just "retro sound".
That is wrong.
MIDI can trigger samples that are as lowbit or real life realitic as you want.
Midi is also used in RPG Maker games, DOOM Wads, SRB2, and other fanmade games.
Banjo Kazooie (and tooie, which introduced tempo transitions) is probably the best argument for "MIDI" (or tracker music) in games, and probably the best example of it, its certainly my fav example.
Also its no mystery FF just recorded their MIDI to the disc, its dev time, it would've been quicker to do that than make a custom midi playback for their game if they didnt already have one from previous games in the same studio.
The extreme example of this would become COD being 200gb, just wasting space to save dev time by avoiding space optimisation.
I think a new tracker music format, like an *updated* Fast Tracker 2 or Impulse Tracker music file format, would be far more useful than MIDI in the gaming industry, provided certain features were implemented. There's no reason the tracker program itself couldn't accept MIDI as an input though.
A few of them actually did. I just checked in DOSBox and Fast Tracker 2 does.
One of the reasons I really like Touhou music is because the creator, ZUN, often releases each soundtrack separately as MIDI files. In the early Windows generation of Touhou games, between Touhou 6 and Touhou 9, there was literally just MIDI files sitting in the game's folder. I've always loved that ZUN did that, because it makes things easier if we want to analyze the soundtrack by opening the MIDI in a DAW. Not only that, but ZUN also released the soundtrack from the PC98 games (Touhou 1 to Touhou 5), which were originally composed with only classic FM synthesis due to the limitations of the PC9800 computers, but he released these songs in a Windows-era styled instrumentation AND in MIDI form. Touhou is great for MIDI music!
I'm not sure if it's using MIDI, but there is a rhythm and action focused game called Hi-Fi Rush that, like Mother 3's combat system, requires you to do combat to the beat of the song, though it's a bit more complex. You can do combos by combining different 1/4 notes, 1/2 notes, and 1/4 rests. It's also just one of the most fresh gaming experiences I've seen in recent years, with a good story, good comedy and good music. I highly recommend!
8:15 Music of Final Fantasy 7 was composed with Yamaha's XG MIDI standard in mind that was incompatible with game consoles. If the game music in the console sounds almost the same as what you would hear from an XG synth, that's exactly why. XG not only offers more instruments (melodic voices or sound effects) and more drum kits than General MIDI, but also offers audio effects (reverb, chrous, delay...) that are simply non-existent in General MIDI (revision one).
(I know this because I develop something that's compatible with multiple MIDI standards XD)
I love this video! I can't wait for your channel to grow :)
Inspiring video, great examples of how midi was superior in many ways, that unreal plugin looks very useful! Subscribed!
This video mixes up midi with chiptunes on the nes. Chiptunes are tied to a particular sound chip and store instructions for it directly. Nes chiptunes always sound like that. Midi is instructions for generic instruments that can sound different based on the soundfont in use and aren't subject to the limitations of something like a square wave generator.
This is really cool! I was just thinking about this very subject the other day. If there was one more game that deserves recognition for using MIDI music, it would have be Doom. It spawned a whole community where MIDIs are about as prevalent as ever there. Those composers have made some astounding pieces of music over the years that have raised the bar for how to get the most mileage out of the same universal 128 virtual instruments. Anyhow, great video man!
Cool video mate!
hi there, we still make music in midi format and then put it in games after being rendered to audio.. it's still midi triggering samples here on our end , i'm a game composer lolol so technically even new games are using "midi music" it's just not pulling sounds out of a chip in a cartridge, it's rendered to audio. put into fmod or wwise then turned into a build file which works with the code of your game, there is no way we are going back to the old way.. i mean you can if your a hobbyist i guess. It would be very pointless. I can still have the same sounds from those chips on my computer right now. In fact i do have them. Impact Soundworks sells Super Audio Cart and also Insidious, the C64 one. I even have a Gameboy sampled into sofware from another company.
Your use of the term MIDI was getting me upset, but then you clarified you're using it loosely. Breath of fresh air, lol... A few other technical issues aside, it's a good video and I agree with the overall message.
Around 2 weeks ago I started making chiptunes because of my interest in sequenced music. Now 2 days ago I discovered the MOD format, now Im making stuff with it.
I don't use midi files for the music in my game, but I do use midi modules(Roland SC-88). It's a vst, but I just love making music from the sounds of my childhood. It's also really cool getting a midi file from a game like final fantasy and having the Roland vst play it exactly how it sounds in game, but inside FL Studio. Great Video!
Good video! Though I wanted to note, 11:57 it's not that complicated. Going underwater in Banjo-Kazooie simply switches to a harp arrangement of the current level theme. Nothing fancy has been done to it, the harp sample just sounds that way. Also, I'm pretty sure the Xbox version still uses MIDI, and is even incorrectly emulated, most notably any time the theremin instrument appears (e.g. the beginning of Mad Monster Mansion) completely lacking the vibrato it had in the N64 version. The organ always playing the same note is just a bug.
I really loved this video! Im currently majoring in video game music and it honestly really surprises me how little midi implementation there is in audio middleware. wwise has it but there's a lot it cannot do (changing tempo being a huge one) that said ALOT of what you talked about with midi here can be and IS done with audio in games with wwise.
The underwater example is super doable!!! you would have both versions of the song playing at the same time. the underwater version at -96 (basically no sound) at the above at 0 (standard volume) and then set all the water to be a trigger to fade out the above water and fade in the bellow. you can even do things withlow pass filters, reverb etc. Ive even seen people do things like that scarecrow song with audio simply by essentially making a sampler out of triggers (though midi would and should be used there)
Theres ATON of rly cool interactive music things still happening and its really really cool! I think my favorite thing right now is how some games will change melody and stong structure on a dime depending on whats going on. I know vampyr does it with its main investigation theme.
I highly recommend looking up Lucasart's iMuse system used in Tie Fighter and Dark Forces from PC DOS. Dynamic music changes based on situation with seemless transitions. Great stuff! Love my Roland SC55.
Hey, don't want to hate but the last part shows kind of a misunderstanding in how Audio works in games nowadays.
Not only are those old techniques of altering the sounds and music basically 101 of Game audio knowledge, we create far more complex systems today (sometimes even based on midi, but mostly if we need constant timing throughout runtime. Midi works great for that and again IS used today).
The in engine tools mostly aren't sufficient for most audio professionals in game audio (not to say that they are great tools nonetheless). Even generative sounds are used for YEEEARS now. For example the suit voice from the first Crysis 1 had only the raw Voice Actor recordings, and the whole voice processing was done in runtime, with commercial available Middleware (Wwise).
As you said yourself: East west content alone is TB of data. Imagine what amounts of raw audio data we as Producers and Composers have on our SSDs and even HDDs. This undersells the effort music producers, composers and audio engineers put into the complete audio package. And it's not because "we just love our hardware". Those are mostly marketing. In the real world, almost everything is done in DAWs, Engines and Middlewares (Wwise, Fmod) directly, because it's faster and cheaper. But also know: Orchestras aren't as expensive as you might think and are present even in many smaller indie projects. But dueto more and more advancement in Machine Learning and CPUs just getting faster and our techniques getting more efficient, its not impossible to see in the future live MODULATED orchestras or other instruments. An example we use in production would be the Plugin: pianoteq. This is like 20mb of filesize and can impressively modulate Grand Pianos and more.
Also, because of the high graphical fidelity of games today, we still have hard limits on CPU times for music and sound processing (sometimes not more than 2% of the game's required processing). Most of this (in mostly 3D games) is used for Reverberation Processing (which is extremely hard to do real-time). In addition to that: Just think of the many games reaching the hundreds of GB they have today. If we add the TB of data (and yes, if you want to have the orchestra sounding remotely original, you need all the audio data), a game could not be shipped at all.
TLDR: Yes, we still use midi from Indie to AAA. Yes, we use far more advanced techniques today. Yes, we, for years now, use real-time synthesized audio content.
PS: This is just a small brain dump but should not be understood as hate or something.
using a NES game, that uses chip music, as an example of midi is really strange, would have worked better with a PC DOS game that actually used midi...
You forgot to mention Amiga which almost exclusively used MOD files which combine samples and playback information. In fact it survived for a long time between the platforms, and even some more recent games used MODs.
For a specific example, I'll nominate 2019's Ion Fury, which uses Fast Tracker 2 XM files.
Sunvox is a music tracker with a powerful synthesis/audio engine, and there is a library available to use its sound engine in other software. I'm not a programmer, but I THINK it can be used to make audio/music in a game that reacts dynamically to in-game events.
I'm excited to see what people do with the Harmonix beatmatching tech that's now in the latest versions of UE. It can basically take the best parts of sequenced MIDI (which drives the technology at the base), and recorded audio, and blend them together.
great overview, thanks for this! big fan of midi & all the possibilities it creates for interactive audio design.
i think you'd be into the game i made the sound & music for, Ape Out (2019). it has a proceedural jazz drumming music system that runs on MIDI behind the scenes. i generated a bunch of midi using a neural net i trained & then implemented a basic virtual drum sampler with velocity sensitivity & round robins to enable the drum performances to react to players' gameplay choices in near real time.
I can't really imagine MIDI coming back full force, seeing as that would mean building a LOT of samples into the game, which, depending the complexity of the instrument, can number in the thousands. I have a jazz brass library that's a whole 75 GB. I don't even know how one would go about converting the sample files into something a game engine could load, much less include things like legato and round robin.
Not to mention that making music in DAWS also involves tons of third-party effects plugins. I have no idea how you would implement commercially-sold VST or Audio Unit technology into a game engine.
For example, would I be able to insert a recreation of the Roland Space Echo by IK Multimedia into my game?
The real question is "How many of those samples do you REALLY need?", and I suspect the answer is "not very many". The Gravis Ultrasound from the early 1990s had a sample library of about 8MB that it used for MIDI playback (though the card's onboard RAM was only 256-1024KB), and that sounded pretty decent with usage of maybe half a dozen samples for some instruments. Other games used music formats (tracker modules) that bundled note data and sample data into the one file (a recent example being 2019's Ion Fury, which uses Fast Tracker 2 .XM files), giving individual music files a size ranging from the high tens of kilobytes up to a meg or so, depending on the samples used.
I think part of the problem is that most composers are stuck in a mindset akin to that of film graphics vs game graphics. A film can take hours or even days to render just a single frame, but a game has a matter of milliseconds to do a frame. When you can just leave a bunch of computers rendering for weeks on end, you can use the most expensive techniques, even if they don't give results that look that much better than the much cheaper ones the game has to use due to its time constraints. I think composers are in a similar situation. Most are working with the film mindset where efficiency is irrelevant, rather than a game one where it's paramount.
I think it would be a good exercise for composers to try and compose a track of moderate complexity (8-16 simultaneous notes) in a tracker program, and working within a sample budget of 512KB. Plenty of space for flexibility, but still constrained enough to learn something. For actual programs that don't need an emulator, there's Milky Tracker, which resembles Fast Tracker 2, and Schism Tracker, which resembles Scream Tracker 3. You can even use FM instruments in S3M files if you use the latter.
Awesome video. Subscribed
I feel like having the computer perform music in real time, especially if it uses a lot of effects, large sample libraries or complex synthesizers, it would take way too much computing power to do that.
I dunno. I think you could go pretty far if you budgeted 20-30% of your CPU usage for audio.
There's a game called FRACT OSC that uses nothing but procedural audio. You can even play around with the synthesiser outside of the game's puzzles.
this is exactly what i needed to watch now
I am aware that Final Fantasy 7 was used as a test-bed for next generation console technology. It's why differnt environments used completely different character models, they combined 3d models on 2d backgrounds and why there's quite so many crazy glitches possible.
So my best guess would be that the plan was always to use orchestral recordings (particularly for that final number with the choral parts), but that the orchestral recordings took too long, or too much money. So the initial release used audio recordings of the sequenced tracks because it would be a completely different use of technology to playback the sequence.
Also, later on it did use orchestral recordings for everything.
I'm so glad that it isn't just me who is like this. I recently tried to find a midi collection website like the ones of old to get that feeling again but they are all behind pay walls now. Ridiculous for what was everywhere only 10 years ago
Midi and programable music as a whole is something very intriguing
And it’s something im definitely planning on using in my next game, like I could change keys based on biome, have it play slower at night
And use different instruments to mimic sounds of the world, like string for wind, or tambourines for magic
Look into fmod :) might be something you can use for that
Might want to look into tracker modules for your music formats (there are quite a few of them). Many of them can have sub-songs which the game can jump between. Unreal does it, as does Deus Ex (both use the same engine, so it's not really surprising). Unreal tends to have a quiet sub-song, an action one, and one or two transition bits. Deus Ex seems to typically have four or five in most of its music files. Apogee's Stargunner (1996) has 4 or 5 sub-songs in its level music tracks. One for the main level, two for different stages of the boss, and a "level complete" one seems to be pretty typical.
If the argument is that Midis should make a comeback, you haven't been listening.
Head toward the homebrew or 'Aftermarket' of retro games. Check for Genesis games like Demons of Asteborg or ZPF, or Neo Geo titles like Project Neon. And if you think they should come to modern consoles or Steam, many usually find their way to those digital storefronts.
Listen to the new generation of Midi.
some games used them to help the game engine control animations and hits to keep them synced to the dynamic music regardless of the player's timing on hits ( wind waker's dynamic music layering comes to mind) - this was on top of the game using sequenced music
The thing is,midi…was never in consoles.midi use external proprietary sound database (Yamaha ,Roland …) .games in the other hand was loading those database from the game and ram.
Making console music closer to tracker music… this is why console and pc music sound so different. It you want music close to the snes,the sega cd or n64 on pc you need a amiga pc,or a Gus sound card.
And for ff7 the game actually uses sequenced music pretty much all the time since the game use the cd player to load backgrounds and video files. The only red book audio the game use is when it read a video and one winged angel…
Some game engines can easily play modules (xm, mod, mptm, etc) and MIDI.
Rhythm based mechanics are not restricted to "MIDI" technology, but it is, for sure, more efficient to run without taking so much storage with MIDI/modules.
Cave Story is a very good example of a PC game that uses modules for the soundtrack.
I believe another good example of the sequencing being used for something other than exclusively music is fortnite's festival gamemode. It uses midi sequencing to control the animations for the instruments and I assume all the effects happening on stage.
Just to add to this, I recently learned that the (one of the?) Genesis/SMS music system had a mechanism for that dynamic game engine interaction called mailboxes. It apparently wasn’t used that often, but it did exist! That said of course, you could still always make your own driver to implement whatever dynamic behavior you want.
Heya, entertaining video, however I'd like to mention some things (if I understood correctly)
All the amazing VST that uses midi messages as the driver still need to be rendered on audio files or they can't be used in the game. Yes VST's are currently and for at least few decades now being used to compose video game/film and other mainstream music but we still need the audio file rendered or we will not have a sound, as you mentioned at the introduction of the MIDI protocol.
Also interactive music, again through use of Audio Middleware or Game engine's own audio tools are often used in our day without the need for midi, because we have the tools and the storage space to add them without using MIDI and soundfonts.
And I'd say the current gold standard is still WWISE, due to its ease of use over Metasounds (ease of use by musicians and audio specific people and due to the use cases being obvious for music and sfx, many of the needed features are in your reach in 10 seconds rather than 15 minutes).
An example would be not being able to see a waveform to set the fades or being able to edit audio files in a few second are big disadvantages of the current Metasound compared to Wwise.
nice and informative video! the no mans sky player-created music would be an example for midi-like implementation ingame that makes sense. apart from systems like that you are not looking for midi/vst in-engine but for more deep dynamic soundtracks. the banjo kazooie situation, scarecrow-song and the other examples are easily solved/ported correctly by logic implementation, typicaly with one of the two major gameaudio-handling middlewares: fmod and wwise. music logic does not depend on in-engine sound generation at all. even if you want to logicallly reharmonize a whole song, in most cases, its easier and cheaper to put it together from pre-rendered audio than to make an in-engine note player generate something new, all while soundgeneration is vastly more costly on memory cpu than people think (i have a essentially a top-class gaming pc in those terrms just to produce music). there are modern games that do what you are hoping to see with recorded audio. ori and the blind forest has seemless and highly defined player-controlled transitions and blades of the shogun plays little flourishes depending on the character and character-action in-sync with the music for almost any action. the reason you dont hear things like that very often is that sound is more often than not handled externally and game-audio-producers dont get their hands on game logic very early that way. what IS happening more because of faster machines is sophisticated realtime audio effects though.
Playing back MIDI files in 21st century with quality that are acceptable to be enough to be immersive, consoles would really need dedicated processing units, dedicated RAM and dedicated storage with those instruments to handle playback - it would be not ideal for music to take away majority of processing power to play MIDI files in pristine quality.
Bringing back MIDI is such a great idea, though - I'm just wondering if there would be efficient and viable option.
I remember Xenogears in PS1 had some kind of Soundfont system - that if music gets interrupted by going into menu or press on Pause, it would miss few NOTE ON event that was triggered before it, resulting in very short moment of weirdly sounding music.
Back in the 1990s, we did have dedicated hardware. The gold standard for MIDI back then was the Roland Sound Canvas, which was an expensive dedicated MIDI device that was basically only owned by musicians. If you were lucky, you may have had a Sound Blaster AWE32 or Gravis Ultrasound, which could do sample mixing in hardware. It didn't last, though. CPU power shot up so dramatically that mixing samples in real-time on the CPU became cheap enough to be ignored as a concern. If you've ever played Deus Ex (2000) or Ion Fury (2019), you've heard music being mixed in real-time on the CPU (though those two games don't use MIDI, but tracker modules instead).
Amusingly, the various games that came with CD Audio tracks often had those tracks be the result of the game's composer just playing MIDI files through a Sound Canvas, and I don't think anyone could tell it wasn't real. We passed the "good enough" point long ago, and the libraries that pass the "orchestral Turing test" are just bragging rights.
If a game developer wants to real-time mix all the game's music, that's easily handled by anything that'll run Windows XP, so long as the composer doesn't insist their music use an overkill library. Some MIDI files and a few meg of good soundfonts fed into a MIDI playback engine will do the job with less than 1% CPU usage on a 5-year-old machine (playing the Descent soundtrack with XMPlay's MIDI plugin right now and the only time it shows anything other than 0% is just after loading a new file, and I think that's just from restarting the MIDI plugin and reloading the soundfont).
my fav thing is that you can play midi music in space station 14 for other players. I have an extensive midi library specifically for that game.
Thanks Nathan 🍻
You answered your own question when you said the sample sizes are huge in the new VSTs.
How could midi notes play back any of the information without having all the samples included in the game install?
If you want to hear Hollywood Strings in your game, you have to literally put it in your game.
If your game has no recorded material and can utilize the synthesizers available in your engine or middleware, then sure, you could go all midi.
But if you have any sampling of real instruments, you cant include those without larger file size.
This might be a long shot, but I would absolutely LOVE a video about some of the games available for the Mac in the late 90's. I specifically think of the Pangea games, like Nanosaur or Bugdom. That era of gaming seems to be completely forgotten nowadays.
Thank you for the comment! Glad to hear you're enjoying the channel. Oddly enough, I grew up using an iMac G3 and only found out about the included Bugdom 2 game quite recently. It was quite a surprise to find a full 3D platformer hidden in a random folder! I'll add these to my list for sure.
Great video!
I actually owe a big part of my career as a music composer to midi - and that is my main occupation!
I scored plenty of films, series and theater plays all with midi and VST's. It is a great tool that empowers composers that otherwise would not have a chance in their life to hear their music played by an orchestra!
You're also right about the conservatism. I've heard some musicians, very snobishly, call the use of midi and samples as being dishonest.
How do you make a video about MIDI music in games and then only talk about games that don't use MIDI??? DOS and early Windows PC games are where MIDI was the big thing… compositions for the Roland MT-32, Roland Sound Canvas series, built-in MIDI synthesizers on sound cards, etc…
As a game developer I will take this into consideration.
I tend to extract the midi from old java phone games and change the instrument to a modern vst.
It giving a new fresh take on the tracks, bc most of the composition is still good even after a decade.
I think having midi inside the game is almost feel like giving music source code for free, pretty sure the game company didn't want that. encrypt might help but people eventually break those protection.
While recorded music even when it stolen or decrypted, they don't have the full source code of the composition.
Old School Runescape has one of the greatest soundtracks in gaming, unironically, and it's all MIDI.
Listen to Amascut's Promise (the final boss theme of the Tombs of Amascut raid) and tell me that game's OST is being held back by its format.