Reminder: TH-cam reserves the right to delete channels at will, without warnings or strikes. Many channels have been deleted in the past for using up too much space and having little to no views.
@@GeneralLDS Jup, but I think no one wants to spend time on watching lots of videos with Black and white nonsense. And the anti-bot to let views not be buyable/ fakeable are there too.
@@icitryfor the qr-code you can also do the folowing: 1 pixle is a RGB tupple (r,g,b) If you have (255,255,255) you get white If you have (0,0,0) you get Black Instead of saving in black and white you can use the RGB min/max for 3x efficiancy You might think that you could lose in compresion but the difference between min/max is so big that you can restore the original values from something like (234,247,219) And (10,9,3) Also storing metadata in Audio or Video description can be great and comments can be used to log smal changes TH-cam gives you up to 250.000 coments i Think and each has a maximum of 500 replies with no apperant limit for text size.
Oh great suggestions, I think those types of RGB codes are called JAB codes if I'm not mistaken, I've discovered them after the project was done 😅 But the comments thing is really smart, didn't even cross my mind - big props!
That's why when making projects, you have to think of everything before starting, and more importantly identify the most critical sections and start with them because if they don't work there is no point in continuing
There is actually a method using discord attachments (split in 25mb chunks) which is much less effort since data does not need to be encoded to a image.
Exactly. Where I work we rush solutions into production because there is no budget for evaluation possible solutions... first solution is good enough. Then after month, it is a unmaintainable mess because no one thought about the complex use cases..
I clicked on the video thinking you're some relatively well known coding youtuber... I finished watching the video and I loved it all, then I got out of full-screen, and man... What on earth?? How do you have less than a hundred subs?? My guy the production of such entertaining content really deserves sooooo much more attention. I dropped a like and a sub, hoping you grow larger in the future. Much love.
That makes me so incredibly happy to hear! I can't even begin to thank you for such kind words, they mean so very much to me ❤️ I'm glad I could create something you could enjoy to this level 😁
@icitry Awh. I'm glad my review meant something. I would recommend opening up a social platform to form a smaller community. Like discord. I believe it would help forming a community around the channel where people may discuss parts of your code more in depth. It may also help growing. Once again, great content, lots of creativity in it. Love it.
@@icitry sadly youtube will not care much because if anyone can keep uploading and doing live steams for 3 mouths for scams or loop of giveaway Why would they try to ban anyone doing that because they think Ad blocker is a proplem ------------------------------------------------------------------------------------------------------------------------------ what about FILES i has in google drive ( 50% of my files are gone ) 15GB - 50% - cost to 7.5GB are the real space you can keep meaning google can spy as they wanted and there no Locks on it even end to end encryption even if you ( PAID their max tier 2TB they still will not give you encryption in anyway ) Only if your in School and business or college to get protected as a USER and not for cookies
Haha wow it's truly amazing to hear that 😄 And if you have any suggestions / improvement ideas / nit-picks, I'm very open to them, given how I'm still figuring stuff out 😅
I feel like this is peak reinventing the wheel. It's not about the core idea though, the idea is great. It's about the implementation. Here's a few suggestions (feel free to completely ignore them lol): - pls avoid webscraping as much as possible if there's a free, working API for the same thing you want to do (AND THERE IS). Google let's you manage(upload, list, edit download) your entire TH-cam channel with their APIs. They even made a wrapper libs for most languages so they can be used even more easily. - follow up to the previous suggestion: the whole problem you had with account sessions is completely solvable with OAuth. Just generate a key and out it in an env var or config file. - don't reinvent the gnu coreutils. Instead, create something that can be mounted as a file system and used with your preferred file management method (commands or a file explorer GUI or anything else). If I were to do this I would -steal- get inspired with *one of* simple S3 server implementations(or WebDAV or FTP or something else. I think S3 is the simplest though) in your preferred language and JUST focus on the CRUD of files (of course using TH-cam APIs). So you'd just replace their createFile function body with your logic and you're done. Then when you have a working S3 server you can either use it in your other apps or mount it as a network drive and just interact with your files in your file explorer. You spent most of this video doing things completely unrelated to the title (storing files on TH-cam). Instead you should've just used all those existing tools and protocols that people before you made and just do one thing that actually matters right and that's converting a file to a video that TH-cam can't destroy with their compression. Dont make your own (worse) cd and ls commands unless you know how to improve them.
Oh wow, thanks so much for taking the time to write this out! 😄 But the thing is I kinda wanted to reinvent the wheel (also why I wanted the focus of the video to be the whole cloud system, not just the file conversion) - sure if I were to make a production grade solution I'd go with tried and tested solutions, buut where's the fun in that if I can fool around yk? 😅 Also, some counter points - the TH-cam API may be free by default, but you still have maximum daily quotas, I wanted to see if I could get a fully free implementation going; and the generating of an OAuth key is a great idea, but not what I was going for, I wanted to have the user do as little as possible, and (totally imaginary challenge) if someone wanted to use this as-is, I wouldn't want to be sharing secrets or have them generate others (maybe I should've been more clear with that intent). But the implementation definitely has a lot of holes, and your suggestions are more than welcome and closer to what should be done in a production app, so thank you very much for the feedback! 😊
@@flintfrommother3gaming I didn't say webscraping was bad morally. Its slower and a lot less reliable than a single API call so when if you have an alternative (and you do, TH-cam has a free public API), don't use web scraping
Wow thank you algorithm for showing me this video, someone make it so I can come back to this video every time and remember this youtuber exists, this is so good
That's what I made yesterday but it doesn't work with youtube compression. (did't bother to fix it) I thought I was a mastermind or something and I realised at 3am of night that I re-invented the wheel (after looking at a repo at github). I went back to sleep with tear in my eye XD. I am not the only one. You got a sub!
Great video. You can definitely experiment with data density of the QR code. You can try different block sizes as VP9 divides frames anywhere from 64x64 down to 4x4. Also increase quantization from 1 bit to maybe 4-6. Perhaps you can even get away with using the full 8 bits if you include some ECC. Though I wouldn't recommend using all 3 channels of a pixel as chroma is more heavily compressed than luma. Though increasing bits per block may cause the data to be more susceptible to temporal artifacts. Duplicating frames can fix it but it's gonna be a balancing act. To further increase density, you can utilize the audio track with the same method except in 1D.
Oh wow those are some incredibly neat suggestions, I have to actually go and search some bits up to make full sense of them 😅 - you seem way more knowledgeable in this area than me so big props and thank you a lot for taking the time to give such thorough solutions! 😊
@@icitry polychromatic QR codes are a thing. As Kay noted : the density will be lower, so maybe you'll need to apply some colorspace transforms such that you allocate like 2x density on luma and 1x on each chroma. YAY ENTROPY!!!!
I was expecting this to be the core of the video. After watching it I see why it wasn't. Just setting up the logistics of a virtual filesystem on youtube was obviously a big enough project.
Using the audio track does not increase density. A silent video is not a video with imagery accompanied by a full-length silent audio track. The audio track is simply absent, and therefore takes up zero space.
I've seen a few implementations/videos on this topic, since we worked on this a few years ago on 4chan. Most of them reimplement what we had already developed, but your route of generating and uploading the frames VIA the API is a new take (at least to me) on the solution. This could alleviate the storage issues we ran into, where generated videos are around ~2x+ larger than the input file(s). If I ever get around to testing it, and if it works well enough, would you be fine with me using your idea in Schillsaver?
I thought of doing the exact same thing. I thought of the first solution and dropped it for the exact same reason. Then thought of dividing the screen into small squares and make them black or white encoding bits. It’s technically similar to the QR code thing. Kudos for actually doing it
I want to make something similar but instead it encrypts the data and is able to split the files and combine storage of the various cloud providers. Also, a possible improvement would be to use bigger QR codes as they can be much bigger, also if you don't mind going out of spec they can be infinitely big if you write a decoder for them
Well go for it! Data encryption is something you'd always want and the idea of a distributed system is really cool! 😄 But for the QR codes you can't have them be infinitely big, as you'd end up with the same problems regarding compression when they become too cluttered. Someone else pointed out that I could've used multiple QR codes in a single frame, all within specs though. But great ideas and thanks for the feedback!
For storage backed directly by file services (Google Drive, Microsoft OneDrive, Dropbox, etc), a project like this happens to already exist. The core project is Least Authority File System (LAFS), which fundamentally handles encryption and manages cross-host storage. I think I recall even a more plug-and-play project that is meant to work with a dozen or so of the largest file storage providers out of the box. I don't remember what that project was called, but I know the core of it is LAFS.
I've done something similar. Encrypt the files -> split them into 24mb chunks -> upload everything to discord and keep a local database of the file names and paths.
many years ago, VHS tapes were used for data storage/backup. but they didn't use crazy compression schemes so it was "easier" (still a crazy challenge but not like this one)
I've seen another person do something similar, but their solution ot the compression problem was just to make the pixels of data bigger until they weren't affected by compression. I do think your solution was more elegant as it uses an already developed system, meaning you can reuse already optimized code and such.
Back in the early days of the Internet, the 1990s, I used free email accounts for cloud storage, as attachments. No coding. Of course, limited to MBs per account, but my files were small.
Oh that's pretty smart, and is still somewhat viable, just that the file sizes have grown so much in time for even the most basic things, you'd have to split them into chunks and rebuild them on retrieval for each query 😅
Oh that's a great idea. From what I know it would be something like 2-300 under perfect conditions for a 1080p frame, although I'm not sure how much compression would affect this. But even using something like 50 is an incredible improvement - nice catch!
Couldn't you exploit Google Drive simply by creating a bunch of accounts? When one account runs out of space, just make a new one, but use the old account's email as the recovery email for the new one. I have 3 Google accounts chained together like that.
Huh that's a fun idea, didn't even cross my mind - but what if you had to store something larger than 15 gigs? Guess you would have to break it into parts and add some linking mechanism to piece them together on request. But it's so cool how many different approaches we can have to the same problem, exactly what I love about programming 😄
@@gior987 When you create a Google account, it asks you for an existing email address to use for recovery purposes, and apparently that can be another GMail address, so it's likely possible to create an unlimited amount of GMail accounts that way.
Just found the video. Thought it was of the same guy who made the discord storage solution,then I realized u r not him. But loved.ur content. Defo subscribing and liking
Why did you make it 1080p? Just use the minimum possible resolution (240p, or lower) and you can use a square video, or any resolution/aspect fine. The only thing that gets scaled by youtube is the resolution, not the aspect ratio
I actually select the worst available format when selecting the stream URL, otherwise I couldn't reach the target FPS set by the videos - trust me I tried 😅 (also I wanted to emulate the video's style so that's why I'm not going for fixed values).
Oh my you got my headgears spinning again... - using Treillis encoding/decoding of a bytestream, or even some kind of OFDM by splitting the screen in 6*4 zones (for luma) and 3*2 zones for chroma(s) and multiplexing/interleaving each field - then of course some Reed-Solomon for good measure - Send 3 identical frames and sample the 3rd one so the decompressor has time to settle the fine details Of course the Shannon limit (entropy) cannot be exceeded, but this raises another question and makes it interesting to move to the spectral encoding space, which is a totally different beast than QR codes, which have a very high high-frequency (hence entropy, hence loss) component. Try playing with 2D Fourier transforms (or even 3D !) so you can encode more bits per pixel !
Oh I can definitely tell they're spinning 😄 But you seem so knowledgeable in the field I can only thank you for taking the time to share such interesting and advanced ideas - now I have to go and study up on all the pointers you gave 😅
@@icitry enjoy the discoveries ! and try to learn how each CODEC works : they mostly start with a space-frequency transform, this is how you will lose the leas bandwidth. QR codes have very strong boundaries so a lot of lost high frequency contents, which is lost encoding space. So start with your data, encode it with some time-to-frequency ("generation") algorithm, allocate a certain number of bits per frequency bin, more bits for lower frequency... The DCT is used in JPEG for example and it's a "composable" transform : you apply it in horizontal then vertical and it makes a 2D transform. Apply this for R, G and B with varying bit allocations and spectra. Then you apply the transform in reverse (after temporal and spacial subsampling to reduce the CODEC's artefacts) to recover the approximate values encoded in the spectrum, you requantize and there you can apply your ECC/Reed Solomon to remove the remaining noise. And while I write about this broad outline, I now think of something else and ... OMG. Anyway, have a look at wavelets, they are not too hard to code. FFT is a bit pedantic but it's simple and explained in many textbooks.. Now remember that modern CODECs use overlapping windows so if you have strong changes from one bloc to another, you'll get "ringing" (smoothing) and loss of information. Yeah compression and encoding is one old hobby... I starting by trying to losslessly compress sounds 😀
I'm glad to know there are others like me out there that take the hard route just so they don't have to pay even the slightest amount. 😄 But big props for going through with that!
why the hell would you use QR codes specifically tho, instead of like...data matrix (used for e.g. chip labels) that can tile better and squeeze more info, or even better, actually colored stuff, such as "High Capacity Color Barcode" or "JAB Code" which would be able to hold WAY more info than pure black and white QR code...
Well I more or less dismissed them (probably shouldn't have, or at least presented the alternatives, but given the silly nature of the project, I thought the idea was more important than implementation details 😅) due to the limited improvement I considered them to be bringing to the table, with a maximum 3x with JAB codes. A more radical solution I found while discussing with other people in the comments is using multiple codes in a single frame leading to (100+)x more storage capacity per frame (which could be then coupled with a higher capacity code model to squeeze in even more extra storage)
Isn't the TH-cam compression lossy? As in, you couldn't store 1920x1080 bits of information per frame because TH-cam would probably distort it? I guess the QR codes are big enough for it not to matter
Glad you enjoyed it! From the limited tests I've done it seems to be around 40:1, so at most you could have file chunks of 6.4gb, as the biggest allowed video size to upload is 256gb. But this is all theoretical, we would never abuse TH-cam like this.. 🤐
Thanks! Yep, people have also been suggesting using multiple codes in a single frame and making use of a format that stores more data like JAB codes - which, when cumulated I guess pretty much ends up as a custom qr code format 😅
Not really, the encoding algorithms YT uses are a complete unknown, so the manner in which they manipulate the bits in a frame might as well be completely random
The only efficient way ti do this is to know the exact encoding algorithm. Then you could maximize data per frame assuming youtube streams every frame. It would be interesting if anyone did this approach though its tough as it requires reverse engineering the compression via trual and error and designing an encoding mapping.
Is there a simple way for dumb people like me to store files in youtube? I remember seeing those project videos showing how to do it but I don't know how to use it
You could, but they are pretty short (5k chars) 😅. There have been other suggestions of using the description for metadata and comments as a way of logging changes
Had this exact thought years ago.. TH-cam is free space before he will put down a wall, I think something could happen that prevents users from uploading nonstop
Ok, now hear me out. I don't quite know how data storage works, but what if you stored data through Captions/Transcript? Or even better, the comments? I bet (from a person who knows little about data storage) that this could definitely store way more than the limit of google drive with a combination of these two and QR codes. Also, nice vid! Hope this blows up :)
You could use captions for a bit of extra storage, but they are limited in capacity. As for the comments, you could fragment the data I suppose, but I don't know if reconstructing and/or updating it is worth it from a cost perspective, although it could be further investigated. Someone said to use the comments as a way of logging changes which I found really clever
I've often thought about this idea. Pretty sure if its not already against the TOS, ot will be as soon as any noticeable amount of people start doing this.
I don't think it's feasible enough to be even considered a threat by the YT overlords, there're enough mechanisms in place that would prevent someone from abusing this, I imagine
@@icitry perhaps... what if someone made available an app to back up your entire hardrive for instance? Even just an app to make a backup of all of your photos could be a lot of data. I've only recently uploaded anything to TH-cam, so I'm really not familiar with what type of limitations they place on uploading.
@@onradioactivewaves Well from what I know the max video filesize is 128 gbs, and a basic account has 10 max uploads per day. So you'd be limited from the start just from those 2 parameters. To that it should be added that YT would more than likely flag videos that large, especially if they're not even for public viewing.
@@icitry makes sense . I haven't thought about all of the data requirements and how much TH-cam is already consuming daily. For certain there is more data centers being planned this moment.
Rule 36: If you have an original idea, you're stupid. Someone already made it. (I was thinking the same thing like 2 days ago. Apparently google can now figure out what you're thinking and suggest related videos)
Honestly same, I more or less gave up on trying to find that one completely original idea and instead try to find my own approach to each problem (Also yeah, it definitely can read minds 😅)
sadly is harder to block because like copyright things QR code need Teamyoutube to check it themself not bots because they just see that image and beause QR can`t be really copyrighted as much as Audio or videos because is just a digial link
@Kyermike Yep, any type of video is compressed on most online platforms. It saves tremendous amounts of storage and by extension makes playing videos faster while not being noticeable in most scenarios
Yes, you can convert TH-cam into a cloud storage service. The video explains how the creator built a system using TH-cam as a cloud storage service. They used Selenium Python for automation and created a language parser with the help of ANTLR to fluidify the creation of new commands. The system allows users to upload, download, query, edit, and delete videos, and store any type of file.
@@icitry exactly! Even in industry, top tech companies, I rarely see anyone make this move, then they regret it within a year and just deal with it for the rest of time because nobody wants to maintain it. Hell, "write an XML parser in Java" is a very common interview question. I always ask if that's a trick question, because why would you ever do that?
and is online drive is not is a Public drive for you and google to see So even you paid Ad-free youtube they don`t give you even half price for something they has owned for devs
Oh I am certain there have been other similar implementations, it's pretty difficult to come with a completely new idea nowadays 😅 I just wanted to explore my personal take on the problem and how I would solve it
This is what happens when you let the shower thoughts win.
How dare you say something that perfectly describes my entire existence so casually
@@icitry Can you release the QR code encode and decode code?
Reminder: TH-cam reserves the right to delete channels at will, without warnings or strikes.
Many channels have been deleted in the past for using up too much space and having little to no views.
Makes sense from a business point of view
the solution is to simply make sure the videos have enough views
@@GeneralLDS Jup, but I think no one wants to spend time on watching lots of videos with Black and white nonsense. And the anti-bot to let views not be buyable/ fakeable are there too.
The QR code hack was really clever👍
Thanks! 😄
@@icitryfor the qr-code you can also do the folowing:
1 pixle is a RGB tupple (r,g,b)
If you have (255,255,255) you get white
If you have (0,0,0) you get Black
Instead of saving in black and white you can use the RGB min/max for 3x efficiancy
You might think that you could lose in compresion but the difference between min/max is so big that you can restore the original values from something like (234,247,219)
And (10,9,3)
Also storing metadata in Audio or Video description can be great and comments can be used to log smal changes
TH-cam gives you up to 250.000 coments i Think and each has a maximum of 500 replies with no apperant limit for text size.
Oh great suggestions, I think those types of RGB codes are called JAB codes if I'm not mistaken, I've discovered them after the project was done 😅 But the comments thing is really smart, didn't even cross my mind - big props!
That's why when making projects, you have to think of everything before starting, and more importantly identify the most critical sections and start with them because if they don't work there is no point in continuing
Yep, an extremely important thing that we (I) often forget about or just do superficially 😅
There is actually a method using discord attachments (split in 25mb chunks) which is much less effort since data does not need to be encoded to a image.
Exactly. Where I work we rush solutions into production because there is no budget for evaluation possible solutions... first solution is good enough. Then after month, it is a unmaintainable mess because no one thought about the complex use cases..
I clicked on the video thinking you're some relatively well known coding youtuber... I finished watching the video and I loved it all, then I got out of full-screen, and man... What on earth?? How do you have less than a hundred subs?? My guy the production of such entertaining content really deserves sooooo much more attention. I dropped a like and a sub, hoping you grow larger in the future. Much love.
That makes me so incredibly happy to hear! I can't even begin to thank you for such kind words, they mean so very much to me ❤️ I'm glad I could create something you could enjoy to this level 😁
Same! :)
@icitry
Awh. I'm glad my review meant something.
I would recommend opening up a social platform to form a smaller community. Like discord.
I believe it would help forming a community around the channel where people may discuss parts of your code more in depth. It may also help growing.
Once again, great content, lots of creativity in it. Love it.
Huh, that didn't even cross my mind, I'll definitely look into it! Thank you so much 😊
🙄
When programmers use 100.1% of their brain
But for actually useful stuff it starts throttling at like 10%
@@icitry
sadly youtube will not care much
because if anyone can keep uploading and doing live steams for 3 mouths for scams or loop of giveaway
Why would they try to ban anyone doing that because they think Ad blocker is a proplem
------------------------------------------------------------------------------------------------------------------------------
what about FILES i has in google drive
( 50% of my files are gone )
15GB - 50% - cost to 7.5GB are the real space you can keep
meaning google can spy as they wanted and there no Locks on it even end to end encryption even if you ( PAID their max tier 2TB they still will not give you encryption in anyway )
Only if your in School and business or college to get protected as a USER and not for cookies
Wow! This is amazing! Seems like you put in a lot of work but glad it paid off.
Haha wow it's truly amazing to hear that 😄 And if you have any suggestions / improvement ideas / nit-picks, I'm very open to them, given how I'm still figuring stuff out 😅
Neat! Now all we need is a user friendly GUI for this. And boom. Solid.
Thanks! Yep, and someone willing to do GUI programming 😅
I feel like this is peak reinventing the wheel. It's not about the core idea though, the idea is great. It's about the implementation.
Here's a few suggestions (feel free to completely ignore them lol):
- pls avoid webscraping as much as possible if there's a free, working API for the same thing you want to do (AND THERE IS). Google let's you manage(upload, list, edit download) your entire TH-cam channel with their APIs. They even made a wrapper libs for most languages so they can be used even more easily.
- follow up to the previous suggestion: the whole problem you had with account sessions is completely solvable with OAuth. Just generate a key and out it in an env var or config file.
- don't reinvent the gnu coreutils. Instead, create something that can be mounted as a file system and used with your preferred file management method (commands or a file explorer GUI or anything else). If I were to do this I would -steal- get inspired with *one of* simple S3 server implementations(or WebDAV or FTP or something else. I think S3 is the simplest though) in your preferred language and JUST focus on the CRUD of files (of course using TH-cam APIs). So you'd just replace their createFile function body with your logic and you're done. Then when you have a working S3 server you can either use it in your other apps or mount it as a network drive and just interact with your files in your file explorer.
You spent most of this video doing things completely unrelated to the title (storing files on TH-cam). Instead you should've just used all those existing tools and protocols that people before you made and just do one thing that actually matters right and that's converting a file to a video that TH-cam can't destroy with their compression. Dont make your own (worse) cd and ls commands unless you know how to improve them.
Oh wow, thanks so much for taking the time to write this out! 😄 But the thing is I kinda wanted to reinvent the wheel (also why I wanted the focus of the video to be the whole cloud system, not just the file conversion) - sure if I were to make a production grade solution I'd go with tried and tested solutions, buut where's the fun in that if I can fool around yk? 😅 Also, some counter points - the TH-cam API may be free by default, but you still have maximum daily quotas, I wanted to see if I could get a fully free implementation going; and the generating of an OAuth key is a great idea, but not what I was going for, I wanted to have the user do as little as possible, and (totally imaginary challenge) if someone wanted to use this as-is, I wouldn't want to be sharing secrets or have them generate others (maybe I should've been more clear with that intent). But the implementation definitely has a lot of holes, and your suggestions are more than welcome and closer to what should be done in a production app, so thank you very much for the feedback! 😊
agreed, web scraping bad if there's a free api
Shit like ByteDance does webscraping, why shouldn't we when the bandwidth used is nearly zero to none?
@@flintfrommother3gaming I didn't say webscraping was bad morally. Its slower and a lot less reliable than a single API call so when if you have an alternative (and you do, TH-cam has a free public API), don't use web scraping
reinventing the wheel is what I do daily, I’m a reinventor of sorts. I thought I’d mention
This is waaaay too high quality for a channel this small. This channel better blow up in the next few months or im asking for my money back
😅 Thank you so much for the words of encouragement! Also hey, you can buy low rn and sell high later
Wow thank you algorithm for showing me this video, someone make it so I can come back to this video every time and remember this youtuber exists, this is so good
And thank you so much for giving it a chance and for the incredibly kind words! ❤
My thoughts exactly. This was excellent
That's what I made yesterday but it doesn't work with youtube compression. (did't bother to fix it)
I thought I was a mastermind or something and I realised at 3am of night that I re-invented the wheel (after looking at a repo at github). I went back to sleep with tear in my eye XD.
I am not the only one. You got a sub!
At this point it's 100% a matter of luck on whether you end up reinventing the wheel or not, but hey at least we did something 😄
You should make a FUSE diver so you can literally mount the filesystem and use it natively
I was just thinking this :D
Great video. You can definitely experiment with data density of the QR code. You can try different block sizes as VP9 divides frames anywhere from 64x64 down to 4x4. Also increase quantization from 1 bit to maybe 4-6. Perhaps you can even get away with using the full 8 bits if you include some ECC. Though I wouldn't recommend using all 3 channels of a pixel as chroma is more heavily compressed than luma. Though increasing bits per block may cause the data to be more susceptible to temporal artifacts. Duplicating frames can fix it but it's gonna be a balancing act. To further increase density, you can utilize the audio track with the same method except in 1D.
Oh wow those are some incredibly neat suggestions, I have to actually go and search some bits up to make full sense of them 😅 - you seem way more knowledgeable in this area than me so big props and thank you a lot for taking the time to give such thorough solutions! 😊
@@icitry polychromatic QR codes are a thing. As Kay noted : the density will be lower, so maybe you'll need to apply some colorspace transforms such that you allocate like 2x density on luma and 1x on each chroma.
YAY ENTROPY!!!!
I was expecting this to be the core of the video. After watching it I see why it wasn't.
Just setting up the logistics of a virtual filesystem on youtube was obviously a big enough project.
Using the audio track does not increase density. A silent video is not a video with imagery accompanied by a full-length silent audio track. The audio track is simply absent, and therefore takes up zero space.
@@EvenTheDogAgrees you could encode a kilobyte or two per second in the audio track if you are very careful :-) a few hundred bytes, maybe metadata ?
I've seen a few implementations/videos on this topic, since we worked on this a few years ago on 4chan. Most of them reimplement what we had already developed, but your route of generating and uploading the frames VIA the API is a new take (at least to me) on the solution. This could alleviate the storage issues we ran into, where generated videos are around ~2x+ larger than the input file(s). If I ever get around to testing it, and if it works well enough, would you be fine with me using your idea in Schillsaver?
Definitely, I intend for all the ideas/code/solutions I present in my videos to be open for anyone to use as they please, have fun! 😄
I thought of doing the exact same thing. I thought of the first solution and dropped it for the exact same reason. Then thought of dividing the screen into small squares and make them black or white encoding bits. It’s technically similar to the QR code thing. Kudos for actually doing it
Thank you! And kudos to you for also thinking it through 😄
I want to make something similar but instead it encrypts the data and is able to split the files and combine storage of the various cloud providers. Also, a possible improvement would be to use bigger QR codes as they can be much bigger, also if you don't mind going out of spec they can be infinitely big if you write a decoder for them
Well go for it! Data encryption is something you'd always want and the idea of a distributed system is really cool! 😄 But for the QR codes you can't have them be infinitely big, as you'd end up with the same problems regarding compression when they become too cluttered. Someone else pointed out that I could've used multiple QR codes in a single frame, all within specs though. But great ideas and thanks for the feedback!
@@icitry Still might be able to make them bigger is what I am saying, I would experiment
Oh yea for sure, just that you still need to be careful not to make them too big. But definitely a really nice improvement!
For storage backed directly by file services (Google Drive, Microsoft OneDrive, Dropbox, etc), a project like this happens to already exist. The core project is Least Authority File System (LAFS), which fundamentally handles encryption and manages cross-host storage. I think I recall even a more plug-and-play project that is meant to work with a dozen or so of the largest file storage providers out of the box. I don't remember what that project was called, but I know the core of it is LAFS.
I've done something similar. Encrypt the files -> split them into 24mb chunks -> upload everything to discord and keep a local database of the file names and paths.
many years ago, VHS tapes were used for data storage/backup.
but they didn't use crazy compression schemes so it was "easier" (still a crazy challenge but not like this one)
I've seen another person do something similar, but their solution ot the compression problem was just to make the pixels of data bigger until they weren't affected by compression. I do think your solution was more elegant as it uses an already developed system, meaning you can reuse already optimized code and such.
Back in the early days of the Internet, the 1990s, I used free email accounts for cloud storage, as attachments. No coding. Of course, limited to MBs per account, but my files were small.
Oh that's pretty smart, and is still somewhat viable, just that the file sizes have grown so much in time for even the most basic things, you'd have to split them into chunks and rebuild them on retrieval for each query 😅
Not me with another google account for another 15 GB of storage
How many QR codes can you reliably fit in a frame? Definitely more than 1.
Oh that's a great idea. From what I know it would be something like 2-300 under perfect conditions for a 1080p frame, although I'm not sure how much compression would affect this. But even using something like 50 is an incredible improvement - nice catch!
Couldn't you exploit Google Drive simply by creating a bunch of accounts? When one account runs out of space, just make a new one, but use the old account's email as the recovery email for the new one. I have 3 Google accounts chained together like that.
Huh that's a fun idea, didn't even cross my mind - but what if you had to store something larger than 15 gigs? Guess you would have to break it into parts and add some linking mechanism to piece them together on request. But it's so cool how many different approaches we can have to the same problem, exactly what I love about programming 😄
wait can you explain the recovery email thing?
@@gior987 When you create a Google account, it asks you for an existing email address to use for recovery purposes, and apparently that can be another GMail address, so it's likely possible to create an unlimited amount of GMail accounts that way.
I have a strong feeling your channel will take off with this type of content. 54th sub ✅️ Keep it up!
So glad to have you here! Thank you so much for the encouraging words 😊
87th sub
771st
You can also store data using sound as demonstrated with ggwave.
Great point! It would considerably reduce the necessary file size
@@icitry looking forward what you'll make next, see ya 👍
Awesome, see ya! 😄
Just found the video. Thought it was of the same guy who made the discord storage solution,then I realized u r not him. But loved.ur content. Defo subscribing and liking
Thank you so much! 😊
Why did you make it 1080p? Just use the minimum possible resolution (240p, or lower) and you can use a square video, or any resolution/aspect fine. The only thing that gets scaled by youtube is the resolution, not the aspect ratio
Also 30fps not 60, but the density is in file size, not file time, unless you were storing more than 26GB file.
I actually select the worst available format when selecting the stream URL, otherwise I couldn't reach the target FPS set by the videos - trust me I tried 😅 (also I wanted to emulate the video's style so that's why I'm not going for fixed values).
Man, you're really underrated. I really enjoyed this, though I don't know much about Python lol. You've got a like and a sub from me!
I'm really glad to hear that, thank you so much for the kind words! 😁
@@icitry 🙂
Oh my you got my headgears spinning again...
- using Treillis encoding/decoding of a bytestream, or even some kind of OFDM by splitting the screen in 6*4 zones (for luma) and 3*2 zones for chroma(s) and multiplexing/interleaving each field
- then of course some Reed-Solomon for good measure
- Send 3 identical frames and sample the 3rd one so the decompressor has time to settle the fine details
Of course the Shannon limit (entropy) cannot be exceeded, but this raises another question and makes it interesting to move to the spectral encoding space, which is a totally different beast than QR codes, which have a very high high-frequency (hence entropy, hence loss) component.
Try playing with 2D Fourier transforms (or even 3D !) so you can encode more bits per pixel !
Oh I can definitely tell they're spinning 😄 But you seem so knowledgeable in the field I can only thank you for taking the time to share such interesting and advanced ideas - now I have to go and study up on all the pointers you gave 😅
@@icitry enjoy the discoveries !
and try to learn how each CODEC works : they mostly start with a space-frequency transform, this is how you will lose the leas bandwidth.
QR codes have very strong boundaries so a lot of lost high frequency contents, which is lost encoding space.
So start with your data, encode it with some time-to-frequency ("generation") algorithm, allocate a certain number of bits per frequency bin, more bits for lower frequency...
The DCT is used in JPEG for example and it's a "composable" transform : you apply it in horizontal then vertical and it makes a 2D transform. Apply this for R, G and B with varying bit allocations and spectra.
Then you apply the transform in reverse (after temporal and spacial subsampling to reduce the CODEC's artefacts) to recover the approximate values encoded in the spectrum, you requantize and there you can apply your ECC/Reed Solomon to remove the remaining noise.
And while I write about this broad outline, I now think of something else and ... OMG.
Anyway, have a look at wavelets, they are not too hard to code.
FFT is a bit pedantic but it's simple and explained in many textbooks..
Now remember that modern CODECs use overlapping windows so if you have strong changes from one bloc to another, you'll get "ringing" (smoothing) and loss of information.
Yeah compression and encoding is one old hobby... I starting by trying to losslessly compress sounds 😀
may I know why not just use the youtube api instead of webdriver to upload and download?
I wanted to see if I could have a completely free solution, as the YT API has a maximum daily quota
Reminds me of when I spent weeks manually recovering my corrupted images because I didn't want to spend £9 on some software!
I'm glad to know there are others like me out there that take the hard route just so they don't have to pay even the slightest amount. 😄 But big props for going through with that!
why the hell would you use QR codes specifically tho, instead of like...data matrix (used for e.g. chip labels) that can tile better and squeeze more info, or even better, actually colored stuff, such as "High Capacity Color Barcode" or "JAB Code" which would be able to hold WAY more info than pure black and white QR code...
data matrix and other such b&w stuff are also old enough that every sensible library has had support for them for the last decade, if not more, lol
Well I more or less dismissed them (probably shouldn't have, or at least presented the alternatives, but given the silly nature of the project, I thought the idea was more important than implementation details 😅) due to the limited improvement I considered them to be bringing to the table, with a maximum 3x with JAB codes. A more radical solution I found while discussing with other people in the comments is using multiple codes in a single frame leading to (100+)x more storage capacity per frame (which could be then coupled with a higher capacity code model to squeeze in even more extra storage)
I imagine the video compression works in YUV. Alignment also isn't an issue. Work using those, you could get good density. Don't forget the sound too.
Isn't the TH-cam compression lossy? As in, you couldn't store 1920x1080 bits of information per frame because TH-cam would probably distort it? I guess the QR codes are big enough for it not to matter
Yup, pretty much
Nice video; what did the video file size to inital file size ratio end up being?
Glad you enjoyed it! From the limited tests I've done it seems to be around 40:1, so at most you could have file chunks of 6.4gb, as the biggest allowed video size to upload is 256gb. But this is all theoretical, we would never abuse TH-cam like this.. 🤐
@@icitry Haha, I see, that is pretty big. But yeah, fun experiment
Oh, yea it's definitely not the most practical thing. Otherwise I'd already be hearing knocking on my door 😄
Any other services allowing similar upload, like Facebook?
Nice work. The only option is to use your own qr code format, at least without orientation squares
Thanks! Yep, people have also been suggesting using multiple codes in a single frame and making use of a format that stores more data like JAB codes - which, when cumulated I guess pretty much ends up as a custom qr code format 😅
I mean nice, but couldn't you just a error correcting code when storing and reading?
Not really, the encoding algorithms YT uses are a complete unknown, so the manner in which they manipulate the bits in a frame might as well be completely random
The only efficient way ti do this is to know the exact encoding algorithm. Then you could maximize data per frame assuming youtube streams every frame. It would be interesting if anyone did this approach though its tough as it requires reverse engineering the compression via trual and error and designing an encoding mapping.
Well said, it would definitely be the ideal solution, but it would take quite the herculean task to pull off 😅
Is there a simple way for dumb people like me to store files in youtube? I remember seeing those project videos showing how to do it but I don't know how to use it
why is tim whatley in the thumbnail?
Hal was missing that day so I had to make do
This is genius, never thought of it before
Thanks! 😁
This guy is destined to be some great TH-camr. Who knows, if he becomes like the next code bullet we might get even 3 videos a year lol
Thank you so much for the encouraging words! Also I'll try my best to overcome that nigh-impossible target 😄
1:00 I love it first video I’ve seen and immediately sub earned
Aww thank you so much, I'm glad to hear you enjoyed it!
How you gonna rick roll me for scanning a QR code lol
When I realized I had the power there was no going back
I mean i saw a guy turning discord into his free, encrypted storage server so why not!? 😄
Oh yeah, I've seen it too, great one 😄
How long can descriptions be?
However you could add short videos and descriptions :D
You could, but they are pretty short (5k chars) 😅. There have been other suggestions of using the description for metadata and comments as a way of logging changes
The first thing I did after watching this video was disdislike and ununsubcribe (apply boolean logic here)... Loved these
Well then I have to say I'm very ununhappy to hear you say that
Nice editing bro
Thank you! 😄
Ok ok ok, WHAT IF you upload a movie in this format so they can't block it for copyright? 👀
Hold up, you may be onto something 🤔
Why TH-cam recommend this video after they recommend me watching "Storing Files in Minecraft" 🤣
😄 They know the people yearn for free storage
You seem to be quite adept at coding. How many years of experience do you have?
Hmm, I think somewhere between 4 and 5 years. Started around 11th/12th grade in highschool
This reminds me of the harder drive series
Oh, I hadn't even heard of that, now I gotta look into it
Had this exact thought years ago.. TH-cam is free space before he will put down a wall, I think something could happen that prevents users from uploading nonstop
Oh, it already happened, unverified accounts are limited to 10 uploads per day, for example. So it's mostly a game of cat and mouse 😅
I had fun with treating gmail as a cloud storage from a long time ago.
Hell yeah, that's what we're supposed to do with these big corpos' products
Jea we have definitely the same Ideology.
Solidarity, hell yeah! 😄
Ok, now hear me out. I don't quite know how data storage works, but what if you stored data through Captions/Transcript? Or even better, the comments? I bet (from a person who knows little about data storage) that this could definitely store way more than the limit of google drive with a combination of these two and QR codes.
Also, nice vid! Hope this blows up :)
You could use captions for a bit of extra storage, but they are limited in capacity. As for the comments, you could fragment the data I suppose, but I don't know if reconstructing and/or updating it is worth it from a cost perspective, although it could be further investigated. Someone said to use the comments as a way of logging changes which I found really clever
I've often thought about this idea. Pretty sure if its not already against the TOS, ot will be as soon as any noticeable amount of people start doing this.
I don't think it's feasible enough to be even considered a threat by the YT overlords, there're enough mechanisms in place that would prevent someone from abusing this, I imagine
@@icitry perhaps... what if someone made available an app to back up your entire hardrive for instance? Even just an app to make a backup of all of your photos could be a lot of data. I've only recently uploaded anything to TH-cam, so I'm really not familiar with what type of limitations they place on uploading.
@@onradioactivewaves Well from what I know the max video filesize is 128 gbs, and a basic account has 10 max uploads per day. So you'd be limited from the start just from those 2 parameters. To that it should be added that YT would more than likely flag videos that large, especially if they're not even for public viewing.
@@icitry makes sense . I haven't thought about all of the data requirements and how much TH-cam is already consuming daily. For certain there is more data centers being planned this moment.
Or just store raw data in pixels (or bigger), like "infinite storage glitch"
ur cutting is way to funny 😂🤣🤣😂
Well that's certainly something when taken out of context.. but thanks, glad you liked it! 😄
Remember me when your famous im your 702 sub
😄 Ofc I will, but really - thank you!
Rule 36: If you have an original idea, you're stupid. Someone already made it. (I was thinking the same thing like 2 days ago. Apparently google can now figure out what you're thinking and suggest related videos)
Honestly same, I more or less gave up on trying to find that one completely original idea and instead try to find my own approach to each problem (Also yeah, it definitely can read minds 😅)
sadly is harder to block because
like copyright things QR code need Teamyoutube to check it themself not bots because they just see that image
and beause QR can`t be really copyrighted as much as Audio or videos because is just a digial link
I guarantee I could make a compression algorithm that would destroy a QR code
Just don't let YT get their hands on it and we're gucci 😉
Should I use Google or Google then????
I'd say Google, but Google is also a good alternative
It's a tradeoff between speed and capacity, I presume
Something like that, yeah
Why would you willingly use ANTLR
Sometimes the masochistic side comes to light y'know
Very cool!
Thanks! :)
you could make each pixel a color and each color refers to 9 bits 011,010,100
That would be ideal.. if there wasn't any compression on the video files 😅
wait youtube compresses 1080p and 720p videos???
@@icitry
@Kyermike Yep, any type of video is compressed on most online platforms. It saves tremendous amounts of storage and by extension makes playing videos faster while not being noticeable in most scenarios
TH-cam dialup but data storage instead of data transfer lol
Cool video!
Thank you!
Where u from? U sound familiar
Probably just a coincidence, this channel is the first time I'm ever posting my voice online 😁
@@icitry where u from though?
Sorry, I'd prefer to keep stuff like that private.
@@icitry just your country bro...
You're naughty and I like it hahaha!
I'm not entirely sure if I should be happy seeing this, but oh well, guess I'll take it 😁
Holy god that's crazy!
😄 Glad you liked it!
Hydraulic Press Channel
Goated
it would be very cool if it had FUSE support
Ooh, that is indeed a really cool addition, didn't even cross my mind while elaborating the filesystem, great idea!
please share code for the project
Sorry, but due to the potentially malicious nature of the project I won't be sharing the source code
104th Sub
Glad to have you here!
**subscribing noises**
**happiness + thanking noises**
Yes, you can convert TH-cam into a cloud storage service. The video explains how the creator built a system using TH-cam as a cloud storage service. They used Selenium Python for automation and created a language parser with the help of ANTLR to fluidify the creation of new commands. The system allows users to upload, download, query, edit, and delete videos, and store any type of file.
good, now install arch
Sorry, I'm not that brave yet 😔
Points for making a grammar instead of parsing it in code. Code-based parsers bad.
Thank you! Yep, whenever possible I try to go for the right solution instead of easy hacks that break in a couple of dev cycles
@@icitry exactly! Even in industry, top tech companies, I rarely see anyone make this move, then they regret it within a year and just deal with it for the rest of time because nobody wants to maintain it. Hell, "write an XML parser in Java" is a very common interview question. I always ask if that's a trick question, because why would you ever do that?
Hello
Why hello there
Make project open source
RIP Google drive 😂❤
RIP me after they'll banish me to the shadow realm 😔
and is online drive
is not is a Public drive for you and google to see
So even you paid Ad-free youtube they don`t give you even half price for something they has owned for devs
I thought this was already done by several people. eg th-cam.com/video/8I4fd_Sap-g/w-d-xo.html
Oh I am certain there have been other similar implementations, it's pretty difficult to come with a completely new idea nowadays 😅 I just wanted to explore my personal take on the problem and how I would solve it
YOU ARE SO EVIL!!!
I'll take that as a compliment 😁