Reminds me of when translation software first started becoming popular, the advice was to translate the text then translate it back to see if it still made sense.
One I saw of this recently was English "cologne" -> Spanish "colonia" -> English "colony". It was in a Google-Translate-Sings version of "bad guy" by Billie Eilish
The application for MR to CT is radiation therapy. The soft tissue contrast from MRI is often used to determine the target region, but the CT carries the radio-density information needed for determining the radiation therapy dose distribution plan. Instead of taking both MRI and CT and registering them, the synthetic CT from MRI lets you get the job done without the extra time, expense, and radiation dose of the CT.
Here's an idea: Train a GANN to convert to and from Disney art style, to real life. Then Disney can use it to create a live action remake with basically no budget. Instant profit!
One part of this that I'd like to know more about is the final, round-trip comparison. I'm sure that calculating the 'distance' between original and round-trip image involves something more sophisticated than a pixel-by-pixel comparison.
Fun fact its actually not. There's a few more things that go into 'telling' the generator how it did, but the cycle loss is literally just finding the average difference between each pixel
Great video ! Especially the discussion about how reliable or trustworthy super-resolution can be when using a single image. If the information is not captured by the camera, there's nothing you can do about it. You need some extra knowledge about what you're looking at, like for instance more low resolution pictures.
Yes. It is pretty sad how most peoples scientific knowledge and understanding comes from the TV and films. Even Sci Fi films. Though in this instance I think CSI could be as or more responsible. But many people really do not understand. They really do think a computer can somehow magically come up with unknown information and apply it to an image to enhance it to a much higher resolution, in order to then extract actual information from the picture. But they have seen it happen on TV. So it must be possible. I have had this very argument on more than one occasion.
Oh finally! I've got all these horse pictures left to do and the kindergarten kids are really bad at doing proper shadows, especially around the neck. You're a life saver!
There were cases where the GAN that transforms Horses to Zebras cryptographically encoded the original Horse image into the Zebra image, so that the second GAN can then easily reproduce the original image.
huh “hiding information” in the encoding like that sounds interesting! However, if you not only did H->Z->H but also Z->H->Z, and those come from real images with random nonsense for that, how does it “know” whether it should interpret those or not?
@@AlexKnauth I think most likely it would just interpret what it thinks is the hidden info, resulting a some small random noise. Check out the original paper "CycleGAN, a Master of Steganography" by Chu et al.
My favourite CycleGAN fail story is one that was made to take areal photos and turn into maps. It made nonsense maps and was suspiciously good at recreating lamp posts, fans on top of buildings, and so on. Turns out it did make nonsense maps, then hid information about the original image in the least significant bits of the pixels in those maps. I don't remember where I read about it, though. So you might have to take its truth with a grain of salt.
MasterHigure huh, that “hiding information in the least significant bits” thing sounds interesting! However, if you not only did H->Z->H but also Z->H->Z, and those come from real images with random nonsense for those bits, how does it “know” whether it should interpret those or not?
Oh, another commenter named Karl Kastor also mentoined this. They pointed me to a paper called "CycleGAN, a Master of Steganography" by Chu et al. Is that where you read about this?
This reminded me of the "game" of translating a text, translating it back and laughing at the differences. Could CycleGAN help improve machine translations?
DB Fuller oh yeah maybe. Like one I say recently was English “cologne” -> Spanish “colonia” -> English “colony”. The CylcleGan would have to encode the meaning of “cologne” into it in a way that didn’t lose the information that it wasn’t a colony, somehow.
Possibly, though GANs up until now have been better at images than languages. I imagine the idea of having another network actively undoing the work of the first network could help a lot, though.
I wonder if it is possible to train these systems to undo image processing effects such as a Gaussian blur, and if so how accurate they could be since you could easily create a huge number of paired data points? I know that there are inverse Gaussian blur equations that can be used but these do create a certain amount of error, so it would be interesting to compare the two approaches to see which could produce the most accurate results.
So I can see how the loss functions for the individual images would be applied to their respective networks, but how would the loss function between the two horse images be applied? To each network separately? All together as one big system?
Also, why having two discriminators? If you are comparing the output horse with the input horse, you are already penalising the network if it cannot reproduce the original (real) picture. I understand the one in the middle, not the last one
Yeah that's what I don't understand, it's there to prevent mode collapse but that problem originates from the 1st generator. I'm curious how you would go about applying that loss function to the generator. The only thing I can think is that you would preform backprop and start from the 2nd discriminator and work all the way back to the 1st generator and use those gradients to update the 1st generator. But I have no idea if that's actually how its done.
So because both GANs are used during a full cycle, horse -> fake_zebra uses gan A and fake_zebra -> cycle_horse uses gan B, the loss between those images would be applied to both GANs, but not the discriminators. You need two discriminators because each one is applied to one GAN. GAN A will need a dedicated discriminator to take the fake zebra it produces and the real zebra. The discriminator here scores both of these individually, giving a higher score if it thinks its a 'real zebra'. We also need a discriminator to do this for knowing what a 'real horse' looks like as well for GAN B, hence two discriminators.
But isn’t “zebrafying” a horse a lossy process? What I mean is if you convert a brown horse to a zebra and you convert a white horse of the exact same size to a zebra they should produce the same result. But then how will the inverse function know what color to turn it back into?
Ankith Udupa I would like to know this as well. I wonder if it would try to “hide” the encoded information of the horse color in something like the placement of the stripes? But then when the inverse network is given a real Zebra that information wouldn’t be there... would it come out as a nonsense horse-color? Another commenter said a similar thing about lossy process with converting to a Monet-painting
Another commenter named Karl Kastor also mentioned something like this. They pointed me to a paper called "CycleGAN, a Master of Steganography" by Chu et al. I thought that was interesting, it used conventing aerial photos -> maps as an example of a lossy process, and showed that adding tiny imperceptible random noise to the map changed the aerial photos dramatically, so somehow it was encoding that "lost" information where it thought we wouldn't notice
One generator is predicting the input image, and the other is predicting the zebra / modified input image. So I think whatever we are reverse-predicting is just whatever the input image is. The wrong color would have a high loss.
Probably not. There's two options, horse or zebra. The discriminator can't say "neither" so it won't try to convince it that "neither" is the wrong option. If there's no zebra then it's likely to make the terrain look like it would be around a zebra. Like he mentioned in the video that the grass changes colour, probably because where zebras naturally live the grass is a different colour.
I wonder what happens if you do it with very different images, like horses to oranges back to horses, or Computerfile videos to Numberfile images back to Computerfile videos.
I saw NASA pull out a licence plate number from a few pixels by tracking the differences between frames on a movie and back calculating what the image must have been to produce that. Maybe combining that idea with this one can get a better 2001.
@@lucidmoses yes. If its a video it can be sampled in deepth. This increase the virtual resolution quite a bit There is software that used in cameras to take several puctueres to make s Sharp One.
If one were to convert to a monet painting then it would be lossy so wouldn't a cyclegan incentivise it to be a bit less like a monet so that there is enough data to covert it back to the original? If the network gives you the same zebra every time but places the original, shrunk down and in the corner then the reverse network can just take that corner and expand it? Does that ever happen?
First question: That is probably true and depends on how you weight the two losses. Second question: A similar thing like what you said actually happened: The network stenographically encoded the original input into the Zebra image and then the reverse network could decode this.
Karl Kastor what happens when the reverse network is given a real Zebra image? or a real Monet painting is the more interesting case. The reverse network might be looking for encoded information that isn’t there, so would it be nonsense data when it tries to interpret it?
@@AlexKnauth It might not stenographically hide the whole image, but just some parts that need to be changed. But these parts would be random if the hidden info is not there, yeah. Check out the paper "CycleGAN, a Master of Steganography" by Chu et al. for more info
For fixing the flickering: DVDs record the things which change between two consecutive images in a movie. Could we train an additional GAN to distinguish between such changes in normal movies (basically DVD data for real movies), perhaps even the original (style-unchanged) movie, and the changes which produce flickers in the style-changed product movies?
The real loss needs to calculate how the fake prodused is like intended product. I.e. photo to rembrant painting style, then how well the produced painting is similar to rembrant style..
Absolutely. What's more to reach further and assume unicorn's should only be done based on pretty sound demonstrable evidence. Evidence that at least to my knowledge hasn't been remotely detected, let alone reasonably, scientifically demonstrated.
What if you could reduce images into its descriptive parts, then they could be removed from the image....each part can be compressed separately, then you have a higher ratio compression for video....and you can create internet 2.0...lol.
I guess the results could be improved a bit by adding a few free output neurons that are not part of the zebra image. These could be used by the first GAN to encode the information it destroys by turning a horse to a zebra. This could then be used by the second GAN to reconstruct a better image. It should improve the quality of the result because it gives more freedom to the first GAN to modify the image, but not so much that it could just throw a generic zebra.
Dr. Pound shoulder shirt tug tally: 6. We need an algorithm to predict if the next video will have increasing or decreasing number of shoulder shirt tug twitches.
That part in the beginning about commissioning artists to paint pictures for the GAN as direct examples... I've long had this idea that Hollywood needs to take digital cameras, and film cameras, and have them shoot at a variety of things in tandem, then run that data through a GAN, so we produce a system to make digital footage look like film footage, and rid ourselves of all the weird digital artifacts we see in films today. Mainly I'm just on a campaign to change the way digital photography handles bright coloured lights.
Most brutal insult to artist: "An untrained GAN is better than you."
That is the best compliment.
i'd honestly be kind of embarrassed for the GAN, if an artist was better than it.
I’ll take that challenge
I'll start on the ai art critic and feed it disney and all episodes of the simpsons ....
Shows a total lack of understanding of art. Technical proficiency isnt what art is.
Please...can we just appreciate how clean Mike's board is?
I see Dr Mike Pound, I watch
Reminds me of when translation software first started becoming popular, the advice was to translate the text then translate it back to see if it still made sense.
Hence the likely apocryphal "The vodka is agreeable, but the meat has gone bad."
One I saw of this recently was English "cologne" -> Spanish "colonia" -> English "colony". It was in a Google-Translate-Sings version of "bad guy" by Billie Eilish
Summer Clothes and Mike Pound = legend
Ambient temps srsly f*ing with his bioware.
I like how he is even optimising the scribble time by painting the generator differently at 5:26 -before in 1:30 was slower.
Dr Mike Pound! This guy is the bessttt 😎
I don't care if I'm first or second or anything like that, just wanted to use this opportunity to tell that this channel is amazing!
The application for MR to CT is radiation therapy. The soft tissue contrast from MRI is often used to determine the target region, but the CT carries the radio-density information needed for determining the radiation therapy dose distribution plan. Instead of taking both MRI and CT and registering them, the synthetic CT from MRI lets you get the job done without the extra time, expense, and radiation dose of the CT.
Ah, you've said what I was going to say.
Here's an idea: Train a GANN to convert to and from Disney art style, to real life. Then Disney can use it to create a live action remake with basically no budget. Instant profit!
Shhh!! Why would you want to give them even _more_ money?!
I would wanna use it the opposite way. Take a live-action movie like Star Wars or something, and turn it into an animated movie!
Skynet is not an improvement over Disney, please, stop having ideas... ;)
That's not new; animated picture industry is actually actively dabbing into that kind of stuff. Please look forward to it.
One of the best educational channels on TH-cam. Period.
I can recommend the 2001 SO Picasso version, it's amazing and should have been linked in the description.
11:58 The algorythm zebraed the wooden pole on the left too 😂😂
Tommaso Morandini oh wow, Hahhaahahhha!
Dr. Pound is the best
It can also be used for 'Blind-Denoising' for cleaning EEG signals.
One part of this that I'd like to know more about is the final, round-trip comparison. I'm sure that calculating the 'distance' between original and round-trip image involves something more sophisticated than a pixel-by-pixel comparison.
Fun fact its actually not. There's a few more things that go into 'telling' the generator how it did, but the cycle loss is literally just finding the average difference between each pixel
Great video ! Especially the discussion about how reliable or trustworthy super-resolution can be when using a single image.
If the information is not captured by the camera, there's nothing you can do about it. You need some extra knowledge about what you're looking at, like for instance more low resolution pictures.
Yes. It is pretty sad how most peoples scientific knowledge and understanding comes from the TV and films. Even Sci Fi films. Though in this instance I think CSI could be as or more responsible.
But many people really do not understand. They really do think a computer can somehow magically come up with unknown information and apply it to an image to enhance it to a much higher resolution, in order to then extract actual information from the picture.
But they have seen it happen on TV. So it must be possible.
I have had this very argument on more than one occasion.
Oh finally! I've got all these horse pictures left to do and the kindergarten kids are really bad at doing proper shadows, especially around the neck.
You're a life saver!
these days i just use the cyclegan to do voice conversion, what a coincidence to see this!
😊 Thank you for explaining this optimization-loop
Excellent explanation❤
Im going to be honest here, I will watch any video by this guy just to listen to his voice.
There were cases where the GAN that transforms Horses to Zebras cryptographically encoded the original Horse image into the Zebra image, so that the second GAN can then easily reproduce the original image.
These damn AI continuously working around the problem set... Almost human-like!
Can you link where you got that from?
huh “hiding information” in the encoding like that sounds interesting! However, if you not only did H->Z->H but also Z->H->Z, and those come from real images with random nonsense for that, how does it “know” whether it should interpret those or not?
@@AlexKnauth I think most likely it would just interpret what it thinks is the hidden info, resulting a some small random noise. Check out the original paper "CycleGAN, a Master of Steganography" by Chu et al.
@@karlkastor Oh, thank you!
“We’ll put a link in the description”, he sounds like a regular youtuber now
and then there is no link
GAN of Zebras could be an amazing story to tell children!
More zebras? I like this idea already.
basically you are assuring that F (and in return also G) are bijections, i.e. they don't "collapse" the domains
I need to get into this kind of thing.
I'd love to build one to remove telephone/power lines from my photos.
Stuff about Generative Adversarial Networks? This a-GAN?!
(sorry)
get out lol
Don't be such a discriminator.
Dr Pound rocks.
Finally I can fulfill my life-dream of knowing what kind of bread I am
Thanks BuzzFeed
Sooo can you convert a real Monet painting into a photograph?
My favourite CycleGAN fail story is one that was made to take areal photos and turn into maps. It made nonsense maps and was suspiciously good at recreating lamp posts, fans on top of buildings, and so on.
Turns out it did make nonsense maps, then hid information about the original image in the least significant bits of the pixels in those maps.
I don't remember where I read about it, though. So you might have to take its truth with a grain of salt.
MasterHigure huh, that “hiding information in the least significant bits” thing sounds interesting! However, if you not only did H->Z->H but also Z->H->Z, and those come from real images with random nonsense for those bits, how does it “know” whether it should interpret those or not?
Oh, another commenter named Karl Kastor also mentoined this. They pointed me to a paper called "CycleGAN, a Master of Steganography" by Chu et al. Is that where you read about this?
I love Computerphile
This reminded me of the "game" of translating a text, translating it back and laughing at the differences. Could CycleGAN help improve machine translations?
DB Fuller oh yeah maybe. Like one I say recently was English “cologne” -> Spanish “colonia” -> English “colony”. The CylcleGan would have to encode the meaning of “cologne” into it in a way that didn’t lose the information that it wasn’t a colony, somehow.
Possibly, though GANs up until now have been better at images than languages. I imagine the idea of having another network actively undoing the work of the first network could help a lot, though.
I wonder if it is possible to train these systems to undo image processing effects such as a Gaussian blur, and if so how accurate they could be since you could easily create a huge number of paired data points? I know that there are inverse Gaussian blur equations that can be used but these do create a certain amount of error, so it would be interesting to compare the two approaches to see which could produce the most accurate results.
🍻 Thank You!
Sadly there are lots of mistakes in the subtitles.
How does this person have all this knowledge for all those different topics?
He watches Computerphile
Here's a business oppourtunity: train a GAN to covert my crappy photos into professional photos, and then become a professional photographer.
So I can see how the loss functions for the individual images would be applied to their respective networks, but how would the loss function between the two horse images be applied? To each network separately? All together as one big system?
Also, why having two discriminators? If you are comparing the output horse with the input horse, you are already penalising the network if it cannot reproduce the original (real) picture. I understand the one in the middle, not the last one
Yeah that's what I don't understand, it's there to prevent mode collapse but that problem originates from the 1st generator. I'm curious how you would go about applying that loss function to the generator.
The only thing I can think is that you would preform backprop and start from the 2nd discriminator and work all the way back to the 1st generator and use those gradients to update the 1st generator. But I have no idea if that's actually how its done.
So because both GANs are used during a full cycle, horse -> fake_zebra uses gan A and fake_zebra -> cycle_horse uses gan B, the loss between those images would be applied to both GANs, but not the discriminators.
You need two discriminators because each one is applied to one GAN. GAN A will need a dedicated discriminator to take the fake zebra it produces and the real zebra. The discriminator here scores both of these individually, giving a higher score if it thinks its a 'real zebra'. We also need a discriminator to do this for knowing what a 'real horse' looks like as well for GAN B, hence two discriminators.
But isn’t “zebrafying” a horse a lossy process? What I mean is if you convert a brown horse to a zebra and you convert a white horse of the exact same size to a zebra they should produce the same result. But then how will the inverse function know what color to turn it back into?
Ankith Udupa I would like to know this as well. I wonder if it would try to “hide” the encoded information of the horse color in something like the placement of the stripes? But then when the inverse network is given a real Zebra that information wouldn’t be there... would it come out as a nonsense horse-color? Another commenter said a similar thing about lossy process with converting to a Monet-painting
Another commenter named Karl Kastor also mentioned something like this. They pointed me to a paper called "CycleGAN, a Master of Steganography" by Chu et al. I thought that was interesting, it used conventing aerial photos -> maps as an example of a lossy process, and showed that adding tiny imperceptible random noise to the map changed the aerial photos dramatically, so somehow it was encoding that "lost" information where it thought we wouldn't notice
One generator is predicting the input image, and the other is predicting the zebra / modified input image.
So I think whatever we are reverse-predicting is just whatever the input image is. The wrong color would have a high loss.
We want to see this done! Please show
Smart guy. Vexed by the letter r. Great video all the same.
I was wondering if you could use this time of strategy for text translation? Say spanish to english
Right?
First time i'm seeing mike wearing a not-so-much corporate shirt OMEGALUL
Please Please Please do a StyleGan video
Could you use a Gan like this to turn live-action movies into animated movies? Basically the reverse of a disney-remake?
Great video as usual , Can you talk about the technology used by faceapp ?
What would happen if you give it a photo that doesn't include a horse? Would it try to add zebra stripes to anything?
Probably not. There's two options, horse or zebra. The discriminator can't say "neither" so it won't try to convince it that "neither" is the wrong option.
If there's no zebra then it's likely to make the terrain look like it would be around a zebra. Like he mentioned in the video that the grass changes colour, probably because where zebras naturally live the grass is a different colour.
What prevents the two GANs from cooperating?
InputA->FalseContextButValidB->ReproducedA
The next big conspiracy theory?
The most amazing thing for me is 5:25 instant improvement of drawing generators/that symbol!
who is ROB? is there a link to his video?
I wonder what happens if you do it with very different images, like horses to oranges back to horses, or Computerfile videos to Numberfile images back to Computerfile videos.
Joshua Hillerup check out the paper website. They do people to vegetables and back.
@@MusicBent thanks! My favourite is the face/ramen one.
can someone explain at 8:25, styling and content?
I saw NASA pull out a licence plate number from a few pixels by tracking the differences between frames on a movie and back calculating what the image must have been to produce that.
Maybe combining that idea with this one can get a better 2001.
That is a deconvolution and it works if you have calibrated optics. Used in astronomy.
Didnt they use video? Interpoliate in time does increase the accuracy.
@@matsv201 Is there a relevant difference between 'movie' and 'video' that you feel is important?
@@lucidmoses yes. If its a video it can be sampled in deepth. This increase the virtual resolution quite a bit
There is software that used in cameras to take several puctueres to make s Sharp One.
@@matsv201 The main question being... How is that different then a movie.
Isn't that the horse from 'horsin' around'?
CycleGAN is the new GucciGang
Are we also able to produce a okapi in the end?
If one were to convert to a monet painting then it would be lossy so wouldn't a cyclegan incentivise it to be a bit less like a monet so that there is enough data to covert it back to the original?
If the network gives you the same zebra every time but places the original, shrunk down and in the corner then the reverse network can just take that corner and expand it? Does that ever happen?
First question: That is probably true and depends on how you weight the two losses.
Second question: A similar thing like what you said actually happened: The network stenographically encoded the original input into the Zebra image and then the reverse network could decode this.
Karl Kastor what happens when the reverse network is given a real Zebra image? or a real Monet painting is the more interesting case. The reverse network might be looking for encoded information that isn’t there, so would it be nonsense data when it tries to interpret it?
@@AlexKnauth It might not stenographically hide the whole image, but just some parts that need to be changed. But these parts would be random if the hidden info is not there, yeah. Check out the paper "CycleGAN, a Master of Steganography" by Chu et al. for more info
@@karlkastor Oh, thank you!
Pound the like button for Dr Mike
For fixing the flickering: DVDs record the things which change between two consecutive images in a movie. Could we train an additional GAN to distinguish between such changes in normal movies (basically DVD data for real movies), perhaps even the original (style-unchanged) movie, and the changes which produce flickers in the style-changed product movies?
Loss function? More like I II II IL ()
Nice
Is this...
The real loss needs to calculate how the fake prodused is like intended product. I.e. photo to rembrant painting style, then how well the produced painting is similar to rembrant style..
thank you!!!
Didn't know Stingy from LazyTown was so smart!
And why exactly am I still answering ReCaptchas???
Surely the "F" generator is a "GG" generator
Completely random, but does anyone see the similarity between Tim Urban and Dr. Mike Pound?
Dr Mike Pound + ukiyoe GAN = Steve Buscemi
“An untrained GAN might have been better.” Haha I’m such a nerd.
So we have 2 gans, because if 1 is good then 2 will be better
Its pronounced zebra not zebra
10:08 damn you compression artifacts
I was expecting you to reveal at the end of the video that the styling of your shirt was being CycleGAN-generated all along.
could a yolo into cycle gan be a thing to smooth out the noise?
Yolo is for object detection, how would it help?
@@WhoForgot2Flush well you only want to change the horse to zebra so detect the horse and use that as input
Zebras are painted that way
He lets it slip @3:38. Look at that grin.
The caption is full of errors...
Mikey dressing for English weather 🙃
When you hear hoof beats think horses not zebras.
Absolutely.
What's more to reach further and assume unicorn's should only be done based on pretty sound demonstrable evidence.
Evidence that at least to my knowledge hasn't been remotely detected, let alone reasonably, scientifically demonstrated.
cool
The video is great and all but am i the only one bothered by the sketching sound of the marker?
Take progressive scan as input. Generate interlaced video as output :-)
What if you could reduce images into its descriptive parts, then they could be removed from the image....each part can be compressed separately, then you have a higher ratio compression for video....and you can create internet 2.0...lol.
Peter Parker appears again
Take oooon meeee
I guess the results could be improved a bit by adding a few free output neurons that are not part of the zebra image. These could be used by the first GAN to encode the information it destroys by turning a horse to a zebra. This could then be used by the second GAN to reconstruct a better image.
It should improve the quality of the result because it gives more freedom to the first GAN to modify the image, but not so much that it could just throw a generic zebra.
"an untrained GAN might have been better" x) xd
Is tail of a zebra black and white like in video they showed? 😂
This seems like something that could be solved with dropout. Over fitting problem
“Zeb bra”
Can I work as a discriminator?
I have a gun!
I mean... a GAN :-)
Zero
Maybe it will help Mass produce VR world's????
Nvidia has been looking into that. You can find TH-cam videos about it.
@@WhoForgot2Flush cool thanks
How to turn horses into zebras? A lot of unpaid interns and black markers!
Dr. Pound shoulder shirt tug tally: 6. We need an algorithm to predict if the next video will have increasing or decreasing number of shoulder shirt tug twitches.
Woah, I'm early.
That part in the beginning about commissioning artists to paint pictures for the GAN as direct examples... I've long had this idea that Hollywood needs to take digital cameras, and film cameras, and have them shoot at a variety of things in tandem, then run that data through a GAN, so we produce a system to make digital footage look like film footage, and rid ourselves of all the weird digital artifacts we see in films today.
Mainly I'm just on a campaign to change the way digital photography handles bright coloured lights.