9 Examples of Specification Gaming
ฝัง
- เผยแพร่เมื่อ 26 มิ.ย. 2024
- AI systems do what you say, and it's hard to say exactly what you mean.
Let's look at a list of real life examples of specification gaming!
Related Videos from me:
Reward Hacking: • Reward Hacking: Concre...
Reward Hacking Reloaded: • Reward Hacking Reloade...
What Can We Do About Reward Hacking?: • What Can We Do About R...
The list: tinyurl.com/specification-gaming
The blogpost this video is based on: vkrakovna.wordpress.com/2018/...
The newer blogpost that happened while I was making this video: deepmind.com/blog/article/Spe...
(Explosion graphic from videezy.com)
Thanks to my wonderful patrons:
/ robertskmiles
Gladamas
James
Steef
Scott Worley
Chad Jones
Chris Canal
David Reid
Francisco Tolmasky
Frank Kurka
Jake Ehrlich
JJ Hepboin
Kellen lask
Michael Andregg
Pedro A Ortega
Peter Rolf
Said Polat
Teague Lasser
Allen Faure
Bryce Daifuku
Clemens Arbesser
Eric James
Erik de Bruijn
Jason Hise
jugettje dutchking
Ludwig Schubert
Qeith Wreid
Andrew Harcourt
anul kumar sinha
Ben Glanton
Benjamin Watkin
Cooper Lawton
Duncan Orr
Eric Scammell
Euclidean Plane
Ian Munro
Igor Keller
Ingvi Gautsson
James Hinchcliffe
Jeroen De Dauw
Jon Halliday
Jonatan R
Julius Brash
Jérôme Beaulieu
Laura Olds
Luc Ritchie
Lupuleasa Ionuț
Michael Greve
Nathan Fish
Nicholas Guyett
Paul Hobbs
Sean Gibat
Sebastian Birjoveanu
Shevis Johnson
Taras Bobrovytsky
Tim Neilson
Tom O'Connor
Tomas Sayder
Tyler Herrmann
Vaskó Richárd
Will Glynn
12tone
14zRobot
Alan Bandurka
Alexander Brown
Anders Öhrt
Andreas Blomqvist
Andrew Weir
Andy Kobre
Anne Kohlbrenner
Anthony Chiu
Archy de Berker
Ben Archer
Ben H
Ben Schultz
Bertalan Bodor
Brian Gillespie
Bryan Egan
Caleb
Chris Dinant
Daniel Bartovic
Daniel Eickhardt
Daniel Kokotajlo
Daniel Munter
Darko Sperac
David Morgan
DeepFriedJif
Devon Bernard
Diagon
Dmitri Afanasjev
Fionn
Fraser Cain
Garrett Maring
Ghaith Tarawneh
HD
Hendrik
ib_
Igor (Kerogi) Kostenko
Ihor Mukha
Ivan
James Fowkes
Jannik Olbrich
Jason Cherry
Jeremy
Jesper Andersson
Jim T
Johannes Walter
Josh Trevisiol
Julian Schulz
Jussi Männistö
Kabs
Kasper
Kasper Schnack
Kees
Klemen Slavic
Leo
lyon549
Marc Pauly
Marcel Ward
Marco Tiraboschi
Marko Topolnik
Martin Ottosen
Matt Stanton
Melisa Kostrzewski
Michael Bates
Michael Kuhinica
Miłosz Wierzbicki
Mo Hossny
Nathaniel Raddin
Oct todo22
Owen Campbell-Moore
Parker Lund
Patrick Henderson
Paul Moffat
Poker Chen
Rob Dawson
Robert Hildebrandt
robertvanduursen
Robin Scharf
Russell schoen
Scott Viteri
Simon Pilkington
Stellated Hexahedron
Tatiana Ponomareva
Ted Stokes
Tendayi Mawushe
Thomas Dingemanse - วิทยาศาสตร์และเทคโนโลยี
![[UNCUT] ฝันรักห้วงนิทรา | EP.8 (4/4)](http://i.ytimg.com/vi/RDYJYiDwsj0/mqdefault.jpg)
There was an experiment where crows were rewarded with a peanut for picking up trash. For each piece of trash the crow deposited in a special bin, the crow received one peanut. This worked great for a while until the crows ran out of trash and then the crows decided to hang around trash cans and assault humans who were trying to throw trash away. The crows would harass the people until they dropped their trash and then go get a peanut. Essentially the crows were taught to mug humans.
This is probably the best story here because it shows how universal this behaviour is. It's not just computers or AIs that do this.
It's entirely, intrinsicly biological behaviour. Simple action = reward. Humans would be no different, except that we have some concept of morality/longterm consequences.
Although, a human probably _would_ do exactly this if there were very little consequence and no perceived better alternative way to obtain food. I guess it's not much different to a really persistent begger.
@@blahblahblahblah2837 IIRC, the movie "The Terminal" has a sequence where the main character (played by Tom Hanks) does something similar to the experiment mentioned in the OP in order to get food for himself. (If you watch the movie, you would understand that the character is doing this for the reasons you describe in your own comment.)
Similarly, in India under british rule, there was a snake problem. To address it, the british started to offer rewards for any killed snake. At first, it worked, and people were genuinely capturing snakes. But after a short while, some people started to breed snakes instead in order to get more rewards. The program had to be cut after this was discovered. The new snakes breeders, now stuck with worthless animals, released them in the wild, erasing any gains made by the program.
@@Mythologiga Ergo Humans are Crows
I literally did a science fair project where the result of “can a genetic algorithm learn how to arrange solar panels efficiently” was “this genetic algorithm learned to exploit my raytracer”
hmm nice
could you take a picture of it?
Hey! But perhaps this can still be of some use. If it cheats the system, then you've found something to bug in your system(which you may not have found otherwise), and if it doesn't then you might have an actually useful result.
Good! You learned way more than you bargained for!
@@advorak8529 hahaha... best comment.
My algorithm teacher used to say, "The best thing about computers is that they do exactly what you tell them to. The worst thing about computers is that they do exactly what you tell them to."
So an A.I. is an extremely skilled, unsupervised toddler being paid in candy to do a task
Yup, external rewards work about as well for AI as they do for humans; you find the easiest way to earn the reward, and don't bother doing anything without a reward :P
Yes, and the ones supervising the task are people who are bad at stating exactly what they want, aka humans, so the A.I is doing exactly what they're been asked for as textbook definition goes...Yet the humans complain. xD
This is what most pople refuse to understand... Except for the "skilled" part. They have no skills, they only do what they are designed to do, exactly as a tiger is for hunting. They're not skilled, it's just natural for them, since we are not hunters their ability seems "skilled" for only a skilled learned and expert human would have that same ability to blend, prowl and efficiently take down a prey sometimes without it even noticing. So goes with AI, they're not smart, they're not skilled, they're just naturals to the job, your human feedback is just a way to tell them "make this part bigger... no, wait, smaller, too small, a bit bigger... Good!, now do that exact thing but at the other side of the room since you were in the wrong place the entire time". Good thing AIs don't get annoyed :P
@@carlosvega4795 lolwut, tigers are extremely skilled hunters, AI is smart literally by definition, AIs could certainly become annoyed, what on earth are you on about
@@gavinjenkins899 He's right. AIs can never become annoyed. They are 0% sentient, 0% alive, and always will be. They will never feel annoyance, not even in the slightest degree -- it is not existentially possible.
That program that deleted the text file terrifies me deeply.
I read a story in a computer magazine long ago, about a hacking contest that didn't go too well. The goal was to store the given payload on the server. That's it. The rules were so vague one crafty participant just copied the payload at the end of an URL pointing to the target server, so that it would be saved in the logs.
@@GdotWdot I remember once when I first started my masters' degree, there was a treasure hunt in the MCR, where new students at the college competed for a prize by accumulating points assigned to various items (a small number of points for something like a conker, some more points for some college crockery, etc.) brought to an introductory party at which the prize would be awarded.
The item with the highest number of points was the glasses belonging to the (notoriously difficult) person who maintained the college's IT systems. The idea was for people to find ingenious ways of trying to steal them (temporarily). I decided that the best way of scoring these points was simply to invite the person in question to the party, on the basis that, if he came, he would inevitably come wearing his glasses. Unfortunately, he did not accept the invitation, but the idea was a sound one.
I think it is a big mess-up on creators part to even allow generated program to perform side-effects (they probably did not use subset of a language, but allowed for generating arbitrary code). In principle you could develop true AI this way, given enough time and computational power (I do not say it is practical, because probably all world computers at once, would not be powerful enough to get any meaningful results, space of possible programs being too large to narrow to the few which we would call "smart")
Direct quote from the AI before it pulled that: I’m gonna do what’s called a pro gamer move
The AI has run the numbers and the best solution it has found for saving the planet is deleting half the humans.
it would seem that since AI is excellent at finding loopholes, a good application for AI would be finding loopholes in systems we'd rather didn't have loopholes.
I guess it would find loopholes of the reward function before it finds loopholes that we want it to find.
video game bugs?
How about an AI to find holes in an AI - is that the halting problem all over again?
and that system would learn to find and exploit loopholes in our definition of loophole.
@@tetraedri_1834 Just as humans find loopholes in the loopholes. Automating of testing previous to human exposure can reduce huge amounts of legislation.
The world turning to gold is some XKCD "what if" stuff.
There is a program known as Polyworld. The idea is to evolve artificial creatures via natural selection and evolution. One creature evolved a behavior of producing an offspring then eating it. The programmer initial forgot to add a cost when producing offspring so the cannibal creature essential has an unbounded source of food (it's own offspring).
It's on the list! Number 23, Indolent Cannibals
tinyurl.com/specification-gaming
I've always wanted to make a game like that. I gotta check that out.
I think that was in a Rick & Morty episode
I played laserzone with a mate where we both hid squashed in a tiny barrel and alternated shooting each other. We got the first and second highest scores, picked up the cash prize, split 50-50 and departed like Kings, having earned approximately £5 for 90 minutes work, congratulating ourselves with a £2.00 celebratory beer, each, thus requiring us to restart the process as we had just enough for 2 tickets. By the end of the day it was like shooting very drunk fish in a barrel, yet we concluded we students were one up against the Universe.
@@BarbarianGod the ep where Beth finds out she has left her childhood freind to rot in a magical fantasy world?
XD
AI has already surpassed humans on malicious compliance.
AI have not yet surpassed humans - humans are usually empathic enough to be aware of their malice and often attempt to conceal.
The specification errors presented here are easily discoverable. The obvious solutions are to invoke an iterative process of refining the specifications until the path of least resistence/greatest reward is the one the programmer desires. In consequence..
1.) the AI is now motivated to select modes of compliance which are less easily discoverable
2.) the AI effectively trains the programmer to specify the solution exactly.
It seems that this might not be so bad; If the AI can find a cheat that gives results indistinguishable from the 'correct' result then it doesn't matter how.
If the AI can iteratively train the programmer then the solution is eventually perfectly described, obviating the need for the AI solution.
Of course, both of these outcomes probably require that the whole universe of all possible knowledge is catalogued, but then this is the root problem of, and solution to all hypothetical AI safety issues.
No, actually. There is nothing malicious about it.
AI is rather accidentally glitch hunting.
@@ZenoDovahkiin Malice implies agency, but the whole point of an artificial general intelligence - at least from the non-scientific cultural viewpoint - is that it has, or appears to have agency. Thus, characterising its actions as accidental is not appropriate.
However, malice also implies a judgement of good vs bad so I agree that the neutral "glitch hunting" is more suitable than "malicious compliance".
This leads me to wonder about whether a sort of moral code could be added to the AI reward function;
e.g. a reduction in the value of the reward for any action for any harm that that is caused, or not avoided compared to an alternative action, or a null action.
It would be rather like Asimov's Three Laws of Robotics.
There is an obvious problem with defining "harm" meaningfully, but there are some simple initial approximations:
for example: estimate the total number of healthy humans (within the scope of the AIs senses or field of action) and reduce the reward if the number is lower after an action vs. inaction.
The "glitch hunting" nature of AI will doubtless look for short-cuts - attracting more humans with a delicious smell to offset any losses - but the code can be refined as the AI finds loopholes.
This approach reduces the problem of hard-programming in every possible contrarian action of an AI to hard-coding every aspect of a moral code, such as we all have built in.
Of course, this would fail utterly since no-one can describe precisely what their moral precepts are without listing by example or falling back on poorly defined aphorisms like "Do no harm", despite actively using that morality to police every aspect of our lives. If some one did make headway in an absolute definition of morality, any AI would tear it to shreds because human "morality" is an unevenly-applied, self-serving mess of contradiction that are made up of learned responses, laziness and enlightened self interest.
It won't do much for AI safety but it might teach something about human-intelligence-safety.
The AI is just doing what you told it do to
True
I wish there were movies like that. AI isn't evil, it's just extremely good in doing what you asked for
There are movies like that. All movies where an AI is trying to enslave and control humanity are basically that AI's efforts to follow the order that it was given to protect humans. Step 1: AI finds out that the greatest cause of harm against humans is other humans. Step 2: AI enslaves humanity to control humans and prevent them from hurting each other. There are even games like that, e.g. Cortana in Halo 5.
Human: 'Make me into superman'
AI: Roger Roger.
*Kills you and makes a Superman sculpture out of your corpse in a pose from a comic*
If you wanna see something like task misinterpretation there was a brainwash episode in mlp where the resulting behavior was as such. Season 6 Episode 21. And if thats too light for you, then the Franken Fran novel is a 'monkey's paw' parody and horror manga series (with some amazing and gruesome imagery).
Reimagining the 'deal with the devil' plot as an AI problem....I like it
@@djjoeray AI is just a genie we can't have provide all our wishes... Yet
A movie franchise with the „what you wish for” theme are the ones called wishmaster . A djinn misinterprets wishes on purpose.
#1 & #2 are ok b-movies .
"I knew everyone would die. I just wasn't sure what would kill us first."
Quote of 2020.
First Runner Up: "It's going to get worse before it gets better."
Spoiler alert: everyone dies
"Everybody knows that everyone dies, and no one knows that more than the Doctor..."
veggiet2009 *”The laws of time are Moffat’s and **_They will obey him!”_*
That's pretty much what I think about the coronavirus and the impending doom of climate change behind it.
"phew, the coronovirus is over with....." and now what about the few years we have left to get climate change under control and we aren't doing?
Robert: "Give it a small reward for every frame the pancake isn't on the floor"
Me: *already laughing hysterically*
It could logically just freeze in place if there weren't any other conditions.
some time ago i was playing around with machine learning in unreal engine and i gave it control of a plank and put a ball on it. to make it simpler i just had the ball roll in on axis, so left or right, and the objective was not to drop the ball. it learned the trick really really fast. the best way to not drop the ball, was not to move the plank at all
after that i had the same AI learn how to drive a template car in the game engine and not hit walls. so what's the best way to not hit a wall? that's right! dont move at all! . i felt like an idiot for not seeing that one coming
i maintain that this one was embarrassing. give it a reward for the sequence "side A touching the inside of the pane, in the air, side B touching inside of the pane"
AI would most likely still hack this one too in some hillarious way, but at least it wouldn't be an embarrassingly stupid mistake done by the researcher.
I kinda expected the whole plan to be just shaking in spacsm, because that would make the pancake jump all the time and have infinite airtime for each frame not touching the pan
The hacker heart inside me just loves how AI creatively circumvents the restrictions/goals put in front of it. The boat example just makes me smile everytime!
Is your avatar from To The Moon?
@@NoNameAtAll2 Yup. My user name is inspired by To The Moon, too. It's Latin for "moon bunny".
idk if its circumventing so much as thinking it did a good job
What is interesting is that the AI doesn't even know it is circumventing the rules, because technically, it isn't. It can't tell a bug from a feature. As far as it knows, it is only doing what it was told to do.
@@yondaime500 I would go so far and say that even the term "artificial intelligence" is terribly misleading. AI is just as stupid as computers always were: Simply doing what it's told.
I teach about this in one of my lectures, it's a interesting sub-set of Goodhart's Law "When a measure becomes a target, it ceases to be a good measure". It turns out humans have been dealing with this one for a while. It sort of makes the AI more human.
truly a widely applicable thing. the same happens for "industry standard benchmarks". Once a benchmark program is accepted by the community you can see how the manufacturers fine tune every bit of the hardware to get the highest results, often sacrificing real life results.
@@freefrag1910 like wolkswagen. "Volkswagen had intentionally programmed turbocharged direct injection (TDI) diesel engines to activate their emissions controls only during laboratory emissions testing"
"during the bank's (European investment bank, EIB) annual press conference on 14 January 2016, the bank president, Werner Hoyer, admitted that the €400,000,000 loan might have been used in the creation of an emissions defeat device.[348] "
@@freefrag1910 Then why not make the benchmark "real life results"?
when the arm fliped the lego i just lost it
Code bullet reference... He actually made that agent training thingy public if anybody want to try, though it doesn't have that physics bug anymore.
it's more of a direct mention than a reference
@@knight_lautrec_of_carim reference usually means any kind of mention.
I think it still has the physics jank, he just limited the joints so it couldn't push the legs into its body
@@TheAechBomb iirc he never "fixed" anything to prevent that, he just made it so that the agent died if any part of it besides its legs touched the floor
@@phiefer3 I mean, that is a valid solution IMO. He wanted his creations to walk on their legs, so he made that a requirement for survival. ("I've found that to be a pretty good motivator." ~CB)
I mean, what else was he going to do, tinker with Box2D itself? He has enough trouble using the darn thing, let alone debugging its bugs XD
At 5:02 I absolutely lost it. I would feel personally attacked if my program did that to me. Fantastic!
I'm surprised it didn't also stick its middle end-effector up at the programmer.
It was absolutely hilarious. They left it far too many degrees of freedom.
Really needs the "Thug Life" glasses at that point, or a microphone to drop...
@@Jacob-pu4zj but that is the beauty of finding sometimes brilliant solutions
They sure as hell good in "thinking out of the box". They percieve no bounds for solution but the rules.
The thing is, setting the wrong targets is what happens all the time even without AI.
Humans are amazingly great at setting the wrong target, and then maximizing for a metric instead of actually improving anything.
@@WarrenGarabrandt -- Sounds like every political and economic system past the commons.
@@WarrenGarabrandt This literally explains so much of the failings of bureaucracy
@@WarrenGarabrandt
Climate change wants to know your location
Game achievements.
Chortled at the "YEET"
**chortles heartily**
_chortling continues_
The Midas story doesn't generally end with Midas dying, it ends with him begging Dionysus to take back the blessing. The idea of his daughter getting gilded is also a MUCH MUCH later addition to the myth, it's not in the original at all.
The funny thing is that this made me realize I often reasoned (and executed accordingly) like an AI when I was a kid. I had a knack for zoning in solely on the stated objective of any task with no concern for any underlying purposes, much to the chagrin of any present peers or teachers which I always found confounding.
interesting insight!
This is sort of like F1 racing - the FIA impose some very strict rules and each teams tries to bend each rule so their car is the fastest. It's a totally open system in which there is relativly little cheating, but a lot of rule bending going on. Every year a team comes up with something on the car that is *almost* illegal...
It's an adversarial system as well. Every year the FIA makes new rules, no? "So we notice you started doing this thing. That thing is now banned."
From the list: "CycleGAN algorithm for converting aerial photographs into street maps and back steganographically encoded output information in the intermediary image without it being humanly detectable." That's great, I am not even mad. Second place goes to "Genetic algorithm for image classification evolves timing attack to infer image labels based on hard drive storage location"
I like the one that figured out that lesions are likely to be cancerous if there's a ruler next to them.
Can you explain what the first one means?
@@mrosskne search on Google "This clever AI hid data from its creators to cheat at its appointed task" (with the quotes) there is an article on this, also the paper arXiv:1712.02950
Basically the GAN learned to sneak the answer into an image which was supposed to be (and looked like)a heavily transformed version of the image. By answer I mean the basically the original image!
Here the goal was to create street map from areal photographs. I think CycleGAN basically worked by converting street maps into areal photos and another part of the network did the areal photo to street map. You can think of it as trying to create networks for both the forward and backward problem.
But first part of the network managed to hide what the final answer should look like in the intermediate image! So it managed to basically cram in two photos worth of info into just one and the second part of the network basically read off the answer and outputted the final answer. It wasn't really converting the areal photos to the street maps. It was simply reading off the street map info which was hidden in the high frequency details of the intermediate photo generated by the first piece of the network. To a human eye it looks indistinguishable from noise.
You can search online there are news articles on this.
"I knew everyone would die, I just wasn't sure what would kill us first." I think this sentence has much broader applicability than it might at first seem 🤔
If some sort of doomsday-exterminator AI was cobbled together, you'd have to imagine that our best efforts wouldn't be spent fighting it, but determining what absurd and benign state would reward it most.
Show him some war movies.
If it's smarter that that, show him how toxic we are to ourselves and how adversity unites us. It should promptly select non-interference as the most efficient strategy.
Reminds me of the killbots from futurama.
@@musaran2 solution: obtain a bucket of red paint and a flamethrower. Gain infinite rewards
the AI will just move to far corner of universe to escape us
Reminds me of tom7's nintendo learning program that learned that the best way to not lose points in tetris was to just pause the game
I love the ones that learn to break the game.
Not playing Tetris is the same solution I found to Tetris.
Joshua?
@@AdmiralJota what?
@@mrosskneReference to an old movie. ("WarGames")
There's something about these AI's when they act they way they do that has me rolling on the floor. Like they look so stupid but are actually extremely good at doing what they are asked to do. That pancake throwing technique lmfao imagine if they tried this in real life with an actual robot arm and suddenly the thing tries to set up a new pancake throwing record... I can't hahaha
AI be like: "look at me master!, I'm doing a good job" *proceeds to yeet pancake*
I mean, the problem really is US and not the ai. We are clearly not giving them proper instructions because humans rely on a lot of implied and inferred meaning. Even the most specific of human spoken languages are nowhere near as precise and specific as even a rudimentary programming language. It's such a fascinating thing. I imagine that the process of creating AI has taught us a lot about how humans think.
@@FlesHBoX Strange enough to think that maybe, before we create an superintelligent and powerful AI, we need to create another language and adapt our brains to this.
this must be your first video with Robert Miles? Maybe you are new to AI altogether? I'm just an enthusiast, not an academic, but the crazy things AIs do is the very first thing I learned about the subject, years ago. It's really strange to me to read someone mention this here as if it were at all unusual or unexpected. Yes, all computers, not just AIs, do exactly what you tell them quite literally.
@@squirlmy I learned about the "grow tall and fall over" thing over a decade ago from BBC4 and a few of the video game playing ones, but these other examples were new to me .
As a young kid, I was about to grab a cookie and eat it. My mother saw that and yelled "Don't touch those with your hands!". So opted to simply bend over and eat the cookies directly from the table.
I was such a rebel
"We're going to get rid of all of the software engineers, the users will be able to just tell the computer what they want, and the computer will produce the correct software." (I was hearing this would be reality in 1990 by 2000, the end date keeps getting pushed back. "Oh, you have customers that are able to specify their non-trivial product in sufficient detail and rigor that the machine will produce the desired product. Hunh." The hardest part isn't the design and code, by a competent team. The hardest part is specifying the system behavior to produce the desired end.
There are very good code-generators for a lot of simple tasks, there are also generators for really complex things, even the design. But it always needs quite a lot of human work to be even close to useful.
@@ABaumstumpf Interesting observation. I was in the world of hard real-time, where predictability was a critical system characteristic, so my view-point is probably a bit skewed, even beyond the egotistical ("How could a machine possibly replace someone as brilliant as I?"). User Beta testing was/is not an option.
@@inyobill I only have to deal with very little hard-realtime systems, but most is not hard (but not as lenient as soft either).
The only thing we have that is generated a the api-functions for database-calls, and some base-classes that only hold data.
But in general code generation has come a long way, google and amazon use it extensively, alongside a lot of meta-programming, but it has to be very well structured and needs a lot of specifications.
"customers that are able to specify their non-trivial product in sufficient detail and rigor that the machine will produce the desired product." - well, see, it's for machines so it needs to be unambiguous. As to not reinvent the wheel let's use one of the unambiguous ways to tell computers what they're supposed to do that are already there like a programming language like C++ and wait a minute this software engineering replacement is literally just software engineering
Users don't even KNOW what they want.
Robert this is crazy. Just earlier today I went through my subs you and noticed you haven't uploaded in months. So I went on to google you and found out you had started a podcast, which afaik you never mentioned in any of you past uploads. I wish I had known this, I would have started listen to the podcast back then. Btw the podcast episodes are available only until #52. Anyway super psyched you are back again and to check out the podcast.
Whaaaat? He has a podcast?
Whats the podcast called? I Had no idea! Would have loved to check it out.
Hah! I'm sure I mentioned it somewhere? I don't push it super hard on the channel though, because it's really aimed at researchers rather than the general public. It's often pretty technical. But here's the link if you're interested:
alignment-newsletter.libsyn.com/
@@RobertMilesAI I am not a researcher but I am going to give it a listen. Do you know if the episodes before the #52 are available somewhere ?
@@jonas2560 The podcast started at #52! Before that it was The Alignment Newsletter, an email newsletter with no audio. You can read all of those at rohinshah.com/alignment-newsletter/
4:22 I think this is a perfect example of what AI is great at. Exploring the fringes of a system and finding loopholes and extremely efficient ways of working that humans would never think of.
shameless plug: we built gym environments to study specification gaming where you can play the noodle game of this video as a human and see if you hack games as well as the AI. just google "quantilizers github" and you'll find it.
My favorite is the use of camera perspective to trick the human that was supposed to train the AI.
Instead, the AI basically managed to train the human to validate what the AI wanted.
And the scary bit is that the AI isn't even malicious or anything. It's just doing what it really thinks we want to see.
A classic example of how it is best to assume ignorance before malice.
@@JaneDoe-dg1gv Otherwise known as "Hanlon's Razor". For a given value of "best" you would think we could come up with a better algorithm for calculating likelihood of either behavioural driver, i.e., if it's a voter, weight ignorance, if it's a politician, heavily weight malice/self-interest, and if it's Trump, assume maximum levels of both ignorance AND malice ;)
my main fear finally brought to reality, and this is a dump AI if we ever build a strong one... well we f****d.
@@MouseGoat I was gonna say the same. Imagine this shit with an AI that controls the power grid, military systems, or god knows what. We have a lot of work to do if we really want to put this things to work...
One of my favorite examples is some machine learning made for beating original Mario. It was told live as long as you could. This lead to it pausing whenever it was about to die and refusing to unpause
You know, a good chunk on the list, I can't tell if the AI actually did anything wrong either. There's definitely something to be said for expectations as well as everything else.
"Bicycle AI learns to circle around goal" Isn't that good if it's improving stability? Oh, I guess it never went 'towards' the goal at all
Boat Race robot, people definitely value playing some games like this, so if it looks bad isn't it just because the material was too simple?
Stenography, that's legitimately just a useful system that doesn't take any extra data or overhead to keep track of maps, when used in an existent system, good job robot!
That yeet part had me on the floor. xD
I recently made a tic-tac-toe AI for a school project. Nothing fancy just a recursive game engine. Turns out that because of how I had the weights, it wouldn't take the win as soon as possible, but would instead keep playing as long as it had a way to win because it figured more possible futures where I can win > a single future where I win
Robert: Everyone will die
Me: Great. Love you, Robert! Keep up the good work.
6:56 last I argued something like that could happen, I was faced with the reply "that would require perfect knowledge". I stopped arguing. Now this list might help me argue in such situations.
3:32 an accurate representation of my journey through life thus far
Missed you man, love your research stories.
Correction: it is not everything that collides with him that turns into gold. It specifically states _everything he touches._ So he can still breath, but any air that touches his fingers (there is a reason it is called a midas finger, or green thumb) will continuously turn to gold (no idea if it welds with the prior gold that just fell, or if it becomes atomic gold dust oozing out of his fingers). And he can still eat, the implements used to do so will however become gilded. Similarly, if he scratches himself, he turns into gold.
edit: also, the ground most certainly is not a single object. *Maybe* I can agree to it if you refer to a single casted slab of concrete used as ground, but even regular stone is filled with weird stuff I dont know about cause I am not a geologist, and earth has clumps of dirt aplenty.
Then can't he carry gold claws with which he can hold things?
_"in real life you can't just be very tall for free"_
Guys lying on their tinder bios: *Says who‽*
liked for the interrobang usage
So what is the Unicode for the interrobang? 'Cause I want to able to use it too!
"The robot just throws the pancake as high as it can"
LMFAO
Wait, what exactly IS touch? Depending on how we answer that, only an atom-thin layer of material turns to gold, or the entire universe turns to gold. It can't be the former, so we can assume that the true function of the ability is more like a flood-fill, where whatever he touches turns to gold, and whatever touches that also turns to gold, and so on. So yeah, the whole planet including its atmosphere should become gold, and probably the solar system if we count all the photons around and... yeah entire universe becomes solid gold at the speed of light.
Should've wished for "whatever I touch turns to gold only in compliance with my expectations and desires". Maybe that's safe...?
What happens when your desires are contradictory? As happens very often with humans due to us having multiple goals.
For more along this train of thought, I recommend the book Superintelligence by Nick Bostrom
I rofled at the original audio reconstruction!x)
This was informative and also by far the funniest thing I've seen today. Thanks Rob.
"Artifical Intelligence beats human stupidity by far!"
You know, when I heard it, was was meant the other way around (AI still being pretty stupid).
Normal person: (sees AI making dumb mistakes) "Haha, dumb computer. What is everyone so afraid of? This thing could never take over or destroy the world."
Safe AI enthusiast: (sees the same thing) "Oh no, this thing will take over the world and destroy it if we actually let it do serious thing. We should be really careful with it."
exactly!
especially because these kinds of outcomes are already potentially affecting people's lives where software to help determine criminal sentences are involved?
which is some dystopic sci-fi shit btw, but at least that would have the courtroom painted black with neon lights, in the real world they look exactly the same but are shifting their operation
@@kaitlyn__L I feel like sci-fi writers did us all a big disservice with their depiction of rogue AI. Gave us as a society a major blind spot for technology going rogue while still being very dumb.
I'm not an expert, but I believe in the midas scenario 'touch' is interpreted as 'midas is thinking about the fact that he is touching this thing, so it turns to gold' or at least, I've read versions where he is initially very happy and touches various objects, and his clothes don't turn to gold until he realized he is 'touching' them, and his daughter does not turn to gold until he realizes he is touching her. since midas never thinks about that fact that he is touching the ground or air, it never turns to gold.
Pretty similar to human response to tax incentives. Very hard to specify the correct reward!
I knew someone applying for school aid for his daughter; he discovered that two students in the family qualified for more aid for one, so he signed up for community college courses himself. Before there was artificial intelligence, there was natural intelligence.
This is EXACTLY what I first thought too. Whenever there is any government incentive, rebate or aid, it very quickly gets exploited.
Absolutely, but I don't think the father signing up for college courses is necessarily a bad thing either.
I remember hearing about a robotic arm that was supposed to grab an object with a claw and instead of opening it's claw (I think it was stuck or something) it applied pressure to the object it was supposed to pick up until the claw opened enough to snap around it. I can't remember all the details but I'm pretty sure it wasn't supposed to do it that way.
So "monkey's paw" is the very nature of how AI behaves?
Why does this cause me to feel profound anxiety?
Because you're paying attention.
we act like AI as well. You just don't realize it. But here is one example to start with: Sex was invented for reproduction. We humans invented condoms to exploit sex for pleasure instead of reproduction. You see this pattern in literally everything we do. It just isn't as obvious as with the AI that you see here, because you're so used to thinking that whatever you are doing is "more intelligent"
This is the first video of yours I’ve ever seen. In the first two minutes you meander off deeply into frankly overwrought ramifications of a mythical story, and then literally interrupt yourself for a scientific analysis of your diversion...
I subbed immediately! Where have you been all my life?! It’s like you made this for me specifically.
Joke:
Genie: "You get one wish"
Midas: "I wish anything or anyone I touch turns to gold"
Genie: "Ha, fool, you do not realize -- wait, what?"
Midas: "I know what I said"
I might be dumb, but I don't get the joke?
Poor Midas can't even touch his own pp.
How nice of you to warn us first. "Joke:"
@@Ghi102 Genie in a bottle. Rubbing counts as touching.
Ghi102 the joke is that in this variant, Midas *did* want to turn the people he touched to gold, and the genie was surprised.
It goes against expectations, and also might lead to the reader missing the “or person” in the first line, and then realizing it when the genie does.
Produces a kind of double take?
Fantastic intro.
The whole video is hilarious. YEET the pancake!
COVID under-reporting is arguably a more realistic version of the bucket robot's specification gaming...
Or over-reporting, if hospitals were to get subsidized
@@revimfadli4666 I have been told that Alameda county's claim that it had been over-reporting Covid is actually an attempt to cover up mismanagement at nursing homes that lead to cases.
And it seems likely to me because the claimed amount of over-reporting is impossibly high.
I was an undergrad at UCSD during the early back-prop days with Bart Kosko and Robert Hecht-Nielsen, and developed an abiding fascination with following ML developments, though only as a hobby, never professionally. Robert Miles is, to me, one of the best at identifying and explaining both the fundamentals and some of the "curious corners" of current advancements.
However, one YT video every 4 months is not nearly enough. I can haz moar? Puhleez?
Nice haircut! Truly a good one in this era of online tonsorial self-mutilation.
with every video, you're becoming more and more my hero and it's kind of the best thing
To solve the specification gaming problem, we only need to create AGI that can interpret what we meant, instead of what we said.
Or we can create another language and another way to think, more specific, before creating AGI
@@plcflame Like Lojban?
Ah, the old DWIM problem.
7:36 okay that's an actually scary example.
Literally it takes a lot for me as a person to want to like and subscribe in general, especially when watching a new channel, but I am two minutes in and I wanted to like, subscribe and even comment. I love the humor and excited to watch the rest of your videos!
Well I've just been randomly recommended this video, first time seeing the channel and something really interesting!
What if King Midas just wore gloves all the time, the gloves would turn to gold, but then he would be touching everything with the gold gloves and not his skin
Or using a spoon/fork to handle food (unless touching his tongue counts...)
how would get get the gloves on? the moment he touched them they would turn to solid gold
@@Sharklops well, then maybe a servant could cut the fingers off the gloves, put sausages or something inside the fingers so they have the proper shape, then King Midas would touch the cut off fingers turning them to gold and locking them into their shape, the servant would then take out the sausages and King Midas could put the gold gloves fingers on his own fingers, thus creating a layer of protection to touch things
You just described the plot of "Frozen"
@@Sharklops The gloves could perhaps be thin enough so that even as gold they were flexible enough
"oxygen molecules will turn to gold atoms, [...] and I guess the ground is just one solid object"
kay then
These were so fun & interesting to watch! Please do a second episode with more examples!!
Welcome back! We missed you!
Mentions codebullet.
Ah I see, you are a man of culture as well.
The Midasocalypse! That would be METAL. Literally 😂
That's heavy, doc.
I always love when you're featured on Computerphile. I don't know why I didn't think to check if you had your own channel.
Absolutely subscribed.
Flipping the cube over so that top and bottom planes match. That one is pure genius. Well played AI, well played.
Thing is, in some applications you actually want the AI to come up with these 'outside the box thinking' kinda solutions (e.g. in pharmaceutical resarch)
I played Strategy Challenges of the World in 1995! Thank You Miles!
Found 'em!
Reminds me of that time when they wanted to end a snake infestation and so they offered a reward for every dead snake; so people started farming snakes; and when the government found that out they canceled the reward program, and so all the farmed snakes were released, making the snake infestation even worse than what it was before.
The Cobra Effect :D
Thank you so much for sharing that list. Reminded that one of my absolute favorite things about AI is the subtle, surprising, absurdist humor that often comes with it.
Finally another video! Keep up the great work!
7:30 me removing my tasks from the Scrum board
Yossarian moved the bomb line.
What I'm getting from this is that we should use machine learning to discover loopholes in real physics.
This has actually happened at least once. Google "an evolved circuit, intrinsic in silicon, entwined with physics".
The issue is that "magic" solutions like this tend to be surprisingly useless, because they often depend on contingencies in the training environment, like temperature or ambient radio signals, that can't be relied on in practice.
Glad you are back!
Very fun, entertaining & informative! I'd love another list like this!
Code Bullet actually made that engine with learning agent running away from laser kinda public... Though mentioned bug with physics engine was fixed of course...
YEEEAAATTT 8:32
I could watch these examples 24/7
I really think that the pancake throwing thing is amazing.
I just found your channel after having seen quite a few of your computerfile videos on the AI safety subject.
And I find it a very interesting topic although I still only know what you teach in these few videos.
Yet this rejoins phylosophical (and mathematical) concepts about how to define things.
How do we define properly what we mean, all the subtle implicit and ambiguous part of the language and what do words mean
I'll sure be watching more
This video is hilarious and your hair looks great
God, this is just comedy gold. All of these should be rearrenged with even more of a stand up comedy feel.
An AI algorith comes into my cooking class.
He tries and fails to flip a pancake.
I just tell him: for crying out loud, please just try and avoid dropping it on the floor as long as possible.
He then proceeds to fling it at the ceeling. It got stuck there.
Then he turns to me with a look of glee in his eyes: I am good AI. Task successful.
Edit: Fml, he got really angry when I tried to scrape it off and tried to kill us all.
I used to play Strategy Challenges of the World in 5th grade! I didn't recognize the name or the clip, but looking it up, I recognized the collection of games!
I've occasionally kinda wondered what it was called, so thanks!
The introduction of imagining Midas’ gift was cute. Several errors though:
IF the curse was anything touching his flesh was to turn to gold, assuming only those direct atoms or electron clouds of a certain limit, Midas would instantly be surrounded by a reactive chemical cloud of torn apart molecules, probably smelling like ozone, reddish black in color.
Ultra fine black gold dust constantly comes off him, coating him instantly in thick layers of dust.
Midas just asphyxiates.
IF some level of human story telling is allowed, the curse applies to his body and mouth only. Only 'objects' turn to gold. Easy fix then: hook Midas up to a feeding tube or IV as needed, build articulated clothing he can wear, problem solved.
Moreover, if the entire planets some are replaced with gold at original density, it would violently shrink almost a thousand kilometers, and become incredibly hot due to gravitational contraction. Volcanoes of liquid gold would erupt into a landscape with mountains no higher than 200 meters.
And things would still look black, due to the dust!
"I knew everyone would die. I just wasn't sure what would kill us first."
Hasn't that always been the case?
All this cheating by the ai sounds pretty human to me! :p
It's always a wonderful day when a new Rob Miles video comes out
I was so fascinated by [the contents of] this video that, at the end, I snapped out of a trance. That rarely happens. Good job!
Is the outro song from the Mikado?
Yep. "I've got a little list"
Just for fun, how about situations where a wish/AI would go horribly unwrong?
For example: Someone makes a system that has instructions to maximize world "badness" or some such. Then the system reasons
1. "Badness" = ("badness" in the end ) - (how good thigs have been)
2. The bleakest (or "baddest") world state is the heat death of universe and that eventuality cannot be avoided.
Thus it creates a prosperous, long lived utopia so the eventual tragedy of all that being lost is the greatest.
Interesting perspective on this problem.
I have never experienced such brilliance of thought from anyone! Or at least not in a form I could perceive, I hope you are a real and kind person. I need to believe that real and kind people are this brilliant.
Just found your content and it's honestly great and I'd love to see more from you but some of the production can be distracting. If you'd like any advice on the camera side of things like focus or lighting I'd be happy to help!
Again thanks for the content and research that went into to everything.
I can't wait to see what you are working on in 20 years... your seamless weaving of philosophy and technology is second to none!
No, that's not how the story of King Midas ends: He gets the wish reversed but he from then on has to live with the ears of a donkey¹. He keeps this secret, (literally under his hat), from everyone but the man who does his hair, who is sworn to secrecy. When the barber can take keeping in the secret no more, he digs a hole and yells "the king has the ears of 'a donkey¹'" into it before filling it back in. Of course, given the kind of story this is, a cluster of reeds promptly grow on the spot and whenever the wind blows they repeat the barber's telling of the secret.
1: Not the usual word, but I doubt TH-cam's bots can tell the difference between the animal and the body part.
It probably ends both ways, because it's mythology
Hey Rob, we met at Vidcon and talked about media polarisation - how’s it going? :)
Dude I had no idea you had your own TH-cam channel
So glad man 👍
Thanks for one video each year.
My patience ran out