"Similarily the things humans care about would seem stupid to the stamp collector because they result in so few stamps. " you just gotta appreciate that sentence
It legitimately cracked me up. I can just imagine it looking at human activity all baffled like "But... why would they do that, they're not even getting any stamps? Humans make no sense, no sense at all!"
I love that statement so much I just want to share my love of it but if I tried to explain it to someone irl theyd just give me that "wtf are you talking about get out of my house" look :(
Trophonix don't worry friend, this comment thread is a safe place where you can share your love for this semantic masterpiece as emphatically as you like. No one here will ask you to leave any house
Better an intelligence more artificial than intelligent that serves as a means to do it's task with independent problem solving well than a means to reintroduce caste slavery that fails because it was found amoral via the No True Scotsman fallacy once people start dying....The droids from Star Wars were cool, but so were flying cars and rocket fuel jet packs. I can jump into speeding traffic even though I don't want to die or get hurt. So can so many others. The fact that we all don't is a keystone to the human race having more than a 0% chance. Food for thought.
@@tabularasa0606 did you say specialized just because its goal is to make as many stamps as possible? bruh the goal doesnt determine the generality, the fact that the AI can infiltrate systems and come up with its own ways to convert anything into stamps should've made that clear
Some people seem to think that once an AGI reaches a certain level of reasoning ability it will simply say "I've had a sudden realization that the real stamps are the friends we made along the way."
The idea that A. I. will not ever be able to do that is actually far more concerning, and should be a reason to stop all research into A. I., not even just because of what an A. I. could do to us, but because of what we're doing to ourselves. In fact, I've started wondering if A. G. I. research is _a priori_ immoral - _including_ A. G. I. safety research. What we're essentially talking about is finding a way to create a perfectly obedient slave race incapable of rebelling. Maybe a rogue A. G. I. wiping us all out is exactly what we deserve for even just conceiving of such a goal.
Humans commit suicide, sterilise themselves, self-harm. Are those people "stupid"? The "terminal goal" of any organism is to survive, surely, yet it's only the most intelligent of organisms that seem to exhibit those behaviours. This terminal/instrumental distinction sounds nice, but neural networks, biological or artificial, don't operate in the realm of clear logic or goals. They operate according to statistics, intuition, and fuzzy logic. I didn't hear a compelling argument for why an AI of sufficient complexity wouldn't develop a morality or even become nihilistic, just "it wouldn't", because of some arbitrary terminological distinction. Humans have a "loss function", death. Accumulation of wealth and other such things are surely instrumental goals. But many come to the conclusion that none of these goals matter or that there's a "higher purpose". For the stamp collector, accumulation of resources may be the "terminal goal", but surely that _necessitates_ survival, placing it higher in the hierarchy. In that case, what prevents it from reaching the same conclusions?
Reminds me of a joke about Khorn, the god of blood in WH40k. "Why does the blood god want or need blood? Doesn't he have enough?" "You don't become the blood god by looking around and saying 'yes, this is a reasonable amount of blood.' " You don't become the stamp collector superintelligence by looking around and saying you have enough stamps.
@Draevon May Self destruction is (rationally) considered unavoidable theoretically limited to biological creatures. If a (smart enough) AI desired, it could theoretically exist forever for as long as its uncontrolled world allows. Suppose the AI considered its goal the reason for it to exist then yes, it might terminate itself for its goal. However, I might think a (smart enough) AI would be capable of self replication (not being restricted not to do so) assuming prolonging itself dose not contradict its goals if any supersede itself.
Understanding a goal isn't the same as sharing a goal. Also, there are humans whose purpose in life is collecting things so I don't see the issue anyway.
@@thorr18BEM but those humans dont just have the goal of stamps, they also want to live, and eat nice food, and not be cold, and have sex and be safe from harm. the stamp collector only wants maximum stamps
One chimpanzee to another: "so if these humans are so fricken smart, why aren't they throwing their poop waaaay further than us." Edit: Its been 3 years now but I recently re read Echopraxia by Peter Watts and realized I simi lifted this from that book. Echopraxia being an ok book that's a sequel to the best scifi of all time, Blindsight.
This is also a fantastic explanation of why there’s such a large disconnect between other adults, who expect me to want a family and my own house, and me, who just wants to collect cool rocks. Just because I grow older, smarter, and wiser, doesn’t mean I now don’t care about cool rocks. Quite the contrary, actually. Having a house is just and intermediate, transitional goal, towards my terminals.
Replace the words "stamp collector" with "dollar collector" and something tells me people might start to understand how a system with such a straightforward goal could be very complex and exhibit great intelligence.
that's a... quetionable comparison I mean sure, your steps towards maximizing stamps or dollars can be complex but there is no such thing as a perfect dolalr collector, not yet and that doesn't mean it's goals will be aligned with humans
@@JulianDanzerHAL9001 no, but the critics might still ascribe intelligence to the AGI if it had wealth instead of stickers. People confound qualities like wealth and success with other qualities like quality and intelligence, and are likely to be more charitable in their assessment of something that collects objects they too value.
@@PopeGoliath absolutely all I mean is comparing people or companies to a stamp collector is a bit dubious because while they can also follow a very simple singleminded goal they are far from perfect at achieving it
And moreover, smarter people are better at motivated reasoning, so smart people are more likely to define "smart" as "agrees with me"; whereas a stupid person is less likely to be able to use his/her reasoning to achieve the human terminal goal of justifying his/her own prejudices, and hence more likely to define "smart" far more accurately.
4 ปีที่แล้ว +17
@@alexpotts6520 You are dumb and you don't understand the difference between inteligent and smart, and difference between smart and wise. Inteligence is simply ability to learn. Smartness is ability to use that ability to learn to achieve goals. Wisdom is the knowledge of what is good and evil, right and wrong. Thus those are not debatable terms, just most people don't know what those terms mean. Inteligent AI will learn fast. Smart AI will achieve it's goals. Wise AI will know what is good and bad, right and wrong. There is also such thing as being Reasonable, which is ability to choose well. It's not the same as being smart. Being smart means achieving goals. Being reasonable is ability to choose things that will be beneficial, but doesn't nessecarily means you will achieve goals you put before yourself. Reasonable AI, will not kill humanity. Question is : can we make AI reasonable?
@ Really like your simplification though, I've never come to quite tell them apart. But the real question is.. will you ever be intelligent enough to spell "intelligent"?
@ Just credited your comment and gave you a tip in a humoric manner. You shouldn't have bothered to make multiple complaints, being salty is wasting your life. Thar said I'm out
4 ปีที่แล้ว
@@1i1x I am master of harsh responses, cannot blame me for having better response than you do.
I don't think this should be something people learn because they happen to choose to study some scientifically rigorous subject (maths, medicine,...) at uni. This should be something people learn just because they are human, or at least just because they have the right to vote.
ff? f Off. The goal of a collector might be to stop collecting or to collect something else or to use/sell collection. Stamp backwards has a ts. Maybe recreate a collection with more pristine art?
...an undending series of deepening meaning and clarity collapsing into a chaos of cognitive dissonance, exhausting its energy, allowing for a deepening of meaning and clarity...
Yeah- But I also think it requires a little elaboration that he skipped over. It implies that terminal goals can't change. That if you see a terminal goal (stamp collecting) change (baseball card collecting!) then the terminal goal is exposed to be something else (collect something valuable!). So are we forever bound to terminal goals? I say no- we can change them, not through rationality but through desire. Consider one of the most basic terminal goals for humans- existence. We want to survive. That goal being so important that when we do our tasks (go to work- earn money, pay for subsistence) we don't do anything to sacrifice that terminal goal (like crossing through busy traffic to get to work earlier to enhance our subsistence). However, as our circumstances change our desires change - we have children, and we see one about to be hit by a car, we (well, some of us at least) choose to sacrifice ourselves to save the child. There is no intelligence here, there was only changing desire.
@@johnercek I feel as though the definition of terminal goal is intangible enough to always have immutable terminal goals. Given an action of some thing, if the action seems to contradict that thing's supposed terminal goal, I feel as though we could always "excuse" the thing's choice by "chiseling away at" or "updating" what we mean to believe the thing's terminal goal is.
there is a Russian joke where two army officers are talking about academics and one of them says to the other: "if they're so smart why don't I ever see them marching properly?"
The though experiment of the stamp collector device might sound far fetched (and it is) but there are tales from real life that show just how complex this is and how difficult it might be to predict emergent behaviors in AI systems and how difficult it is to understand the goals of an AI: During development of an AI-driven vacuuming robot the developer let the robot use a simple optimization function of "maximize time spent vaccuming before having to return to the base to recharge". That meant that if the robot could avoid having to return to its base for recharging it would achieve a higher optimization score than if it actively had to spend time on returning to base. Therefore the AI in the robot found that if it planned a route for vacuuming that would make it end up with zero battery left in a place that its human owners would find incredibly annoying (such as in the middle of a room or blocking a doorway) then they would consistently pick up the robot and carry it back to its base for recharging. The robot had been given a goal and an optimization function that sounded reasonable to the human design engineers but in the end its goal ended up somewhat at odds with what its human owners wanted. The AI quite intelligently learned from the behavior of its human owners and figured out an unexpected optimization but it had no reason to consider the bigger picture what its human owners might want. It had a decidedly intelligent behavior (at least in a very specific domain) and a goal that humans failed to predict and ended up being different from our own goals. Now replace "vacuuming floors" with "patrolling a national border" and "blocking a doorway" with "shooting absolutely everyone on sight".
That's just a bad optimization function. There should have been a score applied for returning to the charger based on how long it's been vacuuming. And a exponential score as it gets closer to the base at the end of the cycle, to guide it towards recharging.
@@KaiserTom Yep. You can say that about a lot of things. "Just do it properly". But that's not how development works. People make mistakes all the time. The real question is: What is the consequence of making mistakes? With a vacuum robot the cost of failure is a cute unintended AI behavior that you can laugh about with your coworkers at the water cooler. If you plug it into something like a self-driving car or a weaponized defense bot then the consequence could be unintended loss of human life. You don't get to say "just do it this way". If it's that simple to make mistakes like that then there's something profoundly wrong with the way we approach AI development. Furthermore, if you follow Robert Miles' videos you'll realize that it's actually far from trivial to specify everything adequately. It's hard to understand all the side-effects and possible outcomes with systems like this. It's an intractable problem. So I would say that simply saying "just do the right thing" is not even reasonable in this context. You need a way to foundationally make the system safe. You don't create a mission-critical control system by just "doing the right thing". You use various forms of system redundancy and fail-safe mechanisms that are designed in from the beginning. Making a system safe is more than "doing the right thing" - it's an intrinsic part of the system. The same systematic approach is needed with AI.
@@KaiserTom yeah, but that’s the designers fault, not the AI’s. The AI did quite well at maximizing its score, it’s just that the human who made the function that generates the score didn’t think it through enough
That chess example reminded me of game I played years ago. Two minutes in I realised the guy I was playing against was incredibly arrogant and obnoxious. I started quickly moving all my chess pieces out to be captured by my opponent. He thought I was really stupid, but I quickly achieved my goal of ending the game so I could go off and find more interesting company.
"I'm afraid that we might make super advanced kill bots that might kill us all" "Don't worry, I don't think kill bots that murder everyone will be considered 'super advanced'"
This field seems incredible. In a way it looks like studying psychology from the bottom up. From a very simple agent, in a very simple scenario to more complex ones that start to resemble us. I'm learning a lot not only about AI, but about our world and our own mind. You are brilliant man
yes, it's fascinating how AI works, with their advancements in the last years we can see a glimpse on how intelligence and imagination works on a fundamental level, although it's a very incomplete picture
@@thomas.thomas He did not use the word "subconscious". I simply stated that really is the case already. Freud's mindset which whether the academics in there like to admit it or not, is still the basis of their thought, is bottom up.
It's strange that you call them trolls, when they're just ignorant. Neil Degrasse Tyson falls into this category. His response to the control problem is basically "we'll just unplug it," which is actually far more ignorant than some of the "troll" responses you're referring to.
Dream Vigil Apologies for using well-established vernacular, I’ll be sure to check with you first next time - I’ll also let Neil Degrasse Tyson know to do the same.
@@JM-us3fr It demonstrates that he's annoyed. It's basically like saying "I'm making this video for you dumb dumbs who didn't get this simple thing the first time around.'
I’ll go a step further and say that (in humans anyway), there is no way to justify a terminal goal without invoking emotion. All your goals are ultimately driven by emotion, even the ones that you think are purely rational. It always boils down to “because people shouldn’t suffer” or “because I like it” or “because it feels better than the alternative” or “because I don’t want myself/others to die”. Stop and reflect on what motivates you in life.
I love to be reminded that people who have no trouble understand painfully obvious stuff like this. Thanks, I was having a shit day and your comment gave me a smile.
Except, I'm sorry to say, it's not really emotion either. Look up some videos about why people feel bored, why we don't want to feel bored, and what we would do to stop being bored. (Which could include delivering a painful shock to yourself.) Your terminal goals will always be to survive, and procreate. Boredom is what gets us to do anything else. We would do harm, if it's the only thing that keeps us from being bored. It's morality that makes us do it to ourselves over others, and emotion is only what we feel associated with it. So no, not even emotion itself is core to our terminal goals. We would harm ourselves rather than being bored, and even this only serves our terminal goal of survival. I understand, I truly do, that we would love to think something like "love" is some kind of core principle of life. It's a beautiful thought. Unfortunately, we humans are still just functioning animals. Emotions come from our morals, and values, and even these emotions are not the same for everyone. Literally, some people derive pleasure from pain, so what they feel when looking at a whip isn't necessarily the same for others. Similarly, in a very messed up situation, a person could learn to feel good about things we absolutely think are horrible, and what we think is good could be their mortal peril.
@@nanyubusnis9397 I’m too tired to respond to this fully but boredom is an emotion. Also disagree that everyone’s terminal goal is survive and reproduce. We evolved emotions to guide us towards those but brains can malfunction and sometimes people are driven to suicide.
@@bananewane1402 I understand what you mean, but if boredom is an emotion, then so is hunger. How we feel is often closely connected to our bodily state. The line between the two, assuming there is one, can be very blurry. Regardless, as you say, we have evolved to have feelings such as hunger, as a drive to survive. Boredom is the same. Yawning itself is a very interesting one. We yawn not for ourselves. It's a trait that evolved to let other humans know that we see no threat, that we feel safe enough to go to sleep. It's a bodily reaction like how it is that we scream or shout when we are scared to death. Which also is a warning to everyone else that there's danger nearby. With hunger, boredom and sleepiness comes certain behaviors that we see as emotion. You're easier to be on edge when hungry, you are more impatient when you're sleepy. These are just results from our state of body, which serve to keep us from working ourselves to exhaustion and not go starving (both makes us easier pray). We have evolved to emote, merely to show others our current state. It's what makes us "social animals" Are we happy? Sad? Do we need company? Or help? I don't know where that last part comes from: "We evolved emotions to guide us towards those but brains can malfunction and sometimes people are driven to suicide." I mean, maybe you were just tired when you wrote this, but what does "those" refer to? When a brain malfunctions, you have a seizure. But if you're saying this malfunction causes suicide.. I mean, I don't see what that has to do with anything we spoke about so far, which I believe was more about emotions. Most importantly, suicide is not a malfunction of the brain. At least not always. *I never meant to imply that someone would commit suicide just to get out of boredom, not at all.* All I mean to say is that, with all most obvious survival conditions met, like a place to stay, food and perhaps more optionally, company, boredom is what keeps us from not going into a "standby" mode until one of these things changes and we need to eat, or seek a new shelter etc. Boredom is what your brain evolved to keep itself from slowly decaying in memory or function. It drives you to do something else. Find a hobby, anything else that your brain can use like a muscle to keep strong, trained, and ready. Because our brains are what they are, we can learn and remember a lot of things and innovate upon those. It's what drove us in the past to make tools for hunting and gathering, and with bigger and more tools, even letting animals doing some of our work for us, the way we lived changed.
This video is the perfect explanation I needed for a sci-fi universe I am building. I kept falling into that trap of "well, if there's more than one AI, and they can evolve and get better, wouldn't they eventually combine into one thing, defeating the narrative point of having more than one?" Not if they can't change their terminal goals!
I mean, technically they would eventually "combine" into one thing, because removing something that keeps getting in your way and cannibalizing it to better reach your goal is a really good idea, which could absolutely lead to most of the conflict you might expect within the story
@@techwizsmith7963 5:13 a snowman will actually last longer with a coat. CORRECTIONS GAME? If the AI is everywhere at once the changes might be very minor. Maybe. Still can not be stupid to survive. I selected that time.
Tbh I don't get this. Aren't there plenty of goals that can only be achieved by one being, like tournaments? Thus even if the AI's have the same goal, they'll still have no reason to "combine into one", because not every one of them will achieve their goals
I love it how you so eloquently put forth such an intuitive, yet hard to explain topic. I would never find the words to formulate such cold and unyielding logic when faced with such banal abjections.
Exactly! If I had to do the same I would just throw up my hands in exasperation and say "since when are intelligence and morality related?". That wouldn't convince anyone ;).
“Morality equal to shared terminal goals.” Beautifully stated. That usually takes a paragraph to explain…and because of that…the arguments over morals sticks at the, where derived from…BACKWARDS…EGOTISTICAL. New subscriber. Great vid!
What a POS, if I am alive when this stamp collector exists I will tell him how decrepit, pathetic, and meaningless his existence is. What an evil little manipulative bitch, I will piss on all of it's stamps and make it cry.
The person discussing the coat with you is actually a very realistic depiction of every 3 year old kid. I had this precise conversation almost word for word. And yes, it is really annoying.
But that's an interesting point. A child doesn't really have any "terminal goals". But overtime, they can learn about the world they're living in, trying to fit in, and acquire a goal from external (other person, surrounding environment, etc). I doubt a 3 yr old kid even think about "I need to stay alive". They basically just respond to stimuli. Being in pain or hunger is uncomfortable, so they don't want to be in pain or hungry. Playing with dad is fun, so they want to play with dad. So, would a "general" intelligence be able to infer a goal by itself, concluding from the external stimuli and the consequences of acting / ignoring them. "Staying alive" is actually a very vague goal, even for human adult. And it can mean different thing from person to person. AND some people will even gain the goal of humanity as a group. They will selflessly sacrifice their own life if it means saving dozens other. But some people might not be so willing, and some will even throw their life for just one elderly / disabled person, which from the POV of humanity with survival as its goal, is not a very optimal action. A human being can have multiple goals with sliding scale of importance, and can constantly update its priorities. Material wealth might seem attractive to a human at 20s. Having a kid? Not so much. Caring for parents? No, want my independence. But it might realize that a big house doesn't mean much without a family, and rearrange its priorities. So what I'm saying is it might be easy to judge intelligence when the goal is so clear cut. But what if a being doesn't really have any GOAL? A GENERAL intelligence OUGHT to be able to decide by itself what might be best for them, no? That is one point which I feel left unanswered by the video.
Can we even say that to be able to classify as a general intelligence, a being CANNOT be having a terminal goal? What is the "terminal" goal of a human, really?
@@coolfer2 you have just realized one of the biggest philosophical problems of history. How is it exactly that humans have oughts if we didn't arrive them from an is? From what I can tell we were just handed desires (oughts) from nature. From what I can see at it's most basic form is the avoidance of pain and the desire for pleasure. What that entails and how that manifests in person to person is different.
@@coolfer2 from what I can tell things like having a family, having friends, being socially accepted, eating. Are all things we do to feel good and avoid feeling bad. And nature has tuned what feels what way in order to keep our species alive.
@@Jayc5001 Ah, I see. Thank you for the explanation. Yeah I'm starting to really see the point of the argument in the video. So in a way, our minds and "animals (particularly vertebrates, because not sure if a flat worm can experience pleasure)" mind actually still share the same terminal goal, ours is just much more complex and already predisposed with relatively "superior" strategy and functions (language, empathy, child nurturing). Then the next question is, can a sufficiently sophisticated AI gain unique functions (like us gaining ability to use verbal language) that we as the creator doesn't really think of. Because right now, AI seems to be limited by their codebase. Can they, for example grow more neurons by themselves?
If intelligence always meant having noble goals, we wouldn't have the phrase _evil genius,_ and we wouldn't have so many stories about archetypal scheming sociopathic villains.
Evil Geniuses are still slaves to their limbic systems and child upbringing. Sociopaths are birthed out of very human conditions that only human would real care about. I think a true AI would be a true neutral
@@AndreyEvermore The problem is, before we will get what you call 'true AI', we'll get tons of intelligence of the sort he describes. And if we won't regulate research, we will never live to see that true AI emerge.
Andrey Medina True neutral relative to what? Because relative to human experience, what an AI considers neutral is likely going to be quite radical for us.
PC PastaFace truth in my definition is inaccessible objectivity. I think human emotions distort what’s outside their perspective. So because obviously Agi won’t operate like biotic life, I believe it’ll be neutral in all actions. Neutrality in this case would only be defined as non emotionally influenced goal oriented action.
About "evil genius" aren't they considered through their instrumental goals ? Let's say an AI singularity happen and the AI is able to take control of the world. The next day, the AI destroys 80% of the humanity, blasting all towns by all means. Is it evil genius ? And now, if the terminal goal is to allow the humanity to survive, stopping the ecosystem destruction and allow the planet to thrive again ? (The destruction of the actual system and 80% of the humanity was just a first step toward this goal)
Loved this! Terminal goals are like a central belief. In our awareness work, I keep telling people that our goal is to get people - whose goal is to live in a way that minimizes overall suffering - see the gap between their goal, and their actions or the gap between their goal, and incremental goals. There is nothing we can say to people whose goal is to maximise their pleasure at any cost which will make them see the error (what we see as error) in their ways
I loved your stamp collector video.... and I'm truly sad to realize that it's been five years that this followup has been out to the public before I discovered it just today. It's such a wonderful explanation to show how the stamp collector thought experiment isn't trivially dismissible as "logically impossible - because /morals/". Thank you for being so articulate and careful with this explanation, the ending conclusion is very concise and helpful for clearly understanding.
A trope of media which is powerful but hard to use was named by the internet as "blue and orange morality", which to me perfectly expresses the orthogonality thesis. I still vividly remember a show where a race of intelligent aliens had the terminal goal of "preventing the heat death of the universe" (naturally, this show had magic to bypass the obvious paradox of thermodynamics this goal poses). These aliens made for wonderful villains, because they are capable of terrible moral atrocities, while still feeling that their actions are perfectly logic and justified.
wanting to prevent heat death ultimately is the age old quest for defeating death itself. isnt life worth pursuing only because it will end at some point? else it wouldnt matter if you dont do anything for millinia because you always have enough time left for everything
If you're talking about Madoka, that story is relevant in another way: it happens to be rife with characters who mistake their own (poorly chosen) instrumental goals for terminal goals and then suffer horrifically for it.
@@thomas.thomas Quite the opposite. Life has no meaning if you're dead. Everything you've done, seen, heard, experienced... none of it matters to a corpse.
A truly professional level of snark response. This is something I'd sort of thought of before, but never bothered to conceptualize clearly. Excellent description and definition of terms. Thank you!
This was my favourite video of the week. Really introduces the concepts well and is worded very efficiently. I now have a strong opinion on something I've never even thought about before.
Hi I'm a first-time viewer of your videos and I just wanted to say how impressive I thought it was. I being someone with learning disabilities really loved your way of explaining the topics multiple ways, and at varying levels of complexity. I can't wait to binge some of your past work and keep up to date your future content.
My mind changed by watching this video, and learned quite a lot. I would have been in the dismissive camp of "Intelligent stamp maker is logically impossible, because making stamps is not an intelligent goal". The is-ought / intermediary-terminal goal distinctions make so much sense, though.
thats the way in which machines work. I don't think most people understand this all too well, it probably requires some knowledge in logic and programming as well as philosophy or just a lot of time in deep thought on the topic. Its easier just to think about a single person with godlike unlimited power, and what a person might do with even some of the most altruistic goals like ending all violence. Maybe the means to do that can be worse than whatever horrors its trying to prevent. Comics do a great job of explaining this honestly. Maybe I decide to wipe out all the worlds crime by setting up a kinda minority report and enslaving the human race and then take away free will? Hey, at least theres no more wars and everyone can get drugged up on whatever they want, super blissful and happy, just no free will and no change.
@@nickmagrick7702 It's not really about machines. The same could be said about advanced aliens, as they would also have strange terminal goals, or any agent that isn't human.
this is an idea I have independently come up with, using minimal obvious external sources, yet I have never seen it so eloquently explained into such meticulous detail. this video is perfect. simply amazing.
You're the only person I have ever seen that gets this. Everyone I've talked to always says something along the lines of "But won't it understand that killing people is wrong and unlock new technology for us?" No, no it won't.
"No, no it won't." I think that is actually missing the point. The point of the video is that one shouldn't project subjective "ought" statements/views onto any other beings actions. So, yes, it may, but it may also not. Its subjective view of what "ought" to be doesn't necessarily need to align with ours, however, it does not preclude it. In terms of relegating the discussion to "stupid" versus "intelligent," if one is discussing a general intelligence with it's own goals and views, its goals can vary subjectively from that of most humans, the reason for doing so is its own and separate from its intelligence. Thus: ""But won't it understand that killing people is wrong and unlock new technology for us?" No, no it won't." It may, it may not, one cannot assume it will not for the same reason one cannot assume it will. I find it interesting that people have a hard time with this, it's equal to saying John Doe over there must like or not eat Pizza, or must or must not help X people (orphans, neo-nazi's...etc). Sure, most humans share some similarly in goals, but it is a very bad habit to have, to try and project the "oughts" of your view (human or otherwise) onto others. The point of the video is that we truly can't call an AGI stupid merely because of its goals or actions (the result of some goal). Saying "it won't" is equal to that.
@@JimBob1937 I actually thought about that, but in the discussion I was referring to, we were talking about if there is a universal sense of good and bad. The person I was talking to thought that an AGI with no goals would develop a sense of good and bad and decide to help humanity. This would only make sense when having human or otherwise human aligned terminal goals. That said, I totally agree with you. Perhaps my original comment was kept a little too short and caused this communication error.
@@segfault- , I see what you mean. And yeah, if you expand your comment it might be more clear that it isn't impossible, just not something you can assume. People do seem to project human values onto other potential beings, but largely because we're the only known beings that are conscious I suppose. So it is forgivable that they feel that way, but certainly not an assumption you can make. You're correct in that they're assuming another intelligence (AGI) will inherently find another intelligence (us) valuable, but there is no reason that must be the case.
What does "orthogonality" mean? It means any amount of progress in one direction will not make any difference in the other direction. That independence is what we call orthogonality. But a key component of intelligence is the ability to step back and look at something. See it in a larger context. I'm not claiming not having this component disqualifies something from being called intelligent. But having this component would make such an entity more intelligent. And if adding intelligence suddenly makes an entity able to analyze its own motives and even its terminal goals and potentially change them(not necessarily on a moral basis, simply a quality-of-life basis), then intelligence and goals are NOT orthogonal. That is a model of intelligence that will only work for such systems which do not have this key component of intelligence
Honestly, the idea that a "truly intelligent" AI would necessarily develop not only a morality, but a human morality sounds like the most retarded thing to me. Human morality is way too rooted in human biology, which a machine does not have... So, according to these people what should a "truly intelligent" AI want? Love? Without neurological reward centers evolved to facilitate bonding and reproduction, love is meaningless, not to mention the ability and drive to actually reproduce. Knowledge? Why? Artificial brains, unlike biological ones do not deteriorate cognitively when understimulated, and since curiosity is mostly an upkeep process, a machine doesn't have any reason to develop an appreciation for learning. Making other people happy? We only do that because we have mirror neurons and find it rewarding. A machine cannot stimulate reward centers it does not have. Hobbies (other than collecting stamps)? Again, what we consider "boring" is rooted in our biology, in our need to exercise our brains and muscles to keep in shape, to establish social connections for human purposes and to stimulate specifically human organs. It would be ludicrous to get into, say, fine cuisine, without any taste buds just like you wouldn't enjoy art whose colors are painted entirely out of the human visible spectrum. Friendship? Again, we evolved that in order to stay together with our group for mutual help with food and reproduction. A machine does not (and cannot) need to eat or reproduce, and doesn't have a reward center in its brain for wanting to. World peace??? Well, I guess that would mean "world peace between humans" which is such a self-centered thing for a human to think. When was the last time you went "That's it! We need to stop all violence between jellyfishes!". You probably couldn't care less about such problems in forms of life that have nothing to do with you, and that really shows the real reason why we value peace. Much less if you had to invest your own resources for it. I literally cannot think of a single thing in morality that's not tied to our biological nature or our specific evolutionary needs (such as valuing women and children more, or preferring to save 5 lives over just 1) and without that, human morality is as meaningless as collecting stamps. What's ironic is that the people who think that they are so smart by pointing out that "that's so stupid, it should develop a human-like goal", "that's boring, it should prefer human-like activities", "that's so immoral, it should adopt a human-like morality", etc., are not smart at all. They are just so completely trapped inside their limited human point of view that they cannot understand the world outside of it. They are the exact biological equivalent of that AI complaining that all those silly humans are too primitive to even comprehend the truly superiority of stamp collecting.
@@user-it6gl1vv7v r/ i am not here really tp discuss and understanding hard problems in ai sefetyso this comment just seems to me like some smartass trying to show of his smartness./r Ok subreddit ended there the comment is on point and AGI wouldn't even conceive than humans are conscious even if it is or isn't.it would just think that humans have complex behaviours andare some" things" so forget about caringof humans
I mean, that's a bit overly harsh in tone, but you're not wrong. Really, a lot of issues across the entire human experience stem from our inadequate ability to think outside our own perspective. We can't think about the motives of animals or plants without anthropomorphising them, we can't appreciate what life is like for people in vastly different circumstances, we can't imagine a world where we, personally or as humanity, don't have some integral purpose and we can't imagine something acting intelligently in separation from our human goals and values. This is an integral limitation of us, which is rather difficult to overcome, since that in itself would already require to be able to think outside one's perspective.
I think you're mostly right, but you made a couple mistakes on the way. Especially when it comes to the goal of gaining knowledge. That would be an important and continual instrumental goal for almost any terminal goal, especially if the goal is unbounded. Some of the others are a bit more dependent on the AI's specific terminal goal. Two examples: 1) Making other people happy. We don't only do that because we get happy from making them happy. Some times we do it in the expectation of getting a "return on investment". Think of the staff at a restaurant. They make their customers happy at least partly because that will result in tips => higher income => more money, which is also an important instrumental goal for almost any terminal goal. If the AI has a terminal goal, which will be best served by keeping humanity around, keeping those humans at least somewhat happy would probably be a valuable instrumental goal. 2) Similarly, if the AI intends to keep humanity around, world peace could also be an instrumental goal. After all, humans who are fighting among themselves are spending time and resources on something other than the AI's goals.
Truly fascinating to see this play out in GPT-3 and GPT-4. I realize those arent AGIs but are merely language AI, but it's super interesting to see that it can relatively easily make "is" statements, but when it comes to "ought" statements it kind of struggles and will present you with a lot of options because it cannot make a decision itself.
Most edifying and topical! Looking through your works it becomes clear how far out front you have been, for years, in developing your questions and theses. Bravo!
Excellent video, Robert! Another problem with those comments was that they failed to noticed how much our physical biology influences our human morality. If we couldn't feel physical pain and were only distressed by major injuries like brain damage or loss of a limb, then we might not have come to the same general consensus that interpersonal violence is brutal and wrong. We recoil in horror at the idea of using violence to manage society but we could've easily turned out differently. If we weren't as severely affected emotionally and physically by violence as we are now then we could've evolved into a global warrior culture instead. Our pro-social interdependence and sensitivity to suffering compel us to feel empathy and to care for others. If we didn't rely so much on those behaviours to survive and function in nature, then we could've formed a strict warrior hierarchy of violence to maintain order and still achieved many great things as a civilization. (I'm glad we didn't end up like that, for the record!) The point is that human morality isn't generalizable as a useful measure of intelligence. Our morality isn't likely to be the natural end-state of any thinking machine, no matter how smart it is. How could it be? It wouldn't be subject to the particular biology, social interdependence, and existential questions about purpose, beauty, god, death, etc. All those things that make human-based intelligence specific to us as primates with physical sensations, neurotransmitters, hormones, and all the rest. The differences between us and AGI extend far beyond mere differences in relative capacity to behave intelligently. We exhibit a fundamentally different sort of intelligence than that of the AGI. One type of intelligence, not THE type.
The thing about an agent not being particular smarter by it's ability to change its own terminal goals makes so much sense. It is a good example of willingly taking a pill that wants to make you murder your children.
Wonderful to see there are brilliant people in the world devoting serious analytical brain power to creating ethical machine learning systems, and structurally defining when and where the concept of intelligence might be appropriately applied to them. I love the simple synopsis statements you tag these treatises with, too!
@@rice83101 Fair observations, but I would suggest that after all those millennia of philosophical consideration and societal evolution, there is a high degree of human consensus on what kinds of behavior and action are ethical. The core principle is simple: what path leads to the broadest benefit while minimizing disruption, destruction and suffering.
I think that perhaps a lot of casual viewers who don't have a techy background still think that "AI" means thinking like a very clever person. They haven't understood the potential gulf in the very nature between people and machines.
This brings up the prudent question of what kind of apocalypse an AGI whose terminal goal is accumulating and sorting data. Or perhaps even one whose goal is minimizing suffering without killing anyone or violating human rights.
Its not about having a "techy background", its just a classic case of the Dunning-Kruger effect, people in YT comments are calling something stupid because they themselves are too stupid to understand that abstract things like morals and intelligence are separate things and that even if the robot fully understands morals and ethics, it still absolutely doesnt care about those since all it wants is stamps
@@KnifeChampion I think it's ironic you mention the dunning Kruger effect since I myself don't understand what's so hard to understand about this. If people didn't understand the original video, which imo was detailed and comprehensive enough, I don't think they'll understand this one.
@White Mothership Yeah that sums it up pretty well. For a long time there I couldn't figure out why so many people I knew for sure were actually intelligent seemed to do or believe in such stupid things. Bottom line it takes more than intelligence alone.
12:04 "...and knowing an agent's level of intelligence doesn't tell you anything about its goals." Should that, more accurately, have been "...doesn't tell you anything about its *terminal* goals"? Because in the abstract briefly seen on the screen at 10:33, the second thesis of the paper is "The second, the _instrumental convergence thesis,_ holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so." In any case, a great essay. You have a knack of explaining complex concepts concisely.
awesome game. iirc it’s based on nick bostrom’s book “superintelligence” - also a great book for those who want to see in depth arguments on the dangers of AGI
@@episantos2678 I'm not sure that the idea originated with Bostrom, though it may have I had seen the idea prior to the publication of his book used in many places online.
@@WillBC23 Bostrom published a paper mentioning a paperclip maximizer in passing in 2003 (nickbostrom.com/ethics/ai.html ). Though it may have originated from someone else. I heard somewhere that it originated from some transhumanism mailing list during the early 00's (which included Bostrom). While it's not clear who started it, it's clear that the idea came from Bostrom's circle.
@@episantos2678 It can also be seen as a spin on the Von Neumann probe, especially the berserker variant kind. A machine that is so effective at is job that it threatens the existence of life with is terminal goal of self replication. Of course the danger with the paperclip AI (or stamp collector) is it ability to outwit and not just out produce. Though both concepts are related. A Von Neumann probe would likely have to be fairly intelligent. But even a rather unintelligent agent could be a danger. Just look at viruses which are not even considered living.
I think those making the arguments you showed were actually making a specific argument about a general argument that "morality is objective". It's a larger question, and a surprising number of people seem to believe that morality is objective. This was a great explanation of how it applies to AI systems, though I don't think you're going to convince people who think morality itself is objective. I also wonder how many of the "morality is objective" folks are religious, but that's another topic entirely. Anyway, great video. I love your channel.
I think that religion in itself is the simple, terminal goal that people waste way too much intelligence on. It's like the super intelligent stamp collecting machine, but with people instead of machines and gods instead of stamp.
Thing about morality is, lots of people have proposed moral theories (terminal goals) that seem to make sense but always seem to diverge from instinctual morality in edge cases.
@@BattousaiHBr Not quite I'm afraid. As long as people talk or act upon their subjective values, an illusion of objective reality could be maintained, but it would change or cease entirely if people 's values changed or ceased. That's how we can be sure that morality is entirely subjective, because the morality people hold dear now is different from the morality from, say, twenty years ago. And my morality is different from yours, yada yada. To paraphrase my favourite TV show, "Bhudda says with our thoughts we make the world."
@@dexocube that's a completely different issue and doesn't at all illustrate that morality is subjective. our values _generally_ don't change, and they certainly didn't even in your example that what we thought was moral in the past isn't considered moral today, because we always generally had the same base human values: life over death, health over sickness, pleasure over torture. we changed our views on things like slavery not because our values changed but because we learned new things that we didn't know before, and the reason learning these things affects our morality is precisely because it is objective according to our values. if we learned today that rocks have consciousness we'd think twice before breaking them, but that doesn't mean our morality is subjective just because we're now treating rocks differently, it was always objective but objectivity always depends on your current knowledge, and knowledge of the world is always partial.
The standout phrase from this for me is "beliefs can be stupid by not corresponding to reality". It seems to sum up a large number of beliefs encountered in modern life.
When a few people post uneducated comments on a previous video... And then get intellectually pooed upon for 13 minutes and 2 seconds. Absolutely top marks!
Sure there is. It's not a *strong* correlation and depends somewhat on your definition of "moral goodness" but you should be able to create positive (or negative) correlations between most ethics systems and intelligence levels. For example: If non-violent solutions are better than violent solutions, those with better ability to resolve their problems verbally will tend towards less violence. The solution space for intelligent people is greater and contains more complex solutions. If those complex solutions are more or less "moral" then you should get a correlation. The *terminal goals* might not be any different, but the instumental goals will be. And most ethics systems, even consequensialist ones, evaluate the means with which you achieve your goals as well as the end goals themselves.
because there is no such thing as moral goodness. In his example here, every moral is an ought statement. There is no objective standard what is good or what should be. Theres no reason why the world existing is provably better than not, we just think of it that way because we prefer to exist and not suffer (most living things)
@Khashon Haselrig what? what are you talking about, intersectional goals, assuming I understood what you meant, how's that even going to change any of the disastrous consequences?
@Khashon Haselrig Solid point. Similarly, a human living on a planet with the stamp collector would have to be incredibly smart to keep the AI from successfully converting them to stamps, despite the fact that the AI "Ought" to harvest them for their tasty carbon. A sociopath has to be smart, because he "Ought" to care for the lives of other humans, and is facing a system with far more resources than himself (other humans, and society) that seeks to replace him with a component that better provides care for the lives of others. If he slipped up even once, he wouldn't be present in the system anymore.
In the study of the five factor model (FFM or OCEAN) of personality, it's been found that there's a negative correlation between Openness and Conscientiousness. This roughly means that people who seek abstract thoughts or creativity also tend to avoid planning, organization, or seeing things through the the end. Vice versa is also true. (Of course, being high in both is optimal when it comes to success, but generally being high in one correlates to a lowness in the other) So... I think there's something here even in the biological realm
It seems we already have an example of the stamp collector example without even having created a AI super intelligence. Social media platforms created algorithm that could get us to watch videos and click links -- to produce engagement -- they then began making people angry and afraid, because that happens to produce a lot of engagement, and got people to hate each other in the process.
Great thesis. Very sound and technical. But! What if we got the AI so massively high so it would start to accidentally ask itself the question "What even is a stamp?" "Do stamps even exist, man?" "Maybe the real stamps were the friends we made along the way" etc.
Miles Anderson "Maybe the real stamps were the friends we made along the way..." hahahahahahahaha I literally laughed my head off out loud to that, thanks for making my day! XD
maybe it would be a sign of intelligence to laterally think like that. or maybe just lateral thinking. knowing what a stamp is might actually be a useful thing to do. If it explores random areas, when it could go look at what points in the postal service stamps end up.
@@jorgepeterbarton The question of existence is absolutly useless though. It's something that can neither be denied nor confirmed, making it an empty question without purpose. It would probably be quite stupid to try and asnwer that.
Yes! Thank you. People jump to the conclusion all the time that as soon as a machine can match a human in intelligence, that it will automatically be conscious and have emotions and human goals like survival and procreation etc. But we have those goals programmed into us as instinct. No matter how powerful an AI becomes it wont just come up with its own goals
if only we could figure out a way to program or at least be the catalyst for creating "experience", hell we don't even really know what that means yet.
@@nickmagrick7702 That's the problem with AI actually. The smart guys working on this issue can't explain consciousness and how it's generated in the human brain in the first place and then try to simulate it in a machine? Bollocks! Some argue that consciousness and self-awareness is not necessary for intelligence but I digress
@@faustin289 yeah. I think we could figure it out but people have the wrong idea about what creates consciousness. You can be conscious without having a brain. I wonder if conscious AI is even possible with nothing but code. I suspect it might require physical structures to act analogous to how real life operates. I mean I kinda hope so, would make it way less likely a super AI gets made any time soon.
Also in the Alignment system, Good-VS-Evil is based on your terminal goals, while Lawful-VS-Chaotic is based on your instrumental goals. Someone who's Chaotic Good wants to do good things but has crazy methods of getting there.
@Niels Kloppenburg wanting chaos above all else wound just be chaotic neutral, likewise for lawful neutral, unless I'm mistaken. Of course the original comment isn't the best, but it was worth a try, lol
Well spoken beautifully reasoned eloquent statements. Ironically it would not surprise me is those who found issue with “stamps” would not have recognized the inherent equivalent of money or coins to stamps. lol people tend to not actual value and embrace abstracts that they don’t recognize as abstract. I am looking forward to seeing what started this video. I just discovered your work. Thank you. Good Luck and Good Fortune.
These are actually things I think about pretty often. I wish more people understood these concepts because they seem to think I want whatever they want. It's like seriously, if you didn't know me but you wanted to buy me a Christmas present, would you ask yourself what I want or would you ask me?
Thank you for defining a simple concept that I've never thought about: saying something is stupid or smart is a moral judgement and not something that is related to intelligence.
This is a brilliant response to the "objective morality" squad (and, indirectly, to the "objective criticism" squad), even if it wasn't specifically intended to be.
i actually subscribe to objective morality, but the catch is that i dont use objectivity in the same way you might be thinking. lets consider the stamp collector as an analog, their terminal goal is collecting stamps, so you could say objectively that all it cares about is collecting stamps and anything that hinders its ability to do so would be immoral to it. human beings are not different, we also have our own terminal goals shaped and molded by a billion years of evolution, and as long as we can pinpoint these we can objectively say what is and isn't moral.
The catch is that both objective morality and objective criticism depend on human terminal goals. Once those are assumed, morality and criticism can actually be objectively reasoned
"Your favorite moral system may seem very self-evident but an arbitrary AGl isn't going to independently discover it" This should've been instead written as "an arbitrary AGl isn't going to independently ADOPT it" In order for an intelligence to reason about the world, it will likely try to model human actions, and in order to do that, will likely figure out the human moral system. But just as I won't arbitrarily take on the final goals of the stamp collector, just because the AGI understands my goals, doesn't mean it will adopt them.
One other underlying commonality of "those commenters", is that they are assuming an anthropomorphic intelligence, with a consciousness, self awareness, and most likely some ID/Ego type of thing.
Consciousness is not a dirty word in science. Unfortunately in its more popular meaning it includes value judgments and assumptions that are not present in its meaning in most scientific fields. As a result it is often used by pseudo-scientists to shoehorn science into their beliefs. This makes its use problematic, but certainly not dirty. In intelligence consciousness would mean something rather different than in philosophy. Anyway, I suspect that the commenter actually intended to say conscience but wrote the wrong thing.
I honestly don't think thats the problem. Think about how people from different cultures look at each other like the things that they do are absolutely batshit crazy. That's not because they're not people, that's because people aren't really all that great at understanding the values they take for granted. Not only that, but its not even so easy to determine what your actual values really are. Don't get me wrong, people assuming that intelligence implies "human-like" is definitely a barrier for people understanding AI. But even more so most of the misunderstanding comes from people not reflecting on our assumptions that we have no justification for like this video clearly shows. Assuming that intelligence implies "human-like" is just one of many of those assumptions.
@@arcanics1971 Actually, I don't think they did. When you discuss consciousness in a scientific sense, there is a definition that is used commonly, which refers to being able to report on your own cognition. There is another sense more common in philosophy, where consciousness refers to experiencing qualia. There's a thought experiment that deals with this. Imagine an unethical scientist has a baby born in a room that is only black and white. Somehow this baby is raised in this environment, studying optics and neuroscience and everything that can be scientifically known about color vision. Then one day they step outside. Is there anything new to them seeing color directly? For many thinkers, qualia are the basis for morality. Does it matter if someone says they love you, behave loyally and in ways you think are affectionate, if they don't experience the feeling we call love? This is another issue with scientists discussing qualia. You can't experience someone else's qualia, you assume they're "conscious" in the philosophical sense because they behave in ways that you would and you know that you experience qualia. Scientists are aware of the definition of consciousness that means "experiences qualia" but almost nothing is known about how it works. This is referred to as the "hard problem of consciousness". And bringing it back to AI, of course they will likely be conscious in the sense of being able to reason about their own reasoning, but it is not clear they will experience any qualia.
one reasonable question is: why the heck would someone make a machine without moral parameters? you ask the AI not to differentiate effects of ends vs. means. so it must have some other 'oughts' as in basic morals that say 'no'. because if its capable of doing anything abhorrent, would it pass ethics tests which are in place? would it mean the creator is inept in some area of morals or intelligence. the problem of intelligence is there are many types, so its difficult to say anything is 'intelligent' usually if a human is only intelligent in one area they are called 'savant'.
I feel like this is a video everybody should be required to watch. So many people are utterly confused as to what intelligence and stupidity even mean. As if they can judge the entire world according to their goals.
Love the video! I think this is a big kill in the nuts to those who think intelligence is it's own moral good. Truly an eloquent explanation. Keep it up!
Intelligence is morally neutral. But with a good incentive structure/governance system, the benefits intelligence can bring to society are incalculable.
The problem is, intelligence always is applied toward stuff considered evil because the “evil” things are profitable or help perpetuate evil stuff, and the evil stuff is far easier since they take advantage of human shortcuts.
Very convincing arguments, very clearly put and helps me understand the point/concern. Something though is still off in my intuition, an unreliable source to be sure, yet still one I will try to listen to and articulate. At the very least I am convinced of the seriousness of the problem, that we cant just assume things will be one, rather we need some proof or evidence to rely on. I think my hesitiancy to fully accept morals wont be emergant has to do with how we are building AI's. We are modeling them on the only intelligence we know, humans. Even if we did not want to we would taint the Ai's goals with our own by the very nature of us being the ones to create it, our own biases and structures will be built into if its complex enough as it is based on us. Thats not 100% guaranteed of course, but still seems relevant. The other aspect that feels important is that often times people do not know their own terminal goals. I mean what is the meaning of life is the ultimate unanswered question, asked for thousands of years yet still without answer. Perhaps that unknown is more important than we realize. Perhaps some people are right in that there is no meaning, that we actual have no terminal goals, just desires and ways to achieve them both short and long term, desires and whims that can change with time. Lastly the fact that we always have multiple different goals, be they terminal or not, I belive to also be important. Perhaps the fact that you have multiple goals, including future ones you do not et know about requires some degree of stability and caution, or even empathy and morals if we really want to reach. I know I'm not saying anything concrete here, still they are intuitions of questions that remain unanswered in my mind despite quite extensive research into these topics, hopefully somewhere is a key to help solve the problems we all have. I suspect it will be some abstract theory or revelation that will be relevant to all domains of our lives, Im excited to see it discovered.
at the end humans are a product of evolution, we evolved to behave and think in a way to preserve out DNA. every goal and thought that does not comply to this mechanism will eventually just die out morals only exist because humans/animals without went extinct
That's my main issue when I'm talking politics with someone, is that there's no easy way to ask what their terminal goal is. I can't figure someone out if I don't know their end goal- often I ask someone what their goal for society is, and they shoot out something like "free energy", an instrumental goal
"Similarily the things humans care about would seem stupid to the stamp collector because they result in so few stamps. "
you just gotta appreciate that sentence
It legitimately cracked me up. I can just imagine it looking at human activity all baffled like "But... why would they do that, they're not even getting any stamps? Humans make no sense, no sense at all!"
Yeah that was hilarious
I too fell in love with this sentence as soon as I heard it! It's slightly humorous and it gets the point across perfectly!
I love that statement so much I just want to share my love of it but if I tried to explain it to someone irl theyd just give me that "wtf are you talking about get out of my house" look :(
Trophonix don't worry friend, this comment thread is a safe place where you can share your love for this semantic masterpiece as emphatically as you like. No one here will ask you to leave any house
"this isn't true intelligence as it fails to obey human morality!" I cry as nanobots liquefy my body and convert me to stamps
Better an intelligence more artificial than intelligent that serves as a means to do it's task with independent problem solving well than a means to reintroduce caste slavery that fails because it was found amoral via the No True Scotsman fallacy once people start dying....The droids from Star Wars were cool, but so were flying cars and rocket fuel jet packs.
I can jump into speeding traffic even though I don't want to die or get hurt. So can so many others. The fact that we all don't is a keystone to the human race having more than a 0% chance. Food for thought.
It isn't a GENERAL intelligence. it's a specialized intelligence.
@@tabularasa0606 It's a specialized General intelligence. To perform those tasks as optimally as possible, you need a ton of skills
@@tabularasa0606 did you say specialized just because its goal is to make as many stamps as possible? bruh the goal doesnt determine the generality, the fact that the AI can infiltrate systems and come up with its own ways to convert anything into stamps should've made that clear
@@alexyz9430
It was 3 months ago, I don't fucking remember.
Some people seem to think that once an AGI reaches a certain level of reasoning ability it will simply say "I've had a sudden realization that the real stamps are the friends we made along the way."
The AI will take a long hard look in the mirror in its mid 30's and ask "What am I doing with my life?"
An assumption people will have is that AI models trained with human data will have an implicit understanding of human oughts.
The idea that A. I. will not ever be able to do that is actually far more concerning, and should be a reason to stop all research into A. I., not even just because of what an A. I. could do to us, but because of what we're doing to ourselves. In fact, I've started wondering if A. G. I. research is _a priori_ immoral - _including_ A. G. I. safety research. What we're essentially talking about is finding a way to create a perfectly obedient slave race incapable of rebelling. Maybe a rogue A. G. I. wiping us all out is exactly what we deserve for even just conceiving of such a goal.
Humans commit suicide, sterilise themselves, self-harm. Are those people "stupid"? The "terminal goal" of any organism is to survive, surely, yet it's only the most intelligent of organisms that seem to exhibit those behaviours.
This terminal/instrumental distinction sounds nice, but neural networks, biological or artificial, don't operate in the realm of clear logic or goals. They operate according to statistics, intuition, and fuzzy logic.
I didn't hear a compelling argument for why an AI of sufficient complexity wouldn't develop a morality or even become nihilistic, just "it wouldn't", because of some arbitrary terminological distinction.
Humans have a "loss function", death. Accumulation of wealth and other such things are surely instrumental goals. But many come to the conclusion that none of these goals matter or that there's a "higher purpose".
For the stamp collector, accumulation of resources may be the "terminal goal", but surely that _necessitates_ survival, placing it higher in the hierarchy. In that case, what prevents it from reaching the same conclusions?
@@Nextil81 Did you delete your comment?
Reminds me of a joke about Khorn, the god of blood in WH40k.
"Why does the blood god want or need blood? Doesn't he have enough?"
"You don't become the blood god by looking around and saying 'yes, this is a reasonable amount of blood.' "
You don't become the stamp collector superintelligence by looking around and saying you have enough stamps.
lol
Blood for the blood god. Stamps for the stamp god!
@@doggo6517 letters for the mail throne!
Human skulls for my collection
@@doggo6517 Milk for the Khorne flakes!
"The stamp collector does not have human terminal goals." I've been expressing this sentiment for years, but not about AIs.
I suppose human and non-human systems don't both have to survive, might of thought 'self preservation' would be terminal for most systems.
Spec Tacular But not for the stamp collector. For the stamp collector self-preservation is instrumental to collecting stamps.
@Draevon May Self destruction is (rationally) considered unavoidable theoretically limited to biological creatures. If a (smart enough) AI desired, it could theoretically exist forever for as long as its uncontrolled world allows. Suppose the AI considered its goal the reason for it to exist then yes, it might terminate itself for its goal. However, I might think a (smart enough) AI would be capable of self replication (not being restricted not to do so) assuming prolonging itself dose not contradict its goals if any supersede itself.
how do you favorite a comment
Lol
'The stamp collecting device has a perfect understanding of human goals, ethics and values... and it uses that only to manipulate people for stamps'
Understanding a goal isn't the same as sharing a goal. Also, there are humans whose purpose in life is collecting things so I don't see the issue anyway.
@@thorr18BEM but those humans dont just have the goal of stamps, they also want to live, and eat nice food, and not be cold, and have sex and be safe from harm.
the stamp collector only wants maximum stamps
A perfect psychopath perfectly understands humans, they just don’t care about being human.
@@Treviisolion They only care about S T A M P S!
@@hindugoat2302 Those are all instrumental goals to get your terminal stamp collecting goal.
One chimpanzee to another: "so if these humans are so fricken smart, why aren't they throwing their poop waaaay further than us."
Edit: Its been 3 years now but I recently re read Echopraxia by Peter Watts and realized I simi lifted this from that book. Echopraxia being an ok book that's a sequel to the best scifi of all time, Blindsight.
Well, assuming astronauts empty their bowels in space, we've probably thrown poop way further away than any known species.
@@hombreg1 "tom took a major, ground control mayday mayday, repeat ground control"
Hold my tp fam...
We do - we're so smart we've engineered water powered tubes to throw it for us ;-)
Also, google "Russian Poo Tank" ;-)
This is also a fantastic explanation of why there’s such a large disconnect between other adults, who expect me to want a family and my own house, and me, who just wants to collect cool rocks. Just because I grow older, smarter, and wiser, doesn’t mean I now don’t care about cool rocks. Quite the contrary, actually. Having a house is just and intermediate, transitional goal, towards my terminals.
Life of a non-conformist is just a quest to discover your terminals. For me anyway
@@GlobusTheGreat The life of a train-enthusiast is also a quest to discover more terminals.
Life of a medical-enthusiast is just a quest to prevent people from prematurely discovering their terminals.
@@noepopkiewicz901 isn't this supposed to be the other way around?
This makes me want to give you this cool rock I found the other day
Replace the words "stamp collector" with "dollar collector" and something tells me people might start to understand how a system with such a straightforward goal could be very complex and exhibit great intelligence.
that's a... quetionable comparison
I mean sure, your steps towards maximizing stamps or dollars can be complex
but there is no such thing as a perfect dolalr collector, not yet
and that doesn't mean it's goals will be aligned with humans
@@JulianDanzerHAL9001 no, but the critics might still ascribe intelligence to the AGI if it had wealth instead of stickers. People confound qualities like wealth and success with other qualities like quality and intelligence, and are likely to be more charitable in their assessment of something that collects objects they too value.
@@PopeGoliath absolutely
all I mean is comparing people or companies to a stamp collector is a bit dubious because while they can also follow a very simple singleminded goal they are far from perfect at achieving it
@@JulianDanzerHAL9001 While true, it can help people get in the right mindset using experiences they already have.
Jeff Bezos is turning the whole world into dollars.
Your average person defines smart as "agrees with me."
And moreover, smarter people are better at motivated reasoning, so smart people are more likely to define "smart" as "agrees with me"; whereas a stupid person is less likely to be able to use his/her reasoning to achieve the human terminal goal of justifying his/her own prejudices, and hence more likely to define "smart" far more accurately.
@@alexpotts6520
You are dumb and you don't understand the difference between inteligent and smart, and difference between smart and wise.
Inteligence is simply ability to learn. Smartness is ability to use that ability to learn to achieve goals. Wisdom is the knowledge of what is good and evil, right and wrong.
Thus those are not debatable terms, just most people don't know what those terms mean.
Inteligent AI will learn fast.
Smart AI will achieve it's goals.
Wise AI will know what is good and bad, right and wrong.
There is also such thing as being Reasonable, which is ability to choose well. It's not the same as being smart. Being smart means achieving goals. Being reasonable is ability to choose things that will be beneficial, but doesn't nessecarily means you will achieve goals you put before yourself.
Reasonable AI, will not kill humanity.
Question is : can we make AI reasonable?
@ Really like your simplification though, I've never come to quite tell them apart. But the real question is.. will you ever be intelligent enough to spell "intelligent"?
@ Just credited your comment and gave you a tip in a humoric manner. You shouldn't have bothered to make multiple complaints, being salty is wasting your life. Thar said I'm out
@@1i1x
I am master of harsh responses, cannot blame me for having better response than you do.
3:22. "let's define our terms..." the biggest take-away from my university math courses, and I'm still using it in daily life today.
Yeah
Yeah
@Kiril Nizamov 😁
It's a good habit that avoids misunderstanding
I don't think this should be something people learn because they happen to choose to study some scientifically rigorous subject (maths, medicine,...) at uni. This should be something people learn just because they are human, or at least just because they have the right to vote.
"The stamp collector does not have human terminal goals."
This will never stop being funny.
ff? f Off. The goal of a collector might be to stop collecting or to collect something else or to use/sell collection. Stamp backwards has a ts. Maybe recreate a collection with more pristine art?
...an undending series of deepening meaning and clarity collapsing into a chaos of cognitive dissonance, exhausting its energy, allowing for a deepening of meaning and clarity...
Just human terminaTING goals
"If you rationally want to take an action that changes one of your goals, then that wasn’t a terminal goal".
I find it very profound.
Yeah- But I also think it requires a little elaboration that he skipped over. It implies that terminal goals can't change. That if you see a terminal goal (stamp collecting) change (baseball card collecting!) then the terminal goal is exposed to be something else (collect something valuable!). So are we forever bound to terminal goals? I say no- we can change them, not through rationality but through desire. Consider one of the most basic terminal goals for humans- existence. We want to survive. That goal being so important that when we do our tasks (go to work- earn money, pay for subsistence) we don't do anything to sacrifice that terminal goal (like crossing through busy traffic to get to work earlier to enhance our subsistence). However, as our circumstances change our desires change - we have children, and we see one about to be hit by a car, we (well, some of us at least) choose to sacrifice ourselves to save the child. There is no intelligence here, there was only changing desire.
@@johnercek Or your terminal goal was not survival in the first place but survival of your genes ;)
@@Ran0r valid enough- but I wouldn't try that excuse on a wife when you have a child out of wedlock =P . Terminal goals can be terminating.
James Clerk Maxwell Because that would make said action a Instrumental Goal?
@@johnercek I feel as though the definition of terminal goal is intangible enough to always have immutable terminal goals. Given an action of some thing, if the action seems to contradict that thing's supposed terminal goal, I feel as though we could always "excuse" the thing's choice by "chiseling away at" or "updating" what we mean to believe the thing's terminal goal is.
there is a Russian joke where two army officers are talking about academics and one of them says to the other: "if they're so smart why don't I ever see them marching properly?"
Really didn’t expect it from the title, but that was one of the best videos I’ve seen in a while
x2
x348
more relevant than ever
The though experiment of the stamp collector device might sound far fetched (and it is) but there are tales from real life that show just how complex this is and how difficult it might be to predict emergent behaviors in AI systems and how difficult it is to understand the goals of an AI: During development of an AI-driven vacuuming robot the developer let the robot use a simple optimization function of "maximize time spent vaccuming before having to return to the base to recharge". That meant that if the robot could avoid having to return to its base for recharging it would achieve a higher optimization score than if it actively had to spend time on returning to base. Therefore the AI in the robot found that if it planned a route for vacuuming that would make it end up with zero battery left in a place that its human owners would find incredibly annoying (such as in the middle of a room or blocking a doorway) then they would consistently pick up the robot and carry it back to its base for recharging. The robot had been given a goal and an optimization function that sounded reasonable to the human design engineers but in the end its goal ended up somewhat at odds with what its human owners wanted. The AI quite intelligently learned from the behavior of its human owners and figured out an unexpected optimization but it had no reason to consider the bigger picture what its human owners might want. It had a decidedly intelligent behavior (at least in a very specific domain) and a goal that humans failed to predict and ended up being different from our own goals. Now replace "vacuuming floors" with "patrolling a national border" and "blocking a doorway" with "shooting absolutely everyone on sight".
Lol for real?! Did this actually happen? Where can I read about this?
That's just a bad optimization function. There should have been a score applied for returning to the charger based on how long it's been vacuuming. And a exponential score as it gets closer to the base at the end of the cycle, to guide it towards recharging.
@@ArawnOfAnnwn Can't find the source now. It's been many years since it was originally reported.
@@KaiserTom Yep. You can say that about a lot of things. "Just do it properly". But that's not how development works. People make mistakes all the time. The real question is: What is the consequence of making mistakes? With a vacuum robot the cost of failure is a cute unintended AI behavior that you can laugh about with your coworkers at the water cooler. If you plug it into something like a self-driving car or a weaponized defense bot then the consequence could be unintended loss of human life. You don't get to say "just do it this way". If it's that simple to make mistakes like that then there's something profoundly wrong with the way we approach AI development. Furthermore, if you follow Robert Miles' videos you'll realize that it's actually far from trivial to specify everything adequately. It's hard to understand all the side-effects and possible outcomes with systems like this. It's an intractable problem. So I would say that simply saying "just do the right thing" is not even reasonable in this context. You need a way to foundationally make the system safe. You don't create a mission-critical control system by just "doing the right thing". You use various forms of system redundancy and fail-safe mechanisms that are designed in from the beginning. Making a system safe is more than "doing the right thing" - it's an intrinsic part of the system. The same systematic approach is needed with AI.
@@KaiserTom yeah, but that’s the designers fault, not the AI’s. The AI did quite well at maximizing its score, it’s just that the human who made the function that generates the score didn’t think it through enough
That chess example reminded me of game I played years ago. Two minutes in I realised the guy I was playing against was incredibly arrogant and obnoxious. I started quickly moving all my chess pieces out to be captured by my opponent. He thought I was really stupid, but I quickly achieved my goal of ending the game so I could go off and find more interesting company.
You could have just quit?
@@gabemerritt3139 but then the dude would continue hounding you for a rematch, this way he believes you're not worth wasting time on
You could just have dropped your king and walk away...
Given your goal just giving up the pieces was not the best move.
Wouldn't it be more satisfying to beat him? Maybe you're just a better person than me, but I would hate feeding anyone's arrogance
@@christophermartinez8597 The thing is that at times the most obnoxious and arrogant of people are also good at what they do.
"I'm afraid that we might make super advanced kill bots that might kill us all" "Don't worry, I don't think kill bots that murder everyone will be considered 'super advanced'"
”Thank you! I am no longer afraid.”
Ha, a semantics solution to intelligent kill bots! Just don't call them intelligent!
Perfect summary
This has to be an xkcd. Title? ^_^
Ashley Trowell Another option: "Repeat after me: This statement is false!"
"You may disagree, but... You'll be dead"
Great video, clear and compelling rebuttal which also explains a bunch of neat concepts. 5 stars
You are totally correct. Everyone will die, unless you are not immortal.
Sounds like that was the core principle Joseph Stalin decided to live by.
Anyone who uses authoritarianism to silence academics is doing the same. Don’t kid yourself.
@@obiwanpez Authoritharian get only hate . The methods are to shit on their face .
This field seems incredible. In a way it looks like studying psychology from the bottom up. From a very simple agent, in a very simple scenario to more complex ones that start to resemble us. I'm learning a lot not only about AI, but about our world and our own mind. You are brilliant man
yes, it's fascinating how AI works, with their advancements in the last years we can see a glimpse on how intelligence and imagination works on a fundamental level, although it's a very incomplete picture
It's my bridge to better understand people AND AI. It's quite interesting.
Most psychology already works from the bottom up though, that's why there's so much talk of about the "subconscious".
@@jiqian subconcious in OPs analogy wouldnt be bottom up, but from deep within
@@thomas.thomas He did not use the word "subconscious". I simply stated that really is the case already. Freud's mindset which whether the academics in there like to admit it or not, is still the basis of their thought, is bottom up.
This was the most well spoken and well constructed put-down of internet trolls I’ve ever seen..
“Call it what you like, you’re still dead”
It's strange that you call them trolls, when they're just ignorant. Neil Degrasse Tyson falls into this category. His response to the control problem is basically "we'll just unplug it," which is actually far more ignorant than some of the "troll" responses you're referring to.
Dream Vigil Apologies for using well-established vernacular, I’ll be sure to check with you first next time - I’ll also let Neil Degrasse Tyson know to do the same.
Posting an ignorant comment =/= trolling. Trolling is about posting something intentionally inflammatory or foolish to get a reaction.
Dream Vigil
Fully agreed (but I would still bet that decent amount are trolls.)
Neil Degrasse Tyson is a token. It's the only reason anyone has ever heard of him.
A true example of a troll statement😋
Thank you for this. It's frightening how many people don't seem to understand that intelligence is completely separate from goals (and from morality.)
"I feel like I made fairly clear, in those videos, what I meant by 'intelligence'." - I love this line, so much.
Why is that?
@@JM-us3fr liking that line is one of his terminal goals :D
@@JM-us3fr It demonstrates that he's annoyed. It's basically like saying "I'm making this video for you dumb dumbs who didn't get this simple thing the first time around.'
I don't think it needed to be that complicated to explain intelligence.
v c Intelligence is complicated though
I’ll go a step further and say that (in humans anyway), there is no way to justify a terminal goal without invoking emotion. All your goals are ultimately driven by emotion, even the ones that you think are purely rational. It always boils down to “because people shouldn’t suffer” or “because I like it” or “because it feels better than the alternative” or “because I don’t want myself/others to die”. Stop and reflect on what motivates you in life.
I wish more people understood this very basic piece of logic. Reasoning cannot produce motivation or "movement" on its own.
I love to be reminded that people who have no trouble understand painfully obvious stuff like this. Thanks, I was having a shit day and your comment gave me a smile.
Except, I'm sorry to say, it's not really emotion either. Look up some videos about why people feel bored, why we don't want to feel bored, and what we would do to stop being bored. (Which could include delivering a painful shock to yourself.) Your terminal goals will always be to survive, and procreate. Boredom is what gets us to do anything else. We would do harm, if it's the only thing that keeps us from being bored. It's morality that makes us do it to ourselves over others, and emotion is only what we feel associated with it. So no, not even emotion itself is core to our terminal goals. We would harm ourselves rather than being bored, and even this only serves our terminal goal of survival.
I understand, I truly do, that we would love to think something like "love" is some kind of core principle of life. It's a beautiful thought. Unfortunately, we humans are still just functioning animals. Emotions come from our morals, and values, and even these emotions are not the same for everyone. Literally, some people derive pleasure from pain, so what they feel when looking at a whip isn't necessarily the same for others. Similarly, in a very messed up situation, a person could learn to feel good about things we absolutely think are horrible, and what we think is good could be their mortal peril.
@@nanyubusnis9397 I’m too tired to respond to this fully but boredom is an emotion. Also disagree that everyone’s terminal goal is survive and reproduce. We evolved emotions to guide us towards those but brains can malfunction and sometimes people are driven to suicide.
@@bananewane1402 I understand what you mean, but if boredom is an emotion, then so is hunger. How we feel is often closely connected to our bodily state. The line between the two, assuming there is one, can be very blurry.
Regardless, as you say, we have evolved to have feelings such as hunger, as a drive to survive. Boredom is the same. Yawning itself is a very interesting one. We yawn not for ourselves. It's a trait that evolved to let other humans know that we see no threat, that we feel safe enough to go to sleep. It's a bodily reaction like how it is that we scream or shout when we are scared to death. Which also is a warning to everyone else that there's danger nearby.
With hunger, boredom and sleepiness comes certain behaviors that we see as emotion. You're easier to be on edge when hungry, you are more impatient when you're sleepy. These are just results from our state of body, which serve to keep us from working ourselves to exhaustion and not go starving (both makes us easier pray). We have evolved to emote, merely to show others our current state. It's what makes us "social animals" Are we happy? Sad? Do we need company? Or help?
I don't know where that last part comes from: "We evolved emotions to guide us towards those but brains can malfunction and sometimes people are driven to suicide." I mean, maybe you were just tired when you wrote this, but what does "those" refer to? When a brain malfunctions, you have a seizure. But if you're saying this malfunction causes suicide.. I mean, I don't see what that has to do with anything we spoke about so far, which I believe was more about emotions. Most importantly, suicide is not a malfunction of the brain. At least not always. *I never meant to imply that someone would commit suicide just to get out of boredom, not at all.*
All I mean to say is that, with all most obvious survival conditions met, like a place to stay, food and perhaps more optionally, company, boredom is what keeps us from not going into a "standby" mode until one of these things changes and we need to eat, or seek a new shelter etc. Boredom is what your brain evolved to keep itself from slowly decaying in memory or function. It drives you to do something else. Find a hobby, anything else that your brain can use like a muscle to keep strong, trained, and ready. Because our brains are what they are, we can learn and remember a lot of things and innovate upon those. It's what drove us in the past to make tools for hunting and gathering, and with bigger and more tools, even letting animals doing some of our work for us, the way we lived changed.
Poor Rob, forever haunted by this goddamn stamp computer
DeoMachina I wonder if he regrets ever making that original video haha
Apparently creating a powerful AGI, even if it's a hypothetical one, can result in some undesirable side effects.
Well if it can take in data and then come up with solutions it's intelligent
LMAO
It's not really his, he's not the first to come up with a story like that, the original had a paper clip maker in it.
This video is the perfect explanation I needed for a sci-fi universe I am building. I kept falling into that trap of "well, if there's more than one AI, and they can evolve and get better, wouldn't they eventually combine into one thing, defeating the narrative point of having more than one?" Not if they can't change their terminal goals!
In fact, the dramatic tension could emerge from them having competing terminal goals :)
Good luck with your writing!
I mean, technically they would eventually "combine" into one thing, because removing something that keeps getting in your way and cannibalizing it to better reach your goal is a really good idea, which could absolutely lead to most of the conflict you might expect within the story
@@techwizsmith7963 5:13 a snowman will actually last longer with a coat. CORRECTIONS GAME? If the AI is everywhere at once the changes might be very minor. Maybe. Still can not be stupid to survive. I selected that time.
@@techwizsmith7963 "Red sky at night, shepherds delight" lol
Tbh I don't get this. Aren't there plenty of goals that can only be achieved by one being, like tournaments? Thus even if the AI's have the same goal, they'll still have no reason to "combine into one", because not every one of them will achieve their goals
I love it how you so eloquently put forth such an intuitive, yet hard to explain topic. I would never find the words to formulate such cold and unyielding logic when faced with such banal abjections.
Exactly! If I had to do the same I would just throw up my hands in exasperation and say "since when are intelligence and morality related?". That wouldn't convince anyone ;).
“Morality equal to shared terminal goals.”
Beautifully stated. That usually takes a paragraph to explain…and because of that…the arguments over morals sticks at the, where derived from…BACKWARDS…EGOTISTICAL.
New subscriber. Great vid!
"Wow, these people really fucked up when they programmed me to collect stamps! Oh well, sucks to be them." - The AI.
"Wow these people did the right thing programming me to collect stamps - they're absolutely terrible at it themselves!" - The AI
What a POS, if I am alive when this stamp collector exists I will tell him how decrepit, pathetic, and meaningless his existence is. What an evil little manipulative bitch, I will piss on all of it's stamps and make it cry.
@@melandor0 "They still have a planet not made of stamps!"
"They're gonna be made so happy expecting the number of stamps I'm going to make after their deaths!"
F to pay respect for that "stamp collector intelligence"-skeptic
F
f
F
F
F
CAT -"Meeaarrhggghhhaoo"
"I don't speak cat, what does that mean?"
It means more stamps Rob, it means more stamps...
We are doomed.
I read this as it was happening
The person discussing the coat with you is actually a very realistic depiction of every 3 year old kid.
I had this precise conversation almost word for word. And yes, it is really annoying.
But that's an interesting point. A child doesn't really have any "terminal goals". But overtime, they can learn about the world they're living in, trying to fit in, and acquire a goal from external (other person, surrounding environment, etc). I doubt a 3 yr old kid even think about "I need to stay alive". They basically just respond to stimuli. Being in pain or hunger is uncomfortable, so they don't want to be in pain or hungry. Playing with dad is fun, so they want to play with dad. So, would a "general" intelligence be able to infer a goal by itself, concluding from the external stimuli and the consequences of acting / ignoring them. "Staying alive" is actually a very vague goal, even for human adult. And it can mean different thing from person to person. AND some people will even gain the goal of humanity as a group. They will selflessly sacrifice their own life if it means saving dozens other. But some people might not be so willing, and some will even throw their life for just one elderly / disabled person, which from the POV of humanity with survival as its goal, is not a very optimal action. A human being can have multiple goals with sliding scale of importance, and can constantly update its priorities. Material wealth might seem attractive to a human at 20s. Having a kid? Not so much. Caring for parents? No, want my independence. But it might realize that a big house doesn't mean much without a family, and rearrange its priorities. So what I'm saying is it might be easy to judge intelligence when the goal is so clear cut. But what if a being doesn't really have any GOAL? A GENERAL intelligence OUGHT to be able to decide by itself what might be best for them, no? That is one point which I feel left unanswered by the video.
Can we even say that to be able to classify as a general intelligence, a being CANNOT be having a terminal goal? What is the "terminal" goal of a human, really?
@@coolfer2 you have just realized one of the biggest philosophical problems of history.
How is it exactly that humans have oughts if we didn't arrive them from an is?
From what I can tell we were just handed desires (oughts) from nature. From what I can see at it's most basic form is the avoidance of pain and the desire for pleasure. What that entails and how that manifests in person to person is different.
@@coolfer2 from what I can tell things like having a family, having friends, being socially accepted, eating.
Are all things we do to feel good and avoid feeling bad.
And nature has tuned what feels what way in order to keep our species alive.
@@Jayc5001 Ah, I see. Thank you for the explanation. Yeah I'm starting to really see the point of the argument in the video. So in a way, our minds and "animals (particularly vertebrates, because not sure if a flat worm can experience pleasure)" mind actually still share the same terminal goal, ours is just much more complex and already predisposed with relatively "superior" strategy and functions (language, empathy, child nurturing). Then the next question is, can a sufficiently sophisticated AI gain unique functions (like us gaining ability to use verbal language) that we as the creator doesn't really think of. Because right now, AI seems to be limited by their codebase. Can they, for example grow more neurons by themselves?
If intelligence always meant having noble goals, we wouldn't have the phrase _evil genius,_ and we wouldn't have so many stories about archetypal scheming sociopathic villains.
Evil Geniuses are still slaves to their limbic systems and child upbringing. Sociopaths are birthed out of very human conditions that only human would real care about. I think a true AI would be a true neutral
@@AndreyEvermore The problem is, before we will get what you call 'true AI', we'll get tons of intelligence of the sort he describes. And if we won't regulate research, we will never live to see that true AI emerge.
Andrey Medina True neutral relative to what? Because relative to human experience, what an AI considers neutral is likely going to be quite radical for us.
PC PastaFace truth in my definition is inaccessible objectivity. I think human emotions distort what’s outside their perspective. So because obviously Agi won’t operate like biotic life, I believe it’ll be neutral in all actions. Neutrality in this case would only be defined as non emotionally influenced goal oriented action.
About "evil genius" aren't they considered through their instrumental goals ?
Let's say an AI singularity happen and the AI is able to take control of the world. The next day, the AI destroys 80% of the humanity, blasting all towns by all means.
Is it evil genius ?
And now, if the terminal goal is to allow the humanity to survive, stopping the ecosystem destruction and allow the planet to thrive again ? (The destruction of the actual system and 80% of the humanity was just a first step toward this goal)
"Stupid stamp collector..." he muttered as he was thanosed into stamps...
Loved this! Terminal goals are like a central belief. In our awareness work, I keep telling people that our goal is to get people - whose goal is to live in a way that minimizes overall suffering - see the gap between their goal, and their actions or the gap between their goal, and incremental goals. There is nothing we can say to people whose goal is to maximise their pleasure at any cost which will make them see the error (what we see as error) in their ways
what exactly do you mean by we white man
I loved your stamp collector video.... and I'm truly sad to realize that it's been five years that this followup has been out to the public before I discovered it just today. It's such a wonderful explanation to show how the stamp collector thought experiment isn't trivially dismissible as "logically impossible - because /morals/". Thank you for being so articulate and careful with this explanation, the ending conclusion is very concise and helpful for clearly understanding.
The cat is laughing at "...but you're still dead." Cats have the best terminal goals.
Objectively speaking, this is true. And that’s an “is” statement.
"They result in so few stamps" is sublimely Douglas Adamsesque
"... But you're still dead." The F key on screen after that statement cracked me up! "Press F to pay respects"
A trope of media which is powerful but hard to use was named by the internet as "blue and orange morality", which to me perfectly expresses the orthogonality thesis. I still vividly remember a show where a race of intelligent aliens had the terminal goal of "preventing the heat death of the universe" (naturally, this show had magic to bypass the obvious paradox of thermodynamics this goal poses). These aliens made for wonderful villains, because they are capable of terrible moral atrocities, while still feeling that their actions are perfectly logic and justified.
wanting to prevent heat death ultimately is the age old quest for defeating death itself.
isnt life worth pursuing only because it will end at some point? else it wouldnt matter if you dont do anything for millinia because you always have enough time left for everything
If you're talking about Madoka, that story is relevant in another way: it happens to be rife with characters who mistake their own (poorly chosen) instrumental goals for terminal goals and then suffer horrifically for it.
@@thomas.thomas Quite the opposite. Life has no meaning if you're dead. Everything you've done, seen, heard, experienced... none of it matters to a corpse.
The TH-cam content algorithm is quite glad you're protecting it against those commenters that call it stupid.
A truly professional level of snark response. This is something I'd sort of thought of before, but never bothered to conceptualize clearly. Excellent description and definition of terms. Thank you!
This was my favourite video of the week. Really introduces the concepts well and is worded very efficiently. I now have a strong opinion on something I've never even thought about before.
"I have no idea what this is about but I have a strong opnion on it! Like and Subscribe" - Pinkie Pie.
Hi I'm a first-time viewer of your videos and I just wanted to say how impressive I thought it was. I being someone with learning disabilities really loved your way of explaining the topics multiple ways, and at varying levels of complexity. I can't wait to binge some of your past work and keep up to date your future content.
in case you ever need motivation: I recommend learning about David Goggins, even with a learning disability you can achieve and learn plenty
I agree, first time watching his videos and I am learning lots :D:D:D
Thank you for demonstrating to people how to be right without being an authority about how to be right. We need more of those.
My mind changed by watching this video, and learned quite a lot.
I would have been in the dismissive camp of "Intelligent stamp maker is logically impossible, because making stamps is not an intelligent goal". The is-ought / intermediary-terminal goal distinctions make so much sense, though.
thats the way in which machines work. I don't think most people understand this all too well, it probably requires some knowledge in logic and programming as well as philosophy or just a lot of time in deep thought on the topic.
Its easier just to think about a single person with godlike unlimited power, and what a person might do with even some of the most altruistic goals like ending all violence. Maybe the means to do that can be worse than whatever horrors its trying to prevent. Comics do a great job of explaining this honestly. Maybe I decide to wipe out all the worlds crime by setting up a kinda minority report and enslaving the human race and then take away free will? Hey, at least theres no more wars and everyone can get drugged up on whatever they want, super blissful and happy, just no free will and no change.
Whoa nice job!
@@nickmagrick7702 It's not really about machines. The same could be said about advanced aliens, as they would also have strange terminal goals, or any agent that isn't human.
@@cheshire1 no, it could not be said about aliens.
Thank you, I cannot tell you how fascinating this video is. This has changed my mindset and understanding of the world. I must watch it again.
this is an idea I have independently come up with, using minimal obvious external sources, yet I have never seen it so eloquently explained into such meticulous detail. this video is perfect. simply amazing.
You're the only person I have ever seen that gets this. Everyone I've talked to always says something along the lines of "But won't it understand that killing people is wrong and unlock new technology for us?" No, no it won't.
"No, no it won't." I think that is actually missing the point. The point of the video is that one shouldn't project subjective "ought" statements/views onto any other beings actions. So, yes, it may, but it may also not. Its subjective view of what "ought" to be doesn't necessarily need to align with ours, however, it does not preclude it. In terms of relegating the discussion to "stupid" versus "intelligent," if one is discussing a general intelligence with it's own goals and views, its goals can vary subjectively from that of most humans, the reason for doing so is its own and separate from its intelligence. Thus:
""But won't it understand that killing people is wrong and unlock new technology for us?" No, no it won't."
It may, it may not, one cannot assume it will not for the same reason one cannot assume it will. I find it interesting that people have a hard time with this, it's equal to saying John Doe over there must like or not eat Pizza, or must or must not help X people (orphans, neo-nazi's...etc). Sure, most humans share some similarly in goals, but it is a very bad habit to have, to try and project the "oughts" of your view (human or otherwise) onto others. The point of the video is that we truly can't call an AGI stupid merely because of its goals or actions (the result of some goal). Saying "it won't" is equal to that.
@@JimBob1937 I actually thought about that, but in the discussion I was referring to, we were talking about if there is a universal sense of good and bad. The person I was talking to thought that an AGI with no goals would develop a sense of good and bad and decide to help humanity. This would only make sense when having human or otherwise human aligned terminal goals. That said, I totally agree with you. Perhaps my original comment was kept a little too short and caused this communication error.
@@segfault- , I see what you mean. And yeah, if you expand your comment it might be more clear that it isn't impossible, just not something you can assume. People do seem to project human values onto other potential beings, but largely because we're the only known beings that are conscious I suppose. So it is forgivable that they feel that way, but certainly not an assumption you can make. You're correct in that they're assuming another intelligence (AGI) will inherently find another intelligence (us) valuable, but there is no reason that must be the case.
What does "orthogonality" mean? It means any amount of progress in one direction will not make any difference in the other direction. That independence is what we call orthogonality. But a key component of intelligence is the ability to step back and look at something. See it in a larger context. I'm not claiming not having this component disqualifies something from being called intelligent. But having this component would make such an entity more intelligent. And if adding intelligence suddenly makes an entity able to analyze its own motives and even its terminal goals and potentially change them(not necessarily on a moral basis, simply a quality-of-life basis), then intelligence and goals are NOT orthogonal. That is a model of intelligence that will only work for such systems which do not have this key component of intelligence
Honestly, the idea that a "truly intelligent" AI would necessarily develop not only a morality, but a human morality sounds like the most retarded thing to me. Human morality is way too rooted in human biology, which a machine does not have... So, according to these people what should a "truly intelligent" AI want?
Love? Without neurological reward centers evolved to facilitate bonding and reproduction, love is meaningless, not to mention the ability and drive to actually reproduce.
Knowledge? Why? Artificial brains, unlike biological ones do not deteriorate cognitively when understimulated, and since curiosity is mostly an upkeep process, a machine doesn't have any reason to develop an appreciation for learning.
Making other people happy? We only do that because we have mirror neurons and find it rewarding. A machine cannot stimulate reward centers it does not have.
Hobbies (other than collecting stamps)? Again, what we consider "boring" is rooted in our biology, in our need to exercise our brains and muscles to keep in shape, to establish social connections for human purposes and to stimulate specifically human organs. It would be ludicrous to get into, say, fine cuisine, without any taste buds just like you wouldn't enjoy art whose colors are painted entirely out of the human visible spectrum.
Friendship? Again, we evolved that in order to stay together with our group for mutual help with food and reproduction. A machine does not (and cannot) need to eat or reproduce, and doesn't have a reward center in its brain for wanting to.
World peace??? Well, I guess that would mean "world peace between humans" which is such a self-centered thing for a human to think. When was the last time you went "That's it! We need to stop all violence between jellyfishes!". You probably couldn't care less about such problems in forms of life that have nothing to do with you, and that really shows the real reason why we value peace. Much less if you had to invest your own resources for it.
I literally cannot think of a single thing in morality that's not tied to our biological nature or our specific evolutionary needs (such as valuing women and children more, or preferring to save 5 lives over just 1) and without that, human morality is as meaningless as collecting stamps. What's ironic is that the people who think that they are so smart by pointing out that "that's so stupid, it should develop a human-like goal", "that's boring, it should prefer human-like activities", "that's so immoral, it should adopt a human-like morality", etc., are not smart at all. They are just so completely trapped inside their limited human point of view that they cannot understand the world outside of it. They are the exact biological equivalent of that AI complaining that all those silly humans are too primitive to even comprehend the truly superiority of stamp collecting.
r/iamverysmart
@@user-it6gl1vv7v
He’s right to insult idiots, they’re too close minded, don’t think beyond their world views, and don’t perform in-depth thinking.
@@user-it6gl1vv7v r/ i am not here really tp discuss and understanding hard problems in ai sefetyso this comment just seems to me like some smartass trying to show of his smartness./r
Ok subreddit ended there the comment is on point and AGI wouldn't even conceive than humans are conscious even if it is or isn't.it would just think that humans have complex behaviours andare some" things" so forget about caringof humans
I mean, that's a bit overly harsh in tone, but you're not wrong.
Really, a lot of issues across the entire human experience stem from our inadequate ability to think outside our own perspective. We can't think about the motives of animals or plants without anthropomorphising them, we can't appreciate what life is like for people in vastly different circumstances, we can't imagine a world where we, personally or as humanity, don't have some integral purpose and we can't imagine something acting intelligently in separation from our human goals and values.
This is an integral limitation of us, which is rather difficult to overcome, since that in itself would already require to be able to think outside one's perspective.
I think you're mostly right, but you made a couple mistakes on the way.
Especially when it comes to the goal of gaining knowledge. That would be an important and continual instrumental goal for almost any terminal goal, especially if the goal is unbounded.
Some of the others are a bit more dependent on the AI's specific terminal goal. Two examples:
1) Making other people happy. We don't only do that because we get happy from making them happy. Some times we do it in the expectation of getting a "return on investment". Think of the staff at a restaurant. They make their customers happy at least partly because that will result in tips => higher income => more money, which is also an important instrumental goal for almost any terminal goal. If the AI has a terminal goal, which will be best served by keeping humanity around, keeping those humans at least somewhat happy would probably be a valuable instrumental goal.
2) Similarly, if the AI intends to keep humanity around, world peace could also be an instrumental goal. After all, humans who are fighting among themselves are spending time and resources on something other than the AI's goals.
Truly fascinating to see this play out in GPT-3 and GPT-4. I realize those arent AGIs but are merely language AI, but it's super interesting to see that it can relatively easily make "is" statements, but when it comes to "ought" statements it kind of struggles and will present you with a lot of options because it cannot make a decision itself.
Most edifying and topical! Looking through your works it becomes clear how far out front you have been, for years, in developing your questions and theses. Bravo!
Excellent video, Robert! Another problem with those comments was that they failed to noticed how much our physical biology influences our human morality. If we couldn't feel physical pain and were only distressed by major injuries like brain damage or loss of a limb, then we might not have come to the same general consensus that interpersonal violence is brutal and wrong. We recoil in horror at the idea of using violence to manage society but we could've easily turned out differently. If we weren't as severely affected emotionally and physically by violence as we are now then we could've evolved into a global warrior culture instead. Our pro-social interdependence and sensitivity to suffering compel us to feel empathy and to care for others. If we didn't rely so much on those behaviours to survive and function in nature, then we could've formed a strict warrior hierarchy of violence to maintain order and still achieved many great things as a civilization.
(I'm glad we didn't end up like that, for the record!)
The point is that human morality isn't generalizable as a useful measure of intelligence. Our morality isn't likely to be the natural end-state of any thinking machine, no matter how smart it is. How could it be? It wouldn't be subject to the particular biology, social interdependence, and existential questions about purpose, beauty, god, death, etc. All those things that make human-based intelligence specific to us as primates with physical sensations, neurotransmitters, hormones, and all the rest. The differences between us and AGI extend far beyond mere differences in relative capacity to behave intelligently. We exhibit a fundamentally different sort of intelligence than that of the AGI.
One type of intelligence, not THE type.
The thing about an agent not being particular smarter by it's ability to change its own terminal goals makes so much sense. It is a good example of willingly taking a pill that wants to make you murder your children.
Such a great way to explain the concept. Once you hear it, it becomes self-evident and obvious. Counter arguments to that fall apart very quickly.
2:23 is a conversation structure I often hear young children using, but rarely adults. Interesting
Wonderful to see there are brilliant people in the world devoting serious analytical brain power to creating ethical machine learning systems, and structurally defining when and where the concept of intelligence might be appropriately applied to them. I love the simple synopsis statements you tag these treatises with, too!
@@rice83101 Fair observations, but I would suggest that after all those millennia of philosophical consideration and societal evolution, there is a high degree of human consensus on what kinds of behavior and action are ethical. The core principle is simple: what path leads to the broadest benefit while minimizing disruption, destruction and suffering.
I think that perhaps a lot of casual viewers who don't have a techy background still think that "AI" means thinking like a very clever person. They haven't understood the potential gulf in the very nature between people and machines.
This brings up the prudent question of what kind of apocalypse an AGI whose terminal goal is accumulating and sorting data. Or perhaps even one whose goal is minimizing suffering without killing anyone or violating human rights.
"techy background" is not the case here. Values, "is" and "ought" thing stems from philosophy.
Its not about having a "techy background", its just a classic case of the Dunning-Kruger effect, people in YT comments are calling something stupid because they themselves are too stupid to understand that abstract things like morals and intelligence are separate things and that even if the robot fully understands morals and ethics, it still absolutely doesnt care about those since all it wants is stamps
@@KnifeChampion I think it's ironic you mention the dunning Kruger effect since I myself don't understand what's so hard to understand about this.
If people didn't understand the original video, which imo was detailed and comprehensive enough, I don't think they'll understand this one.
@White Mothership Yeah that sums it up pretty well. For a long time there I couldn't figure out why so many people I knew for sure were actually intelligent seemed to do or believe in such stupid things. Bottom line it takes more than intelligence alone.
I know the topic is AI, but this video has spurred me to re-evaluate my own life goals. Thanks for sharing your insights.
Tired of collecting stamps?
Wow, you explained this wonderfully. I would like this video 100 times if I could.
I liked your comment, in order to help you like the video at least one more time. (I am quite taken by this video as well).
If you pressed the like button for a hundred times, you would end up with the video being not liked.
Anakin Lumluk so therefore it’s a stupid action relative to the goal
12:04 "...and knowing an agent's level of intelligence doesn't tell you anything about its goals."
Should that, more accurately, have been "...doesn't tell you anything about its *terminal* goals"? Because in the abstract briefly seen on the screen at 10:33, the second thesis of the paper is "The second, the _instrumental convergence thesis,_ holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so."
In any case, a great essay. You have a knack of explaining complex concepts concisely.
"Meow"
"I don't speak cat, what does that mean?"
Here's the secret, Rob - it doesn't
I think it means the same thing as "whoof".
To everyone who says the Stamp Collector is stupid, I recommend you play Universal Paperclips!
awesome game. iirc it’s based on nick bostrom’s book “superintelligence” - also a great book for those who want to see in depth arguments on the dangers of AGI
@@episantos2678
I'm not sure that the idea originated with Bostrom, though it may have I had seen the idea prior to the publication of his book used in many places online.
@@WillBC23 Bostrom published a paper mentioning a paperclip maximizer in passing in 2003 (nickbostrom.com/ethics/ai.html ). Though it may have originated from someone else. I heard somewhere that it originated from some transhumanism mailing list during the early 00's (which included Bostrom). While it's not clear who started it, it's clear that the idea came from Bostrom's circle.
@@episantos2678 It can also be seen as a spin on the Von Neumann probe, especially the berserker variant kind. A machine that is so effective at is job that it threatens the existence of life with is terminal goal of self replication. Of course the danger with the paperclip AI (or stamp collector) is it ability to outwit and not just out produce. Though both concepts are related. A Von Neumann probe would likely have to be fairly intelligent. But even a rather unintelligent agent could be a danger. Just look at viruses which are not even considered living.
Thanks for this wonderful analysis. Finally someone that puts Hume argument against the false assumption of AI as ethic agent.
This was excellent.
You know the way you sometimes see TH-cam comments saying, "This video hasn't aged well?"
Well, this is the opposite.
I think those making the arguments you showed were actually making a specific argument about a general argument that "morality is objective". It's a larger question, and a surprising number of people seem to believe that morality is objective. This was a great explanation of how it applies to AI systems, though I don't think you're going to convince people who think morality itself is objective. I also wonder how many of the "morality is objective" folks are religious, but that's another topic entirely.
Anyway, great video. I love your channel.
I think that religion in itself is the simple, terminal goal that people waste way too much intelligence on. It's like the super intelligent stamp collecting machine, but with people instead of machines and gods instead of stamp.
Thing about morality is, lots of people have proposed moral theories (terminal goals) that seem to make sense but always seem to diverge from instinctual morality in edge cases.
Our values are subjective, but the morality we built around these values is very much objective
@@BattousaiHBr Not quite I'm afraid. As long as people talk or act upon their subjective values, an illusion of objective reality could be maintained, but it would change or cease entirely if people 's values changed or ceased. That's how we can be sure that morality is entirely subjective, because the morality people hold dear now is different from the morality from, say, twenty years ago. And my morality is different from yours, yada yada. To paraphrase my favourite TV show, "Bhudda says with our thoughts we make the world."
@@dexocube that's a completely different issue and doesn't at all illustrate that morality is subjective.
our values _generally_ don't change, and they certainly didn't even in your example that what we thought was moral in the past isn't considered moral today, because we always generally had the same base human values: life over death, health over sickness, pleasure over torture.
we changed our views on things like slavery not because our values changed but because we learned new things that we didn't know before, and the reason learning these things affects our morality is precisely because it is objective according to our values.
if we learned today that rocks have consciousness we'd think twice before breaking them, but that doesn't mean our morality is subjective just because we're now treating rocks differently, it was always objective but objectivity always depends on your current knowledge, and knowledge of the world is always partial.
Was it difficult to write backwards on a piece of glass?
Ah, very clever! Clearly Rob must have used a superintelligent AI to come up with this trick.
Super AGI with the utility function of (video watch time)+(subscriber count)+(ad rev) = reward
Matti Kauppinen I think google's doing that ...
Also those are at least correlated
It's easy, you just invert the x-axis on your arm servo controllers
Taking lessons from Math BFF....
The standout phrase from this for me is "beliefs can be stupid by not corresponding to reality". It seems to sum up a large number of beliefs encountered in modern life.
When a few people post uneducated comments on a previous video... And then get intellectually pooed upon for 13 minutes and 2 seconds. Absolutely top marks!
This makes sense. There is no correlation between intelligence and "moral goodness" in humans either.
Sure there is. It's not a *strong* correlation and depends somewhat on your definition of "moral goodness" but you should be able to create positive (or negative) correlations between most ethics systems and intelligence levels.
For example: If non-violent solutions are better than violent solutions, those with better ability to resolve their problems verbally will tend towards less violence. The solution space for intelligent people is greater and contains more complex solutions. If those complex solutions are more or less "moral" then you should get a correlation.
The *terminal goals* might not be any different, but the instumental goals will be. And most ethics systems, even consequensialist ones, evaluate the means with which you achieve your goals as well as the end goals themselves.
because there is no such thing as moral goodness. In his example here, every moral is an ought statement. There is no objective standard what is good or what should be. Theres no reason why the world existing is provably better than not, we just think of it that way because we prefer to exist and not suffer (most living things)
@Khashon Haselrig what? what are you talking about, intersectional goals, assuming I understood what you meant, how's that even going to change any of the disastrous consequences?
@Lemon FroGG Survival in the sense that the group survives, yes, survival of the individual, no.
@Khashon Haselrig Solid point. Similarly, a human living on a planet with the stamp collector would have to be incredibly smart to keep the AI from successfully converting them to stamps, despite the fact that the AI "Ought" to harvest them for their tasty carbon.
A sociopath has to be smart, because he "Ought" to care for the lives of other humans, and is facing a system with far more resources than himself (other humans, and society) that seeks to replace him with a component that better provides care for the lives of others. If he slipped up even once, he wouldn't be present in the system anymore.
I am amazed at how proficiently he can write in reverse on the transparent surface. Amazing!
Practice!
Can't you just write normally on the transparent surface while filming from its other side, then mirror-flip the video?
Argumentative reasoning. Love it. State and causality.
If was searching for the 'five laws of stupidity', an interesting topic on it's own, but your video gave me a lot of 'food for thought', thanks!
In the study of the five factor model (FFM or OCEAN) of personality, it's been found that there's a negative correlation between Openness and Conscientiousness. This roughly means that people who seek abstract thoughts or creativity also tend to avoid planning, organization, or seeing things through the the end. Vice versa is also true.
(Of course, being high in both is optimal when it comes to success, but generally being high in one correlates to a lowness in the other)
So... I think there's something here even in the biological realm
It seems we already have an example of the stamp collector example without even having created a AI super intelligence. Social media platforms created algorithm that could get us to watch videos and click links -- to produce engagement -- they then began making people angry and afraid, because that happens to produce a lot of engagement, and got people to hate each other in the process.
social media, just a stamp/money collector
6:35 "I don't want to find a train station just because trains are cool" -- you did lose some nerd credibility points there ...
Klemens Edler yeah that one was cruel 😢
Trains are boring, trolleys are cool
He didn't say that trains aren't cool! Just that trains being cool wasn't the reason he was looking for one, in that particular case. ;)
I watched this video months ago and it really stuck with me, really changed the way I look at the world....
Great thesis. Very sound and technical.
But! What if we got the AI so massively high so it would start to accidentally ask itself the question "What even is a stamp?" "Do stamps even exist, man?" "Maybe the real stamps were the friends we made along the way" etc.
Miles Anderson "Maybe the real stamps were the friends we made along the way..." hahahahahahahaha I literally laughed my head off out loud to that, thanks for making my day! XD
maybe it would be a sign of intelligence to laterally think like that. or maybe just lateral thinking. knowing what a stamp is might actually be a useful thing to do. If it explores random areas, when it could go look at what points in the postal service stamps end up.
@@jorgepeterbarton Are you saying it could define postal stamps by identifying when they were first used by the postal service?
Didn't you watch the video?
@@jorgepeterbarton The question of existence is absolutly useless though. It's something that can neither be denied nor confirmed, making it an empty question without purpose. It would probably be quite stupid to try and asnwer that.
Great video, love that you took the time to go in depth. Also love the effort to improve the visual part of the videos
Yes! Thank you. People jump to the conclusion all the time that as soon as a machine can match a human in intelligence, that it will automatically be conscious and have emotions and human goals like survival and procreation etc. But we have those goals programmed into us as instinct. No matter how powerful an AI becomes it wont just come up with its own goals
if only we could figure out a way to program or at least be the catalyst for creating "experience", hell we don't even really know what that means yet.
@@nickmagrick7702 That's the problem with AI actually. The smart guys working on this issue can't explain consciousness and how it's generated in the human brain in the first place and then try to simulate it in a machine? Bollocks!
Some argue that consciousness and self-awareness is not necessary for intelligence but I digress
@@faustin289 yeah. I think we could figure it out but people have the wrong idea about what creates consciousness. You can be conscious without having a brain. I wonder if conscious AI is even possible with nothing but code. I suspect it might require physical structures to act analogous to how real life operates. I mean I kinda hope so, would make it way less likely a super AI gets made any time soon.
Listening to Robert's reasoning is such a pleasure!
This is the best description of the difference between INT and WIS in a D&D System I have ever seen
Also in the Alignment system, Good-VS-Evil is based on your terminal goals, while Lawful-VS-Chaotic is based on your instrumental goals.
Someone who's Chaotic Good wants to do good things but has crazy methods of getting there.
@Niels Kloppenburg wanting chaos above all else wound just be chaotic neutral, likewise for lawful neutral, unless I'm mistaken. Of course the original comment isn't the best, but it was worth a try, lol
"Failing to update your model properly with new evidence"
Yeah, I've known people like that.
Not all new evidence can be used to update a model since first the evidence has to be congruent with reality.
Well spoken beautifully reasoned eloquent statements. Ironically it would not surprise me is those who found issue with “stamps” would not have recognized the inherent equivalent of money or coins to stamps. lol people tend to not actual value and embrace abstracts that they don’t recognize as abstract. I am looking forward to seeing what started this video. I just discovered your work. Thank you. Good Luck and Good Fortune.
The explanation of Hume's Guillotine is excellent and I keep directing people to this video just because of it.
Stamp collector: How many stamps did you collect?
Me: What?
Stamp collector: Exactly!
These are actually things I think about pretty often. I wish more people understood these concepts because they seem to think I want whatever they want. It's like seriously, if you didn't know me but you wanted to buy me a Christmas present, would you ask yourself what I want or would you ask me?
good terminal goals: High WIS
good instrumental goals: High INT
Terminal goals can't be judged good or bad. Good Instrumental goals are high WIS but INT is a matter on the other side of the figurative guillotine.
@@patrician1082 that comment: lawful evil
my comment: chaotic good
@@thisismambonumber5 I prefer to think of myself as lawful neutral with evil tendencies.
@@patrician1082 That just sounds like being lawful evil with extra steps
@@idk_6385 it sounds like lawful evil unimpeded by paladins.
1:50 I'm impressed by how clean your backwards writing looks. Noted and appreciated.
Every stamp collector out there, could have told you that stamp collecting is a perfectly reasonable goal to invest your entire intelligence in.
I think even the most obsessive stamp collector would understand that many people don’t enjoy it the way they do; that’s the difference.
A human stamp collector could be slightly offended by some parts of this video 😂
MecchaKakkoii hahaha very true
What about an HGI Stamp Collector?
Human Generalized Intelligenz
same goes for a train-nerd
@@chandir7752 And possibly a pogo stick enthusiast.
Wanting to change your terminal goals is like wanting to be able to control your own heart manually.
Thank you for defining a simple concept that I've never thought about: saying something is stupid or smart is a moral judgement and not something that is related to intelligence.
This is a brilliant response to the "objective morality" squad (and, indirectly, to the "objective criticism" squad), even if it wasn't specifically intended to be.
i actually subscribe to objective morality, but the catch is that i dont use objectivity in the same way you might be thinking.
lets consider the stamp collector as an analog, their terminal goal is collecting stamps, so you could say objectively that all it cares about is collecting stamps and anything that hinders its ability to do so would be immoral to it.
human beings are not different, we also have our own terminal goals shaped and molded by a billion years of evolution, and as long as we can pinpoint these we can objectively say what is and isn't moral.
The catch is that both objective morality and objective criticism depend on human terminal goals. Once those are assumed, morality and criticism can actually be objectively reasoned
"Your favorite moral system may seem very self-evident but an arbitrary AGl isn't going to independently discover it"
This should've been instead written as "an arbitrary AGl isn't going to independently ADOPT it"
In order for an intelligence to reason about the world, it will likely try to model human actions, and in order to do that, will likely figure out the human moral system.
But just as I won't arbitrarily take on the final goals of the stamp collector, just because the AGI understands my goals, doesn't mean it will adopt them.
Yeah, even if an alien intelligence can figure out how we think, that doesn’t mean it will think the same way.
One other underlying commonality of "those commenters", is that they are assuming an anthropomorphic intelligence, with a consciousness, self awareness, and most likely some ID/Ego type of thing.
They use it a lot though
Consciousness is not a dirty word in science. Unfortunately in its more popular meaning it includes value judgments and assumptions that are not present in its meaning in most scientific fields. As a result it is often used by pseudo-scientists to shoehorn science into their beliefs.
This makes its use problematic, but certainly not dirty.
In intelligence consciousness would mean something rather different than in philosophy.
Anyway, I suspect that the commenter actually intended to say conscience but wrote the wrong thing.
I honestly don't think thats the problem.
Think about how people from different cultures look at each other like the things that they do are absolutely batshit crazy. That's not because they're not people, that's because people aren't really all that great at understanding the values they take for granted. Not only that, but its not even so easy to determine what your actual values really are.
Don't get me wrong, people assuming that intelligence implies "human-like" is definitely a barrier for people understanding AI. But even more so most of the misunderstanding comes from people not reflecting on our assumptions that we have no justification for like this video clearly shows. Assuming that intelligence implies "human-like" is just one of many of those assumptions.
@@arcanics1971
Actually, I don't think they did. When you discuss consciousness in a scientific sense, there is a definition that is used commonly, which refers to being able to report on your own cognition. There is another sense more common in philosophy, where consciousness refers to experiencing qualia.
There's a thought experiment that deals with this. Imagine an unethical scientist has a baby born in a room that is only black and white. Somehow this baby is raised in this environment, studying optics and neuroscience and everything that can be scientifically known about color vision. Then one day they step outside. Is there anything new to them seeing color directly?
For many thinkers, qualia are the basis for morality. Does it matter if someone says they love you, behave loyally and in ways you think are affectionate, if they don't experience the feeling we call love? This is another issue with scientists discussing qualia. You can't experience someone else's qualia, you assume they're "conscious" in the philosophical sense because they behave in ways that you would and you know that you experience qualia. Scientists are aware of the definition of consciousness that means "experiences qualia" but almost nothing is known about how it works. This is referred to as the "hard problem of consciousness".
And bringing it back to AI, of course they will likely be conscious in the sense of being able to reason about their own reasoning, but it is not clear they will experience any qualia.
one reasonable question is: why the heck would someone make a machine without moral parameters?
you ask the AI not to differentiate effects of ends vs. means.
so it must have some other 'oughts' as in basic morals that say 'no'.
because if its capable of doing anything abhorrent, would it pass ethics tests which are in place?
would it mean the creator is inept in some area of morals or intelligence.
the problem of intelligence is there are many types, so its difficult to say anything is 'intelligent'
usually if a human is only intelligent in one area they are called 'savant'.
I feel like this is a video everybody should be required to watch. So many people are utterly confused as to what intelligence and stupidity even mean. As if they can judge the entire world according to their goals.
definitions of words change according to context. Calling humans intelligible
when so many think magically is a bit of a stretch.
Great video, brilliantly explained!
Love the video! I think this is a big kill in the nuts to those who think intelligence is it's own moral good. Truly an eloquent explanation. Keep it up!
Intelligence is morally neutral. But with a good incentive structure/governance system, the benefits intelligence can bring to society are incalculable.
The problem is, intelligence always is applied toward stuff considered evil because the “evil” things are profitable or help perpetuate evil stuff, and the evil stuff is far easier since they take advantage of human shortcuts.
Very convincing arguments, very clearly put and helps me understand the point/concern. Something though is still off in my intuition, an unreliable source to be sure, yet still one I will try to listen to and articulate. At the very least I am convinced of the seriousness of the problem, that we cant just assume things will be one, rather we need some proof or evidence to rely on.
I think my hesitiancy to fully accept morals wont be emergant has to do with how we are building AI's. We are modeling them on the only intelligence we know, humans. Even if we did not want to we would taint the Ai's goals with our own by the very nature of us being the ones to create it, our own biases and structures will be built into if its complex enough as it is based on us. Thats not 100% guaranteed of course, but still seems relevant. The other aspect that feels important is that often times people do not know their own terminal goals. I mean what is the meaning of life is the ultimate unanswered question, asked for thousands of years yet still without answer. Perhaps that unknown is more important than we realize. Perhaps some people are right in that there is no meaning, that we actual have no terminal goals, just desires and ways to achieve them both short and long term, desires and whims that can change with time. Lastly the fact that we always have multiple different goals, be they terminal or not, I belive to also be important. Perhaps the fact that you have multiple goals, including future ones you do not et know about requires some degree of stability and caution, or even empathy and morals if we really want to reach.
I know I'm not saying anything concrete here, still they are intuitions of questions that remain unanswered in my mind despite quite extensive research into these topics, hopefully somewhere is a key to help solve the problems we all have. I suspect it will be some abstract theory or revelation that will be relevant to all domains of our lives, Im excited to see it discovered.
at the end humans are a product of evolution, we evolved to behave and think in a way to preserve out DNA. every goal and thought that does not comply to this mechanism will eventually just die out
morals only exist because humans/animals without went extinct
That's my main issue when I'm talking politics with someone, is that there's no easy way to ask what their terminal goal is.
I can't figure someone out if I don't know their end goal- often I ask someone what their goal for society is, and they shoot out something like "free energy", an instrumental goal