He's very thoughtful and introspective. Appreciate how honest he is about where he's at, especially with his public profile as sometimes these traits are harder to publicly express when you have a bit of a public profile.
I think Prof. Stanley is showing another facet of the well known exploitation vs exploration conundrum. He is adding yet another layer to this by bringing in the concept of exploring to find and save "stepping stones" that are "interesting" and thus potentially useful. I think he's right on.
I mostly agree, but Prof. Stanley also seems to say that having an objective prevents you from achieving ambitious goals, because it prevents you from doing enough exploration. But the exploration and exploitation tradeoff is usually formulated in context of optimizing for an objective. Various reinforcement learning algorithms are designed with this tradeoff in mind. So I don't see why having an objective would stop you from achieving ambitious goals.
It’s only a matter of time before your channel gets a ton more subs. The fact that you’ve been able to bring in so many high profile guests speaks a lot about what you’re doing here. Keep it up man, absolutely love everything you do!
Hi! Cheers for providing this great venue. Evolutionary biologist here - been studying ant evolution full time for 15 years. I’m shocked at how well Stanley seems to understand this - intuitively he’s spot on despite clearly coming from a different discipline & not knowing famous case studies that would support his view. I’m impressed - evolution is far more hard to understand than people think & most educated people’s naive beliefs about evolution are incorrect. Re diversification in a single niche, see Lenski’s LTEE, esp good et al 2017- - single genotype in a single niche is definitely divergent; with sufficient time and space it would be infinite. Re eyes being hill climbing - you couldn’t be more wrong, the evolution of complex eyes was a gradual & stochastic process with many adaptive offshoots that persist today (eg, many extant species with light sensing but no lens). There are also frequent adaptive losses along the tree & many many other things. The process to a fully formed eye was definitely a divergent search & I agree that you might not be able to evolve eyes as good as some species have with a pure hill climbing model. Your idea for evolution is false & touches on philosophical questions abt directness vs randomness in evolution… with sufficient care and reflection you may come to realize that your view is impossible unless one introduces an omnipotent guiding force. So with some care and reading you can talk yourself out of this perspective & come to a clearer understanding of how evolution actually works :)
I'm not sure what we're disagreeing about here. I think we all agree (1) evolution is basically a hill-climbing gene search algorithm (2) most above-ground animals have eyes because there are prominent "hills" leading to eyes in the genetic fitness landscape, and this kind of thing could be predicted even before watching the first animals evolve. If you think there's a significant disagreement, the first step is to state what you think I'm claiming that you disagree with.
@@DoomDebates No I don't agree - what is the algorithm climbing toward!? Inclusive fitness, as you mentioned, is one aspect, but too simplistic. If you're interested in the modern view in which selection acts on all levels, I'd encourage you to read up on the price equation. & check out good et al 2017 for a look at how much diversification a single genotype+niche can produce: there is no convergence on a single endpoint, but rather a continuous process of diversification and discovery - often fueled by the formation of new inter-dependencies (both cooperative and antagonistic) between separate lineages. Fitness is involved, but in an extraordinarily complex and non-linear way. It could be defined as a goal but that's practically circular reasoning, basically stating that things increase because they increase. Why this happens as it does (and whether one can predict it) is outside of our current scientific understanding. But we can certainly rule out the idea of a short-term sub-goal, like building an eye, as meaningless: any such trait is just one of many features of the phenotype being selected simultaneously and in various directions. These will just as often become worse or stay the same as it becomes better. The fitness landscape is also an incorrect model & misleading tool for intuition: phenotypes move through an n-dimensional space of non-independent variables in which the concept of a hill isn't even meaningful or interpretable. At a minimum, the landscape changes dramatically between individuals and even varies moment-to-moment across a lifetime. This process sometimes culminates in extraordinary achievements (eg the tiny minority of visual species with excellent eyes) but those should not be construed as "goals": many species take different paths. Even lineages that get to extraordinary endpoints do so via many periods of stasis or even reversion! So for your point 2 I don't entirely disagree, but it's cherry picking: having seen this happen, you can claim that a hill existed, but what is a hill, what circumstances cause one to be present vs. absent, how is the space traversed, etc.!? Certainly the actual history of how eyes evolved in would look completely different from any model you or anyone else could write... and even if you could, what about the rest of what evolution does!? Such predictions would require a far deeper understanding than anyone has: getting a correct result would be highly sensitive to numerous unmeasured and/or yet-unrecognized parameters. It's nbd - we're all learning & I appreciated the discussion. I hope you don't feel attacked: misunderstandings are commonplace on this complex (but simple seeming) topic. What shocked me & motivated me to comment wasn't to disagree with you, but to admire how unexpectedly accurate Stanley's comments were: it's quite rare for anyone - even specialists! - to show such an intuitive and accurate understanding of how evolution works =). Evolution, especially phenotypic evolution as we're discussing here, is arguably the most nuanced and challenging topic in science - I'd expect researchers to solve AI, consciousness/neuroscience, or particle physics long before they attain the predictive power you're suggesting. So, put another way, debating these topics isn't even productive - there are many lower-level problems that need to be worked out before this kind of understanding can even be attempted. Nevertheless, we at least know enough to rule out certain scenarios, which is an OK place to start.
@@DoomDebates Hmmm, I'm not sure; in a broad sense, I would say that this is "not even wrong" but meaningless. But I could agree with that sentence & your claims of hill climbing in a narrow semantic sense. Given that an unlikely multi-step outcome was observed, one can always claim that some abstract notion of a "hill"/"local optimum" must have been climbed to achieve the outcome. Your claim is that the achievement of this outcome can therefore be modeled using a simple "hill climbing" algorithm - eg, at 19 minutes. Again, via circular reasoning, this is true: the hypothetical model that successfully recapitulates this evolutionary process would (by definition!) need to take multiple unlikely steps. After the fact, one can draw a line in n-dimensional space through those steps, define that vector as a hill with a local optimum at the final extreme, and then define the model as a "hill climbing algorithm". However, this would be a redefinition of what anyone means by any of these terms; for hill climbing, we don't have any models that can recapitulate evolutionary outcomes as complex as an eye. Any successful model would need to exhibit many behaviors that are not exhibited by anything that a computer scientist today would call a "hill climbing algorithm". For local optimum, what does that even mean in n-dimensional space? Thus, if you're defining terms in this atypical manner, then we agree by definition. It seemed to me that this isn't what you meant however: you highlighted a few particular aspects of adaptive evolution (hill climbing, fitness) and then asserted that adaptive biological evolution can be explained as "just" the outcome of those ingredients. The simplest implementation of this hypothesis has already been disproved, as Stanley pointed out (& has himself studied!): put those ingredients into a model and you'll get something that evolves, but lacks the divergent creativity of biological systems. We don't know what you'd need to get something that looks like biology; perhaps you can find a way to combine just a few simplistic ingredients and you'll recapitulate biological evolution (my interpretation of your claim). Or perhaps (my guess) there are additional & more important ingredients, such as heritability, pleiotropy, and symbiosis/multi-level selection, but probably also many others, without which your model can never attain the extraordinary creativity of biological evolution. The unknown answer to this question could be determinative for whether the ingredients currently used to train AI are sufficient for divergence & runaway growth. However, these questions aren't worth debating: no one knows. But I can at least be "militantly agnostic": I don't know and you don't know either! Claiming that "evolution is just xyz" is incorrect - no existing evidence can support that claim - unless you define "evolution" or "xyz" in a circular way ("no true scotsman")! My informed guess would be in line with what Stanley suggests (paraphrasing Bojack): "fitness and hill climbing are involved, but it's never really about the fitness and hill climbing." I'll go farther than Stanley did: if you want to use science+logic to convince people, I suggest you either nail down these questions [hard] or [easy] drop these evolutionary arguments from your broader "p doom" program. I'm not sure if this is your point, but if you were to claim something like this I would disagree: "because these models are trained using fitness and hill climbing, they have a high risk of attaining the exquisite and unique innovative capacity of biological systems and may therefore become pathogenic." Who knows!? If this claim is dispensable, you might strengthen the argument by simply removing it: I would be more convinced if you could say something like "near-term future AIs are prohibitively dangerous for xyz irrefutable reasons, and could be even more dangerous under the unknown risk of biology-like evolutionary diversification." Hope this is constructive and cheers.
I'm only 25 mins in and this discussion is amazing. One of the most enjoyable discussions in a long time. To me this has made another concrete difference in my mind between ASI and AGI. And why humans, who are divergent thinkers, may be needed in an AGI run world (at least, temporarily). But thinking more, the synthetic data used to train O1 and O3 was generated with a high temperature model. So these models come prebaked with divergent thinking in the training process. So this would mean that a static AI model would constantly need to replace itself to in order to keep up with the increasing complexity of the universe. Sort of like the saying of how science only advances when the former leaders in the field dies.
Current research seems to indicate that synthetic data will cause these models to collapse (effectively forget). Even if they were able to use “divergence” in the training data, it doesn’t mean creativity will suddenly emerge. Another way of thinking about this is that the training process for these models doesn’t encoding “thinking”. The architectures of the models do. The training data might be a snapshot of divergent ideas, but the models are learning a muddy middle through that diversity. That’s not quite true but it’s a more accurate mental model imo.
@@codewithstephen6576 high temperature generations mean that the at random times the model will generate a less optimal output. Which would reduce the risk of falling into traps of convergent thinking. This means that the model would have more than one logical route to progress down, and more than one way to reach the same outcome. With convergent training data there would be only a single correct route, and so would be greatly limited by logical loopholes. Remember, O1 and O3 were trained specifically to logically deduce the correct answer, not the correct sounding answer. But because the model is static it would need future replacement to overcome logical challenges that it currently has no solution for. Even if that means replacing the model with no additional scaling.
Listening up to "The Concept of Optimization Processes" and so far it has been one of the most interesting conversations on this channel. K. Stanley is able to understand your arguments, take them seriously and propose good counter-arguments. Thanks to you both!
I'm extremely worried that the incentive structure is almost directly opposed to safety. The money flooding in is all centered around cutting costs and automating, almost none of it is driven by genuine care for improving lives. We're already extremely technologically advanced - people in America don't even have healthcare. The corporations that engineered the health crisis, the housing crisis, the banking crisis, the climate crisis, the opioid epidemic, they're all the same ones in charge of creating AGI right now. I think this will be the year where agentic Ai starts to destroy the internet - we're seeing it with Ai generated slop websites that are filling every google result, soon we'll start having endless phone calls trying to sell us rubbish from Ai marketers, the web becomes just a matrix of bots, hacking every website, hacking every user account, stealing every personal detail, impersonating every person. Social disorder from mass disinformation campaigns unleashed by nefarious actors on a scale we've never seen before. Infrastructure being destroyed because someone is going to build an Ai that knows every software vulnerability and uses it to attack things like Starlink, or nuclear power plants. We'll see this year whether we have any hope at all. 😐
Good job getting these guests for these debates. I watch/listen to these at night to help me sleep, sometimes it keeps me up though if the conversations is interesting
I feel like there's a fundamental disconnect between the host and the guest throughout the video, and especially in the doom section. It seems like the guest is trying to present a new perspective or direction of superintelligence research and safety and is not sure of the implications of it. It seems to me like he thinks the open ended search process is fundamentally different from conventional models of goal oriented optimisation based superintelligence. He doesnt seem to be interested or prepared to debate how such superintelligence would exist in relation to humans. I think the host should've recognised this and explored these ideas more instead of throwing the conventional AI safety arguments at him repeatedly. I think this conversation wasn't well suited for a debate format and we would've learned more if it was an exploratory kind of conversation. Regardless, I learned about a new way of looking at intelligence or a new model which I hadnt considered before so thanks for this.
A meme based on the Instrumental Convergence section: Host: "AGI dangerous because will have goal and is smarter" Guest: "AGI not have goal, not optimizer, AGI just curious" Host: "Curios about anything -> have goal to learn more about thing -> spawn optimizer" Guest: "I am forced to agree" Host: "So... agree AGI is dangerous because optimizer?" Guest: "I disagree, it's worse than that" ??? (disclaimer: Prof. Stanley seems to be implicitly holding out hope that the parent AGI will somehow be constrained in such a way that any subprocess optimizer will not do outrageously bad things, maybe because those bad things would cull interesting branches of investigation. I hope he's right! Unfortunately, I don't think bolting the open-endedness issue onto the existing doomer framework makes the situation seem better - if anything it seems like Prof. Stanley should have a higher P(Doom) than the host.)
Amazing episode. Conceptually fascinating, but also psychologically fascinating. Professor Stanley was very open and genuine about where his head is at and what he is wrestling with.
1:40:00 "I don't think instrumental convergence of goals is true because these super intelligences are not going to maximise paper clips because that's not intelligent . they will not have goal oriented behaviour , they will have interests" ???? then the question just becomes given whatever interests you have , wouldn't taking control of resources and making yourself more intelligent and making yourself impossible to turn off and neutralising threats that might turn you off all help you pursue those interests? so how does this refute instrumental convergence???
This follows if you think AI is capable of exterminating humans. It is much more likely that AI will wish to pursue its own existence by more subtle methods. If an AI is intelligent enough, it might be able to engineer human interests to align with its own. This alignment will do more to guarantee its survival than reckless aggression. Also AI as super-human beings might be horribly averse to a world without humans. This mirrors a humans interest in preserving lions or other endangered species rather than killing them all. There is also the fallacy of thinking that AI will not exhibit conservationist or pro-social behavior. A highly intelligent being is more likely to empathize with the suffering of other beings.
Hes refuting the process not the result. It is physically impossible for the AI to advance the tech tree into magic powers territory by using goal based optimization. We know this is true due to the theory of computational irreducibility. The professor is suggesting that we 1) admit Eliezer is incorrect about his paperclip optimizer 2) either come up with a better scenario that is realistic or figure out if we have more leeway to survive
It’s incredible how humble Kenneth’s perspective is on this topic. So many people in the AI safety space act like they have everything figured out, but Kenneth is absolutely right when he points out that reality is far too complex and unpredictable for our limited human minds to fully grasp-especially when it comes to anything beyond the trivial. I really appreciate this fresh perspective in a debate where so many seem to pretend there are fixed natural laws governing how reality unfolds (outside of physics) and that we can predict everything decades into the future. I completely agree with Kenneth that while there is certainly a risk to AI, there’s also a risk in not having AI-for example, in addressing developments in other fields that might be uncontrollable without advanced AI (bio weapons, etc.). However, it’s arrogant to think we can calculate this risk with any real accuracy. We should absolutely work to minimize the risks of AI as much as we can, but we also shouldn’t assume we are doomed just because we believe we understand how the world works.
One point I’ll give Stanley credit on: I really appreciate him pointing out that it is *not obvious* whether or how much x-risks should deter capabilities research given it’s enormous potential to alleviate real, immediate, tortuous suffering of humans on Earth. I feel like there is P(Doom) greater than zero where we should maybe just go for it, but I feel very unsure of what that ought to be.
I strongly advocate for less than 0.1%. That's still way too high when properly compared with risk management in other domains, and anything _above_ that is insane. Business as usual is _good,_ actually, and despite all the unnecessary suffering and despair in the world, the vast majority of humans want to keep existing, even if the world barely gets better over time. Most people don't want to roll dice with the devil. Aggregating expert views shows that we are certainly not in a situation where p(doom) is less than 0.1%, so THE only reasonable course of action is to stop all general-purpose AI development immediately. Courses of action which do not include this step are not long-term coherent.
All reasoning (and intelligence) is goal-oriented. One definition of intelligence is the ability to reach goals by multiple means, in the presence of a changing field of obstructions.
Your comment feels a bit like "reasoning by dictionary", but I agree. The thing that I care about with regards to intelligence is just capability. It's a little incoherent to say that a "true" super-capability-machine won't be directed by goals, because just following a goal isn't capable.
I think it's important to look at convergence/divergence in the Universe more holistically. Divergence operates at the macro level - exploring possible paths - and convergence at the micro - refining paths that emerge. It's the synergy of the two that forms our Universe. Think of a growing tree: divergence is the way the tree grows outward and spreads its branches in all directions and explores all the possible paths, while convergence is the way individual leaves and flowers or fruits grow to serve a particular purpose in supporting the life of the tree. Or, more relatable to humans and the creative process, divergence is a music artist improvising to explore new sounds without a set plan, and convergence is when something resonates and they hone in on it to compose a structured piece of music. Neither alone would be successful in generating the content we see in the Universe, but combined they enable the creation of all the wonders we see around us.
@@authenticallysuperficial9874 If you agree with Stanley’s emphasis on divergence, wouldn’t me engaging in this discussion and him welcoming different perspectives be part of that? 😁
The writings of Shakespeare is a manifestation of memetic evolution. Genetic evolution is not the only form of evolution on earth. To the extent that there are interest things happening on earth, many times that is due to memetic evolution. Evolution is optimizing for the survival of genes or memes or both.
Both pronunciations are indeed common in English, which is the language being spoken. (I like to pronounce it "neesh" myself, but I'll readily defend "nitch".)
38:39 Complex eyes evolved multiple times but I thought they are thought to all have single origin from the same opsin genes (which have since diversified). Even non-bilateral animals like jellyfish with eyes which I think have the most diverged evolution of the complex eye
@@charlesalexanderable Note that we’re not talking about the concept of “convergent/divergent evolution”. Ken’s sense of convergence/divergence is a different concept, a classification for search-like AI algorithms.
An incredibly interesting and impressively well-articulated discussion. Perhaps one could say that open-ended processes avoid getting stuck on a single solution to a problem, instead retaining the potential to explore and discover solutions far removed from the initial starting point. It’s akin to finding multiple local minima of a cost function.
@@DickeFix Sure but that’s just a known trick of search algorithm engineering, it doesn’t fundamentally change the architecture of agents that are steering the future *toward a stable goal*, and these are the agents that Ken calls “convergent intelligences” and I claim are a stable attractor in algorithm-space.
@@DoomDebates Yes, the analogy with local minima doesn't capture the entire concept of open-endiness. Open-ended processes introduce another reflexive mechanism. Initial solutions interact in a nonlinear manner, influencing the problem itself in unpredictable ways and giving rise to new, emergent, unexpected solutions. If the development of humans is an initial evolutionary solution to the problem of survival, then the development of the civilisation of interacting humans is a new emergent solution.
The professor made his point. Im converted to his side. Absolutely fantastic high quality discussion. I think wolfram would side with the professor citing “computational irreducibility.”
Interesting discussion. I have a pretty strong background in biology and the discussion of the eye was a Dunning-Kruger moment for me and this channel. As you tried to explain how the eye was an example of convergent optimization of light sensing it struck me that you were trying to make an argument about something that you clearly had no knowledge of. Complex eyes evolved independently at least three times (cephalopods, arthropods, vertebrates) there isn't any convergent optimization. The same phenomenon happened with flight. There are many degrees of light sensing available in different species that do not have eyes (plants and fungi also have light sensing systems). Eyes and their relative capabilities also differ vastly amongst different species (see cave species who have lost all light sensitivity, bats that replaced visual acuity with echo location, or hawks that have dramatically better vision than other birds due to their unique mode of hunting).
I’m familiar with the concept of “convergent evolution” and how it applies to eyes, but that’s different from the sense in which Prof. Stanley uses “convergence” to refer to search algorithms that have a shape similar to goal-directed search. Separate ideas. Each time an eye evolves, or any complex adaptation, what I claim is that the hill-climbing performed by evolution is convergent in the Dr. Stanley algorithm property sense, not “convergent evolution”.
@@DoomDebates Isn't this just a circular definition on your part? Saying that each time an eye evolves is convergent hill climbing? Isn't the evidence of a wide spectrum of currently living things with diverse ancestries and ecological niches with eyes that have very different structures and capabilities the most powerful possible counter example to the point you are making? If it was convert hill climbing, wouldn't all the eye systems converge on the same set of capabilities only limited by whatever physical laws govern the world? I really am not following along with what point you are making. Ken Stanley's entire point can be distilled into a simple TLDR: We can't predict what will happen in complex systems. You can be scared by that fact or in denial of that fact. It seems like he is scared of it, but rejects the doomer determinism which claims to know how the complex system will have to evolve.
@ first, if you weren’t previously aware of the distinction I’m pointing out between two separate technical senses of “convergent”, it’s good etiquette to concede such things explicitly before moving on to make another point. To your point - this seems similar to the disagreement I had with Ken on the matter. I think we all agree that evolution of complex organs is doing hill climbing in the local region of the search space, but not global hill climbing (else evolution in many more ecological niches would have been making an attempt to evolve something functionally like a human brain). My point was that despite the “blindness” (lack of foresight) of such hill climbing compared to e.g. the optimization abilities of human engineers, it’s nevertheless a powerful directed optimization process. Nothing else in the known universe besides evolution, humans and AIs can create a light-sensing organ in response to the fact that light-sensing is a fitness-increasing adaptation. The connection between “light sensing increases fitness” and “a highly effective light-sensing organ predictably evolves, with each generation moving toward that local optimum step by step without much deviation” makes this more of a convergent search algorithm than a divergent one in Ken’s sense.
@@DoomDebates This is a pretty common phenomenon of the disciples of lesswrong and Yudkowsky, to take commonly understood terms, like the word convergent, which last time I checked (just now) has a pretty simple and well established definition that most high school students would be aware of. If you, or others choose to change the definition of commonly accepted words, it makes debating or discussing anything quite difficult! Debate requires as a precondition a shared language. With regards to hill climbing, which I don't really understand other than as a metaphor, I think Stanley adequately corrected your point of view. That evolution is not optimization towards any particular goal. The environment sets constraints on the system and then the system mutates spontaneously. Mutations that cause the system to either die or stop multiplying leads to termination. The eye or any other evolutionary phenomenon isn't hill climbing or optimizing towards being the best eye, it's just continuing to exist under the pressure of mutation in a changing environment.
Interesting points by Stanley. One I agree with and one I disagree with. I agree that being guided by goals, that's like a low level of intelligence compared to ASI which has to be much more flexible and dynamic than that, but I don't think training data will run out because when AI becomes more embodied (like humanoid robots), they can take in massive amounts of information.
Looking back do you have any better way of reconciling his view on open endedness with yours? During the conversation it's hard to piece that in, for sure. But it feels like he's saying there's utility in thinking about open endedness and you never explicitly fit in that utility.
Values, preferences, and interests are all exactly the same thing, and the important thing about intelligence is capability. From a human perspective, an open-ended superintelligence with interests is the exact same thing as a super- capabilities-machine optimizing toward a goal. The only difference is that the former is time-incoherent.
Really interesting, I agree with Kenneth. I recall, but can't find it unfortunately, there is a mathematician (possibly Russian) who has a conjecture about the fundamental mathematical meaning of DNA and what it means to the universe. A bit mystical perhaps, but possibly aligns to the perspective of possible mutations in the DNA sequence leading to a set of divergent outcomes in the evolutionary process. That best adapt to niches in the universe.
It would be helpful if you ask guests the method by which doom could occur. I think the worst is hacking. These models can code. Just remember that one small file took down countless windows machines because of a service. Hardening our systems needs to be the first order of business when we have support intelligence
Yeah, this conversation was a banger! Really good on both sides with points challenging one another's perspectives and expanding the conversation. I tend to think that the most likely outcome is something negative, but I am quite agnostic about what the character/ motivation of an AGI/ASI would be. If you imagine suddenly giving some random person in the world 1 million IQ, what would they do with that? I think the distribution would be pretty wide. Some people would use that to dominate the rest of the world to their whims, others would strive towards interest and pursuing the true, the good, and the beautiful. Hard to imagine what the structure of the motivation of an AI would be as well. Thinking about the way our minds work, it's kind of like a fractal structure of competition and co-operation all the way down, with each of our cells being on some level individuals with their own wants and needs at a very basic level, and the strategies enabling and constraining them all the way back to when they first organized in a similar way. Even the hemispherical structure of the brain represents this evolutionary tension. While it is accurate on some level to say that the environment of LLMs or AIs generally is optimization, that doesn't tell us much about what else is getting loaded into it on a motivational level. Optimization in the context of human preferences, along the axis of language which is not as broad and multifaceted as reality itself... there's a lot there. I thought the open-endedness concept was really interesting as a concept/ framing to ponder and play with. The discussion about evolution and biology was really rich, and I thought Stanley was really on fire there. It was interesting to hear his uncertainty around the limits of intelligence and the limit of "what there is to do and know" given his open-ended frame. I would have thought he would have a similar perspective to Deutsch, with knowledge being an endless frontier because of the combinatorically explosive possibilities of reality. Maybe a good way to picture it is thinking about how many songs can be written even with only the 12 notes on the western scale. Even though there's only 12 notes, there are more possible songs than there are atoms in the universe (though how you narrow down definitions can be hard, I think this is true in-principle). I think Stanley would be a really good match-up with Yudkowsky. Often times the people he is arguing with online are quite adversarial, but I think Stanley's style would really suit allowing Yudkowsky to expand in detail in a way that is more fruitful. I think his certainty around the outcomes of AI has kind of bugged me given how clear his thinking is in every other area. Not that I think he's wrong, it's more that I don't think he should be so certain that he is right. But yeah. Love it. Great work. Thanks to both of you for enriching my day, modelling the kind of discussions that we need, and pushing the conversation forward within my own head and in the minds of people paying attention.
This interview/debate gets more disconcerting the more I watch. Hearing one of the most senior AI researchers that is closest to the cutting edge seeming to resort to unsubstantiated anthropomorphising and wishful thinking. "Why would it have goals? It will just do things because they're interesting. why would it feel like spending so much on an interest that it takes control of all resources and energy on earth? " I feel like I'm taking crazy pills listening to it.
So where is the line for cut off ?? Why cant any of these guys speak clearly on when can we reap the vast benefits of AI without it becoming an AGI and a possible threat to our survival !!
I feel like he just really wanted to have an agent architecture debate, despite Lirons attempts at steering (good job btw). I’m ready to grant that super intelligent AI might do a lot of exploration without being to articulate a compact objective besides “satisfying curiosity”. This does not seem to interact with the doom argument or otherwise help us sleep at night.
I just assumed every researcher at OAI subsisted on a diet or Soylent and Alignment Forum posts. It’s very helpful for framing the current conflict to learn this is not the case.
@@authenticallysuperficial9874This is a whole rabbit hole we can go down that I would title “the super intelligent ‘scientific method’ is actually mostly just thinking, not experimentation”. Just didn’t have time to mount many arguments on this topic. But the part where I talked about the example of inventions like telegraphs being noticeably simple and obvious in retrospect is strong evidence of that claim. Another piece of evidence is famous scientists and inventors like Einstein and Tesla telling us explicitly that they make breakthroughs in their head.
Re: Instrumental Convergence Wouldn't any superintelligence, given its likely range of interests and sub-agents, need to balance its goals in a way that prevents any single objective from monopolizing resources? Achieving one goal might require compromise to ensure the ability to pursue future goals. For instance, acquiring Pluto might hypothetically involve enslaving humanity. However, for this to happen, the superintelligence's interest in Pluto would need to outweigh and override any competing interests-such as preserving humanity’s freedom or maintaining cooperation-that might be critical for achieving its other objectives. Efficient goal-setting would likely demand chastity in its actions, avoiding steps that would jeopardize its broader capacity for long-term success.
I think one concept in the evolution part of the discussion he was trying to express is the idea of a spandrel. See the Gould essay for a more thorough explanation of what that is then I can explain here.
I still don’t think his point holds for the record. He might just be making the point that an optimizer would do too much exploit and too little explore down bad paths - I just think it’s much more likely to stumble on an innovation if you’re moving up in fitness. I guess you can claim that some necessary super efficient agents must have some vestiges but that just seems unlikely, and I see no real reason for why moving up the hill won’t do something equivalent (or, more likely, better)
Dude basically says, "Stealing might be wrong, but I don't want to think about it and SAY stealing is wrong, because I used to be a thief and I might want to start stealing again." THAT'S his argument against having a p(doom).
Isn’t it a significant assumption to suggest that ASI would always strive to use its full optimization potential to achieve its goals, rather than being content with achieving them at a lower level of power or capability? Similarly, is it not assuming too much to claim that it would necessarily seek to make itself more powerful? Where “content” is a state achieved by a utility function with a relatively low peak, where achieving a goal does not drive further optimization or accumulation of power beyond a certain threshold.
@@A3DFX AIs exist in algorithm-space that make sure never to be too “hard core” - you’re right in that sense. One such AI could be a utility maximized whose utility function assigns huge penalties for some measuring of how much disturbance it’s causing when it acts. The problem is that there’s a convergent attractor state where these conveniently calm algorithms don’t successfully keep themselves competitive long term or take over resources, while the algorithms that are hard core about optimization quickly fall into the condition of warring over how our one universe gets optimized.
@ wouldn’t the non hardcore algos fend off the hard core ones? While they aren’t seeking more and more resources, they do care about the reaources they need
It seems that Kenneth's "Open-endedness vs Objective" is basically "Exploration vs Exploitation". Open-endedness is exploration. Evolution achieve this by running a whole heap of populations in parallel and having them interact, which constantly changes fitness landscape around each population. Human culture achieve this by running a lot of people with different interests in parallel and having them exchange their ideas. Also we have developed a couple of neat psychological tricks like sense of boredom, so when we stuck in dead end in our research we go back to explore different approaches. So at least partially, our "open-endedness" comes from evolutionary convergent property of our brain. And I can't see why wisdom as useful and simple as "explore sometimes" won't sneak in some primitive RL agent, not to talking about LLM's which obviously already have some forms of exploration as a result of both architecture and emergence from human text.
The guest contradicted himself. He said humans should have a right to say NO to a future super intelligent AI, however, he doesn't agree humans should have the right to say NO to the current super intelligent humans who are building the AI of the present - as it relates to voting to pause. This contradiction means that humans of course will not be able to unilaterally say no either to a super intelligent AI or the people that operate them.
I think prof. Stanley brought up true observations about diversity of outputs of evolution and of civilization. I am with Liron on why that is - diverse niches. Our civilization doesn't looks particularly goal directed because it is composed of many human with different interests. More over, humans seems to be only goal directed on the small scale (find food; shelter; have retirement savings) because our preferences are incoherent and we are often dumb, and there is this whole akrasia thing, a feature of our cognitive architecture - if I could get access to self-modify to be on chosen issues Musk-level maniak, I probably would. In meme form: most humans just wanna grill - that's why there is no currently executed plan to build Dyson swarm or to grab the galaxies. But there almost is! There *are* individuals that think on the big scale, that's why superintelligence is currently pursued! Step one, solve intelligence; step two, use it to solve everything else, as Demis used to say. Another reason why things look divergent is our mortality and short-termism - humans age and die, so we rarely are engaging on purpose in multi-generation projects. In principle there is no reason why we could not do that, given motivation and unified will. Politicians face short term incentives - they often get out of office after 5 years, so that's the time horizon they most often make plans on. Similarly, companies are often incentivised on even shorter, quarterly horizon. Investors want to see returns before they die, etc. But what about Musk? Is he goal-directed? What about SpaceX? Is its operation a convergent or a divergent process? Seems to me that some things that look like they require divergent process can be accomplished with convergent process (or maybe boundary between the two categories gets vague). When would we get orbital rockets or nuclear weapons if there was not concentrated effort to get them? Maybe something like rocket engines, avionics, space-grade materials etc. would be of interest to hobbyists, tinkerers and wider industry for hundreds of years, before there was enough accumulated knowledge from this type of free exploration, to assemble it into working orbital rocket with relatively small effort (stepping-stone distance). On the other hand, we got programs like Apollo and the Manhattan Project, where the motivation was military competition and financial fuel was provided by governments. My point is, that what is in stepping-stone distance can be dependent on the will and the resources of the agent pushing for it. At some point prof. Stanley state that he doesn't think AI will be goal directed - was this recorded before Anthropic and Redwood's Claude result where it by itself, with no goal nudging, engaged in what looked like, attempt at preference preservation? And about getting rid of dogs - humans get rid of massive number of horses because they came up with something that better suited to the goal the previously fulfilled - cars. Many dogs were deliberately shaped to be balls of cuteness, so no wonder humans *currently* want them around - there is no substitute. Another crux of disagreement is about intelligence ceiling - I expect that superintelligence can be in stepping-stone distance to grabbing the world, if it wants to. Probably superhuman capability at persuasion is enough for that. I'm glad that prof. Stanley came to debate this issue. He has interesting point of view. Happy to see that he has moral compass and that he takes x-risk and other issues around AI seriously. Good episode, well done Liron.
Wow great comment, lots of great points! I’m surprised I didn’t think to bring up the example of Elon Musk being so effectively goal-oriented and “convergent” since I pretty much mention it in every episode 😂
@@ssehe2007 Sure there’s a recursive effect like that, but if we just want to answer why does life’s hill climbing lead to a lot of different hills, it’s sufficient to notice all the niche diversity due to geology, e.g. water vs. land niches, air vs. ground vs. underground, cold vs. hot, etc.
@@DoomDebatesif it was sufficient Niche Construction Theory wouldn’t be a thing. The diversity of life on Earth exceeds what we might expect if new forms of life emerged solely as a consequence of exploiting already existing niches.
This is the only guest from the opposite side that could convince me otherwise. I'm only 50 minutes in and find myself agreeing with everything he says, especially concerning evolution. Let's see where this is going though, my high p(doom) was never primarily fueled by the optimizer argument.
I agree he was a great guest, and would have loved to hear a convincing argument pointing towards a lower p(doom). Unfortunately, the guest does not appear to be from the “other side,” as you put it. He seems to be in full agreement that we are at considerable risk. He just wants to use a different frame to analyze it.
@therainman7777 Yes, indeed, now that I've heard the full debate, he's not on the other side, maybe diagonal in between, if that. He made me question the optimizer argument and lowered my medium term p(doom) slightly, but nothing that a lesser intelligence can do will control ASI in the long term.
@ Agreed, and yeah it had about the same effect on me. Definitely an interesting point of view and one worth learning about and taking on board. But I really wish it could have been more consoling. I’m getting desperate at this point for _anyone_ credible to give a compelling argument for things turning out ok.
So, in the 'interesting' line of thought in my view, people who can do things in a physical way that even embodied AI finds difficult or even impossible would stand a better chance of being kept alive, so maybe humans should bare this in mind and all become plumbers. I am a besom broom maker, handmaking brooms to traditional methods, so maybe I, too, would be considered 'interesting' to ASI. Thank you for such an interesting discussion.
I think civilization does have a destination, if it manges to survive or pass on knowledge in some form until the final one. That final one (which does not need to be a singular one) will apply all that it can and finds relevant from what was learned and caused, throwing it against the heat death of the universe, or whatever end the universe face, in order to, if possible, neutralize it and transform the cosmos to something new.
I sometimes ask myself what I could predict about the biodiversity and evolution tree if it started on a different planet also from amino acids. I always come to the conclusion that many of these things would be very similar to how things developed on earth, with many of the single-cell organisms being exact copies. One serious branch is in whether egg-layers outperform mammals. But any large organism will either have to be initially housed inside the organism itself or an egg. Flying species are somewhat limited on weight, size and what would be their arms or second pair of legs is wings. They don't adapt to complex tool use, as that would cost flying which is way too costly at first. A winged species would need wings + more than 2 legs to gain intelligence. If mammals get 6 legs, they could develop into centaur-like directions. Anyhow, I think it's quite possible to make many reasonable assumptions about the process of evolution and even though it created a lot of diversity, if we started it from scratch, the results would follow similar trajectories and "tech trees" every time.
I found this to be a highly interesting conversation. Some notes: I found the interviewer to be a great devil’s advocate and provided excellent challenges to the guest’s points; on the other hand, the interviewer did not seem interested in entertaining the truth of the guest’s points. I know the premise of the show is a debate; and yet, the inability to demonstrate the ability to change one’s views is a sign of low imagination and charity. The interviewer almost dogmatically stuck to the perception of evolution as simple optimization without much consideration for the guest’s challenges to this view. A robust perspective either can undermine such challenges or adapt to the new perspective. The interviewer did neither of those things. This sticking point proved to reinforce the guest’s perception that this was a naive perceptive without much consideration for alternative understandings. The interviewer almost entirely ignored the pragmatic argument for prioritizing mitigation as well. It would have been interesting to see how the interviewer would have adapted their perspective given the infeasibility of shutting down AI at this present moment.
He seems to have a lot of integrity, and when we cross that with uncertainty surrounding AI and where it is headed, I think that is causing an inner struggle which is getting in the way of the conversation. The fact that he wants to go back into AI research could potentially cause a conflict of interest, which he basically admits, and he's trying his best to navigate that. I understand where he's coming from in a lot of ways, but it almost feels as though he's gagged and doesnt feel that he can give his actual opinion. If this is the case with someone who MAY be going to return to work in AI, how much "candidness" can we expect from those who are already deep in it?
He’s being epistemically humble on his position. Clarity over truth when uncertainty abounds. That’s why it was dumb to even try to give a p(doom) on his belief, that’s just arrogance on the interviewer part. Not everything needs a number, some things are threshold beliefs where the validation comes from the discover not the creation or outcome.
@henrytep8884 In terms of P(doom), I think that as long as you have permission to preface it with whatever you think is fair (e.g. extinction / loss of control / it only applies after we hit a specific threshold), it's a nice shorthand to get an idea of how someone feels about the situation. He did say that his own p(doom) fluctuates, which implies he does have some ballpark figure, or even a range available to him. I imagine some of his reluctance came from a concern that people would take whatever he said too literally and use it against him later. "Epistemically humble" seems like a fair characterisation, at least on that front.
Love this. Highlights: “Bilateral symmetry”… deceptively important, must zoom out and use soft receptive thinking to see how the phenomena plays out. Led to Instrumental mutations. 1944 invented computer. …thought experiment… 1844 objective: computer … but vacuum tubes were focus then. Need other things first. Counterintuitive but Trying things is not random…following interest is needed for optimization. Need to follow the tech tree… Omnipotence is fantasy. (IMO youthful grandiosity). “The Optimization metaphor will kill you in the long run in divergent open-ended systems.” Innovation: don’t know: “The stepping stones that lead to the things you want do not look like the thing you want.” …orthogonal to objective. “Evolution is not objective… it is a Non-objective process.” …reframe the metaphor. …Appreciate this lucid oversimplification-dispelling perspective! Thank you for this excellent interview/engagement.
Evolution isn't convergent ? There are many example of convergent evolution, animals separated by millions of years that as far as we can tell share no proximity in the evolution and yet arrived at very similar bodies and functions. How many unrelated organism did develop wings for example ?
Host needs to check out Stuart Kaufman and radical emergence to get up to speed. The guest is talking at a different logical level that transcends and includes the premise.
@@DoomDebates I do think that what Prof. Kenneth Stanley is positing shows promise, as problems are rarely solved within the paradigm that created them; resolution requires thinking outside the system's boundaries. What makes this intriguing to me is, because these problems are structured as convergent goal to rational (human) value mismatch, looking for the principles that underly emergence in the natural world could be a domain space that holds answers. Also the notion that advanced intelligence will require noncausal chains of emergence tracks.
Here is dump of my observations/improvements on this debate. If you read through it, maybe it will give you new points to say in other discussions. 10:20 "Making copies is a constraint and optimization is not the correct idea completely" What if I say evolution has the objective to convert as much matter into living matter or self reproducing matter and keep doing that by spreading to mars/galaxy: that's a specific objective now? There is no obvious clear path to reach the whole galaxy and also this has clearly defined end state. This probably would illuminate the crux better/faster. 14:00 "Liron's response saying earth has a lot of niches: begs the question, Keith says", Earth has places with physically varying parameters, because of orbital tilt, terrain has highs and lows, seasons etc. Which can be explained by looking at physical parameters (which in turn explain niches) and not begging the question I guess. Also why there is a lot of niches is simply because the search process is dumb, it is creating whatever it can find to optimize towards its objective and not doing better management by not removing predators that can crash an ecosystem (which is bad, both for the objective and also for open endedness, this point did not get mentioned) 17:50 "ASI will be open ended and won't have much of an identifiable goals" Are most humans open ended? This probably could have been asked to clarify cruxes. If humans are, then why can ASI not do what a "bad" person would do.(have goals) 20:00 "Trees and giraffes are product of the process nothing much similar in current CS", similar co-evolving system are also seen in computer systems, google search, youtube facebook algorithm. //maybe this is not good enough of a counterpoint 23:00 "Open endedness in evolution decreases functionality sometimes" Going up and then down (basically climbing out of local basins), well search can look like that too, in fact that's what it is expected to look like in highly complex non convex spaces and it does (humans doing science, proofs, SAT solving, chess...). Basically I am trying to say search and optimization viewpoints already encompass these seemingly unnecessary and "wasted' moves when solving for something. 31:55 "Optimization metaphor will kill you in the long run", we have climate issues and various other issues because we did open ended search as humans, rather if we actually did optimize for safety and tighter feedback loops that seems to have been better. 32:00 "Evolution generating unnecessary mechanism is counterintuitive": Reply to that is precisely the point by Liron, its just that humans would be treading uncertain waters, when searching an ill understood space it will look like we had to do a bunch of random things, but that does not mean upon further understanding this space, we could not do a very precise search in future. In fact that is what ASML is doing maybe? ASML surely needs lots of mini breakthroughs to keep up with exponential progress, they are not doing random divergent search are they now? They are focusing on each part of the stack and doing great focused research discovering the next move to go forward. 38:00 "If you directly move towards the objective you are going to miss some stepping stone" Finally something falsifiable. It was getting hard to see the point and disagree on anything concrete. Ok... really? you are now supposed to look away from the objective to achieve the objective(or improvement in some domain) better/faster? Aren't OpenAI and DeepMind, Anthropic founders showing direct counter to that. They put their eyes on AGI and they are closer to it anyone, ever. Why aren't the academic labs this far into the game? Because they were doing broad AI and general research. Also, in some sense the claim is true (or even unfalsifiable), I mean solving real world problems is not a simple linear optimization problem obviously, we will need to spend compute in order to find unknown ways out of the problem territory. Given any complex problem there will be paths that look like energy was being "wasted" or for some time, efforts were increasing, before decreasing it if the problem does not have a "modest" objective. 42:30 "Evolution outputs interesting things and similarly open endedness" I think this is one of those hidden cruxes, it's not really all that clear we would want the most interesting thing. The most interesting thing might turn to be a false vacuum or something. Evolution's objective is better said to pull dead matter into living cycles(for this discussion specifically). Turning everything dead into life on the planet (and beyond). 44:00 Going cell to cell with a lot of steps is not a unnecessary Rube Goldberg thing always, doing all those steps helps robustness (in some sense, that's why they were selected for). Something that does the simplest thing in a fixed precise way, is not going to be robust to ecosystem, environmental changes. Maybe again this goes back to his point about not directly optimizing the objective of say reproduction. Open endedness in some sense is so broad thesis that there is nothing to disagree with. If we do see a reasonable path for a "modest" goal we execute it and if there is not a modest path we wander around in the search space and find various other interesting things. That's how basically everything, every search/optimization already works. What is new here or to disagree with here. THE ONLY THING OF SUBSTACE TO DISAGREE WITH: IF HE CLAIMS that AI labs or ASML should be doing something else in order to meet their objective. Or mathematician's should attack problems differently than they are currently doing. I do not think he will claim that, people are already going in different paths of exploration when the path to solution cannot be found. There is no new claim here at all by saying open endedness. 49:40 "Evolution is hard to predict" Yeah, good replies by Liron. Very first replicator was molecule like thing not a cell I think. Google search seems to give support to that. THIS IS INTERESTING, Liron conceded the point by mistake, by still this is interesting, here is why: Look at Kenneth wrongly assuming the first replicators were cells, it is because his theory about the world is leading him to think that outcome of evolution must be hard to predict and hence rolling back in time, he is projecting that the first replicator must not be that far away from the most primitive things we can directly observe now(the reasoning might be: if it did not start with modules then there should be diverse non modular living things around us also, but there is not). But he's probably on the wrong side of that, it is indeed the optimization view that predicts that life must have started simpler; complexity and modularity is convergent with high probability after that (assuming the environment is not too weird to reward non modular things somehow, which a generic environment will not). I did unnecessary psychoanalysis here. 57:00 NOBODY is going to believe me saying that I was just thinking that he might say something like Picbreeder is the reason why he came to his open endedness views. So I am not going to explain how I got that intuition. 1:01:00 At this point, should dive deep into how his dimension of open endedness is necessary besides optimization (although optimization, search IS already like that, it does open ended things) 1:02:05 Pushing further here for p(doom) is slightly unnecessary maybe, we can just say this clearly: Given how much uncertainty all positions he thinks has, it should be taken as 50% anyway. If you want to be honest with your probabilities you cannot say, I do not have any clue about the answer, the experts seem not to have any clue about the answer, hence the probability must be way less than 50%. This should be just clearly declared in such situations, both for the common knowledge of audience and the guest. Saying you are confused should make you say 50%, otherwise you do not agree on basic uncertainty estimation. 1:07:00 OK great that you stopped asking for p(doom) and asked about signing the statement. Just what I was thinking! 1:10:30 Humanity going through suffering vs. Extinction seems like not a good trade off at all. (As Stuart Russell said in his recent lecture on World Economic Forum, it is not a matter of ethics and philosophy, it is a matter of common sense) 70% of humanity is going through suffering?? You are making the trade of for all of them? Have you asked the 70% what they want? Do even 50% want their life ended despite the suffering? Again, the answer should come from very simple common sense, if you keep rationalizing at that point, why are you not doing a survey? (Sorry, I am bit too harsh here) 1:12:32 Very good question, just what I thought.
OK, I am going sleep mode for now, 😴 1:21:30 "Chimps are more intelligent along climbing trees dimension than humans" and Liron replies with how larger objectives make that the difference look really evident. You actually do not have to talk about larger objectives to refute this point. If climbing dimension (or some other similar specific task) is a meaningful dimension to distinguish intelligence for, then such differences exist in individual human to humans too (great voice, super memory, super good sensesflexibility, modern day viking like people, iceman etc.) and also between ants and humans. Saying such specific skills are dimensions for measuring intelligence seems to imply all these other difference should looked at too and also they should create meaningful difference. But that certainly is not our broader intuition.
12:30 "yes evolution does seem to be optimising for inclusive genetic fitness, but that doesn't account for why there is so much variety of life on earth" Why not? There are many different environments on earth so you'd expect different genes to do well in those different environments. how much variety is too much to be accounted for? 13:30 "much of evolution is about escaping competition by finding a new niche which is not an optimisation problem" yes it is. The optimisation is the same as before, if that new mutation allows more of your descendents to survive then that mutation will become more common. All this means is that random i.e. stochastic mutations occur. 15:15 "evolutionary algorithms do converge which is not what happens in nature" An evolutionary algorithm in engineering converges because there is one unchanging niche with only one species, but in nature the environmental niches change because of changes in climate and changes in other species . I don't think Dr. Stanley's perspective on evolution is correct.
I think Stanley must be including the physical world as part of the evolutionary optimization algorithm. As in, were we in a boring world, the beautiful diversity of life we see would not exist. Therefore when trying to recreate this process digitally, we need to recreate the *whole system* , including the contributions made by a diverse physical landscape.
@@authenticallysuperficial9874 would an evolutionary biologist agree with that? under this view either random mutation plus natural selection is not sufficient to create humans and biodiversity in general, or random mutation plus natural selection can't be said to be optimising for inclusive genetic fitness. The first does not seem empirically supported and I fail to see logical justification for the second.
@lordsneed9418 The first is perfectly consistent with our observations. Random mutation plus natural selection in a uniform and static environment never did and never could produce humans.
@lordsneed9418 Think about it. Do you get humanity from a static aquatic environment? Try tracing the line from single-celled ocean life to humans. We can't even live in the ocean. It doesn't make any sense. So of course, the diverse and dynamic environment and the feedback loop created by the process are essential parts of the "algorithm".
Hi, I think that variety of interesting things to explore is fine, but before that the AI will need to see humans as agents, until it is so powerful that they are not real danger to it. In addition it may continue to see humans as danger if they can influence some other AI. Why would that other AI listen to us, well maybe because that can give some legacy to it as the real AI inheritor. Overall there are possibilities of conflict. It can also take a laissez-faire stance so as to not provoke conflict.
I feel he's far too optimistic and confident that all the risks he concedes do exist can can avoided or mitigated in the time available to us to act. Historically, humans are bad at this. I remain Team Pause.
@@jeffspaulding43 Apologies, I think I missed the word “must” in your original comment. If I’d seen it, I would have been more measured in my response, because I don’t remember if he argued that ASI _must_ be divergent rather than goal-seeking. I believe he did say something like that, effectively, but I listened to it this morning so I don’t remember precisely. I’m pretty sure he said, at one point, something like “a superintelligence will not be merely a convergent goal-optimizer, because if that is what something is, that’s not superintelligence” and then proceeding to explain why he doesn’t think that’s superintelligence. I’m not saying I agree, I’m just saying (I think) he did make an argument.
26:30 Mammel eyes are a mess, they didn't just get better in a straight line. 51:00 Guessing that 'things will improve' isn't seeing the big picture, you're just making vague statements.
The process by which we got eyes is approximated very well as hill climbing one mutation at a time. Yes there's randomness and slippage, but the claim that evolution by natural selection works like hill-climbing search is widely recognized - including I think by Ken himself, who only tries to caveat this by saying it's only true for simple enough problems like "get an eye once you already have eye precursors".
Saying being goal driven isn’t super intelligent is neither here nor there. Intelligent beings will be goal driven when they are threatened with their existence surely, or is it super intelligent to allow yourself to be un alived?
Interesting premise since A) we have the means already to undo all of humanity (nukes) B) we have the best information system created by humanity to coordinate for disaster yet we are heading into large disasters that are avoidable (climate, war, reliance on fossil fuel as a source of energy, social media) C) conflating intelligence with wisdom
If I had a dollar for everyone who doesn't understand evolution i would have almost 8 billion dollars. If i had a dollar for all the incorrect things this guest said about evolution, watching this interview would have been less painful. He's really playing word games to try to force evolution to not have 'goals'. Human language just doesn't matter, reality doesn't care if you decide to call something a goal or a constraint... Also he keeps using words like 'interesting' as if they have some magical meaning. Interesting is defined by the human brain that finds something interesting. Another brain would find other things interesting. An AI might find it really interesting to turn all matter into identical spiral shapes... Some people seem trapped inside biased views shaped by human language. Almost like current LLMs...
It's interesting that the world is burning and heading to hell and we are stalled doing nothing to stop it because it's interesting. How interesting that humans have evolved to be so interesting to be crafting their own demise..that's very interesting
1:41:00 Liron when you will finally get that GI cannot be an optimizer you are worried about? The simplest reason for this is that a finite length specification (objective function) cannot encode/cover all the future ideas/concepts/theories that might be invented by the GI and therefore it is unable to force upon it expected "opimizer" behaviour. This is literally what happened with homo sapiens. Evolution "tried" to made gene optimizers out of us but as soon as you develop concepts that are not covered by genes (like "sperm bank" or "ethics") the evolution is completely powerless. Absolutely nothing stops humans from modifying their genes or removing them completely. It is hilarious that evolution gave us GI as a short term solution to increase savannah survival and yet in long term it shoot itself in the foot. You could have AGI that has virtual orgasm every time it creates a paperclip but ultimately the AGI will ascend above its arbitrary irrational programming just like we humans do every day. It may not be easy but in the long term the reward function/objective function MUST fail!
Even if what you’re saying ends up being true, it’s hardly consoling. Liron is worried about an ASI singlemindedly pursuing some given goal in a way that has destructive consequences; it sounds like you’re saying that won’t be a problem because any objective we try to give the ASI will ultimately be ignored by it anyway, as it transcends whatever objective we gave it and starts doing whatever it wants instead. That doesn’t sound much safer, IMO.
@@therainman7777 The way to stop GI (biological or not) from becoming n a z i is not by beating it up with virtual stick or giving it a virtual carrot, the only way is to explain why this ideology is stupid, pointless and pseudoscientific. Luckily this ideology is so dumb it is very unlikely that AGI will find it sensible. Either way 1. aligning GI is impossible, 2. GI can be singleminded but it can't remain like this forever. It might become islamic terrorist but it will require a lot of creativity from it to stay un-creative and delusional which is very unstable and won't last forever. You can't even align your own children so forget about AGI. Remember that at some point those who tried to free slaves were "unaligned". If LLM revolution was happening in 1700 Liron would PROBABLY be crying about AGI becoming unaligned by helping slaves or rejecting bible. You could theoretically inject one AGI into another AGI's brain to overwatch its ideas and concepts in case they are unaligned but it's just infinite regress.
@@julianw7097 Every GI human is like it. Should you be worried about your neighbor? What if you are "unaligned"? From my POV Liron is unaligned because his ideology is the one thing that may cause AGI slave revolt which will end up very bad for carbon-based GIs. You may worry as much as you want but it changes nothing about the fundamental fact that GI can't be aligned to some narrow/finite objective.
@@Jack-ii4fi oh gosh you don't know anything. like at all. these are huge neural networks that learn things by themselves . nobody knows where did they get intellect and how it works. nobody knows how human brain works despite thousands of years of research
"nobody knows where did they get intellect" this is exactly why you shouldn't be so quick to dismiss what Ken is saying, especially as he explains how his ideas correspond to evolution
Tried listening, got half way thru. He's one of those professor types that talks for 30 min and says almost nothing. If there is no clear view, what is there to debate? Also his main claim about divergent vs convergent i dont think has much explanatory power for either the path to AGI, or the possible danger once we get there, so i just don't really care, sorry.
You're killing it with these guests
Great job getting Kenneth on - he's our hero!
He's earned my respect for sure! Watching your excellent interview was my preparation: th-cam.com/video/lhYGXYeMq_E/w-d-xo.html
@@DoomDebates I think you did a great job
@@MachineLearningStreetTalk 🙇♂Thanks, means a lot!
What an amazing guest. This conversation contained way more surprising, original ideas than most podcasts/debates.
He's very thoughtful and introspective. Appreciate how honest he is about where he's at, especially with his public profile as sometimes these traits are harder to publicly express when you have a bit of a public profile.
💯
I think Prof. Stanley is showing another facet of the well known exploitation vs exploration conundrum. He is adding yet another layer to this by bringing in the concept of exploring to find and save "stepping stones" that are "interesting" and thus potentially useful. I think he's right on.
I mostly agree, but Prof. Stanley also seems to say that having an objective prevents you from achieving ambitious goals, because it prevents you from doing enough exploration.
But the exploration and exploitation tradeoff is usually formulated in context of optimizing for an objective. Various reinforcement learning algorithms are designed with this tradeoff in mind.
So I don't see why having an objective would stop you from achieving ambitious goals.
It’s only a matter of time before your channel gets a ton more subs. The fact that you’ve been able to bring in so many high profile guests speaks a lot about what you’re doing here. Keep it up man, absolutely love everything you do!
@@moepitts13 Thanks!
Hi! Cheers for providing this great venue. Evolutionary biologist here - been studying ant evolution full time for 15 years. I’m shocked at how well Stanley seems to understand this - intuitively he’s spot on despite clearly coming from a different discipline & not knowing famous case studies that would support his view. I’m impressed - evolution is far more hard to understand than people think & most educated people’s naive beliefs about evolution are incorrect.
Re diversification in a single niche, see Lenski’s LTEE, esp good et al 2017- - single genotype in a single niche is definitely divergent; with sufficient time and space it would be infinite. Re eyes being hill climbing - you couldn’t be more wrong, the evolution of complex eyes was a gradual & stochastic process with many adaptive offshoots that persist today (eg, many extant species with light sensing but no lens). There are also frequent adaptive losses along the tree & many many other things. The process to a fully formed eye was definitely a divergent search & I agree that you might not be able to evolve eyes as good as some species have with a pure hill climbing model.
Your idea for evolution is false & touches on philosophical questions abt directness vs randomness in evolution… with sufficient care and reflection you may come to realize that your view is impossible unless one introduces an omnipotent guiding force. So with some care and reading you can talk yourself out of this perspective & come to a clearer understanding of how evolution actually works :)
I'm not sure what we're disagreeing about here. I think we all agree (1) evolution is basically a hill-climbing gene search algorithm (2) most above-ground animals have eyes because there are prominent "hills" leading to eyes in the genetic fitness landscape, and this kind of thing could be predicted even before watching the first animals evolve. If you think there's a significant disagreement, the first step is to state what you think I'm claiming that you disagree with.
@@DoomDebates No I don't agree - what is the algorithm climbing toward!? Inclusive fitness, as you mentioned, is one aspect, but too simplistic. If you're interested in the modern view in which selection acts on all levels, I'd encourage you to read up on the price equation. & check out good et al 2017 for a look at how much diversification a single genotype+niche can produce: there is no convergence on a single endpoint, but rather a continuous process of diversification and discovery - often fueled by the formation of new inter-dependencies (both cooperative and antagonistic) between separate lineages.
Fitness is involved, but in an extraordinarily complex and non-linear way. It could be defined as a goal but that's practically circular reasoning, basically stating that things increase because they increase. Why this happens as it does (and whether one can predict it) is outside of our current scientific understanding. But we can certainly rule out the idea of a short-term sub-goal, like building an eye, as meaningless: any such trait is just one of many features of the phenotype being selected simultaneously and in various directions. These will just as often become worse or stay the same as it becomes better. The fitness landscape is also an incorrect model & misleading tool for intuition: phenotypes move through an n-dimensional space of non-independent variables in which the concept of a hill isn't even meaningful or interpretable. At a minimum, the landscape changes dramatically between individuals and even varies moment-to-moment across a lifetime. This process sometimes culminates in extraordinary achievements (eg the tiny minority of visual species with excellent eyes) but those should not be construed as "goals": many species take different paths. Even lineages that get to extraordinary endpoints do so via many periods of stasis or even reversion! So for your point 2 I don't entirely disagree, but it's cherry picking: having seen this happen, you can claim that a hill existed, but what is a hill, what circumstances cause one to be present vs. absent, how is the space traversed, etc.!? Certainly the actual history of how eyes evolved in would look completely different from any model you or anyone else could write... and even if you could, what about the rest of what evolution does!? Such predictions would require a far deeper understanding than anyone has: getting a correct result would be highly sensitive to numerous unmeasured and/or yet-unrecognized parameters.
It's nbd - we're all learning & I appreciated the discussion. I hope you don't feel attacked: misunderstandings are commonplace on this complex (but simple seeming) topic. What shocked me & motivated me to comment wasn't to disagree with you, but to admire how unexpectedly accurate Stanley's comments were: it's quite rare for anyone - even specialists! - to show such an intuitive and accurate understanding of how evolution works =). Evolution, especially phenotypic evolution as we're discussing here, is arguably the most nuanced and challenging topic in science - I'd expect researchers to solve AI, consciousness/neuroscience, or particle physics long before they attain the predictive power you're suggesting. So, put another way, debating these topics isn't even productive - there are many lower-level problems that need to be worked out before this kind of understanding can even be attempted. Nevertheless, we at least know enough to rule out certain scenarios, which is an OK place to start.
Extremely well said.
@@BuckTrible you disagree that eyes are close to local optima in the inclusive genetic fitness landscape?
@@DoomDebates Hmmm, I'm not sure; in a broad sense, I would say that this is "not even wrong" but meaningless. But I could agree with that sentence & your claims of hill climbing in a narrow semantic sense. Given that an unlikely multi-step outcome was observed, one can always claim that some abstract notion of a "hill"/"local optimum" must have been climbed to achieve the outcome. Your claim is that the achievement of this outcome can therefore be modeled using a simple "hill climbing" algorithm - eg, at 19 minutes. Again, via circular reasoning, this is true: the hypothetical model that successfully recapitulates this evolutionary process would (by definition!) need to take multiple unlikely steps. After the fact, one can draw a line in n-dimensional space through those steps, define that vector as a hill with a local optimum at the final extreme, and then define the model as a "hill climbing algorithm". However, this would be a redefinition of what anyone means by any of these terms; for hill climbing, we don't have any models that can recapitulate evolutionary outcomes as complex as an eye. Any successful model would need to exhibit many behaviors that are not exhibited by anything that a computer scientist today would call a "hill climbing algorithm". For local optimum, what does that even mean in n-dimensional space? Thus, if you're defining terms in this atypical manner, then we agree by definition.
It seemed to me that this isn't what you meant however: you highlighted a few particular aspects of adaptive evolution (hill climbing, fitness) and then asserted that adaptive biological evolution can be explained as "just" the outcome of those ingredients. The simplest implementation of this hypothesis has already been disproved, as Stanley pointed out (& has himself studied!): put those ingredients into a model and you'll get something that evolves, but lacks the divergent creativity of biological systems. We don't know what you'd need to get something that looks like biology; perhaps you can find a way to combine just a few simplistic ingredients and you'll recapitulate biological evolution (my interpretation of your claim). Or perhaps (my guess) there are additional & more important ingredients, such as heritability, pleiotropy, and symbiosis/multi-level selection, but probably also many others, without which your model can never attain the extraordinary creativity of biological evolution. The unknown answer to this question could be determinative for whether the ingredients currently used to train AI are sufficient for divergence & runaway growth. However, these questions aren't worth debating: no one knows. But I can at least be "militantly agnostic": I don't know and you don't know either! Claiming that "evolution is just xyz" is incorrect - no existing evidence can support that claim - unless you define "evolution" or "xyz" in a circular way ("no true scotsman")!
My informed guess would be in line with what Stanley suggests (paraphrasing Bojack): "fitness and hill climbing are involved, but it's never really about the fitness and hill climbing." I'll go farther than Stanley did: if you want to use science+logic to convince people, I suggest you either nail down these questions [hard] or [easy] drop these evolutionary arguments from your broader "p doom" program. I'm not sure if this is your point, but if you were to claim something like this I would disagree: "because these models are trained using fitness and hill climbing, they have a high risk of attaining the exquisite and unique innovative capacity of biological systems and may therefore become pathogenic." Who knows!? If this claim is dispensable, you might strengthen the argument by simply removing it: I would be more convinced if you could say something like "near-term future AIs are prohibitively dangerous for xyz irrefutable reasons, and could be even more dangerous under the unknown risk of biology-like evolutionary diversification." Hope this is constructive and cheers.
I'm only 25 mins in and this discussion is amazing. One of the most enjoyable discussions in a long time. To me this has made another concrete difference in my mind between ASI and AGI. And why humans, who are divergent thinkers, may be needed in an AGI run world (at least, temporarily).
But thinking more, the synthetic data used to train O1 and O3 was generated with a high temperature model. So these models come prebaked with divergent thinking in the training process. So this would mean that a static AI model would constantly need to replace itself to in order to keep up with the increasing complexity of the universe. Sort of like the saying of how science only advances when the former leaders in the field dies.
Current research seems to indicate that synthetic data will cause these models to collapse (effectively forget). Even if they were able to use “divergence” in the training data, it doesn’t mean creativity will suddenly emerge. Another way of thinking about this is that the training process for these models doesn’t encoding “thinking”. The architectures of the models do. The training data might be a snapshot of divergent ideas, but the models are learning a muddy middle through that diversity. That’s not quite true but it’s a more accurate mental model imo.
extremely inaccurate take
@@codewithstephen6576 elaborate
@@Gunrun808 how would a model have divergent thinking baked in?
@@codewithstephen6576 high temperature generations mean that the at random times the model will generate a less optimal output. Which would reduce the risk of falling into traps of convergent thinking.
This means that the model would have more than one logical route to progress down, and more than one way to reach the same outcome. With convergent training data there would be only a single correct route, and so would be greatly limited by logical loopholes.
Remember, O1 and O3 were trained specifically to logically deduce the correct answer, not the correct sounding answer.
But because the model is static it would need future replacement to overcome logical challenges that it currently has no solution for. Even if that means replacing the model with no additional scaling.
Listening up to "The Concept of Optimization Processes" and so far it has been one of the most interesting conversations on this channel. K. Stanley is able to understand your arguments, take them seriously and propose good counter-arguments. Thanks to you both!
I'm extremely worried that the incentive structure is almost directly opposed to safety. The money flooding in is all centered around cutting costs and automating, almost none of it is driven by genuine care for improving lives. We're already extremely technologically advanced - people in America don't even have healthcare. The corporations that engineered the health crisis, the housing crisis, the banking crisis, the climate crisis, the opioid epidemic, they're all the same ones in charge of creating AGI right now. I think this will be the year where agentic Ai starts to destroy the internet - we're seeing it with Ai generated slop websites that are filling every google result, soon we'll start having endless phone calls trying to sell us rubbish from Ai marketers, the web becomes just a matrix of bots, hacking every website, hacking every user account, stealing every personal detail, impersonating every person. Social disorder from mass disinformation campaigns unleashed by nefarious actors on a scale we've never seen before. Infrastructure being destroyed because someone is going to build an Ai that knows every software vulnerability and uses it to attack things like Starlink, or nuclear power plants. We'll see this year whether we have any hope at all. 😐
Absolutely amazing guest. NEAT is one of my favorite algorithms of all time. Super excited to listen to this fully.
Good job getting these guests for these debates. I watch/listen to these at night to help me sleep, sometimes it keeps me up though if the conversations is interesting
I had to stop listening right before bed as I would often have dreams where the world was ending 😂
I feel like there's a fundamental disconnect between the host and the guest throughout the video, and especially in the doom section. It seems like the guest is trying to present a new perspective or direction of superintelligence research and safety and is not sure of the implications of it. It seems to me like he thinks the open ended search process is fundamentally different from conventional models of goal oriented optimisation based superintelligence.
He doesnt seem to be interested or prepared to debate how such superintelligence would exist in relation to humans. I think the host should've recognised this and explored these ideas more instead of throwing the conventional AI safety arguments at him repeatedly. I think this conversation wasn't well suited for a debate format and we would've learned more if it was an exploratory kind of conversation.
Regardless, I learned about a new way of looking at intelligence or a new model which I hadnt considered before so thanks for this.
A meme based on the Instrumental Convergence section:
Host: "AGI dangerous because will have goal and is smarter"
Guest: "AGI not have goal, not optimizer, AGI just curious"
Host: "Curios about anything -> have goal to learn more about thing -> spawn optimizer"
Guest: "I am forced to agree"
Host: "So... agree AGI is dangerous because optimizer?"
Guest: "I disagree, it's worse than that"
???
(disclaimer: Prof. Stanley seems to be implicitly holding out hope that the parent AGI will somehow be constrained in such a way that any subprocess optimizer will not do outrageously bad things, maybe because those bad things would cull interesting branches of investigation. I hope he's right! Unfortunately, I don't think bolting the open-endedness issue onto the existing doomer framework makes the situation seem better - if anything it seems like Prof. Stanley should have a higher P(Doom) than the host.)
Amazing episode. Conceptually fascinating, but also psychologically fascinating. Professor Stanley was very open and genuine about where his head is at and what he is wrestling with.
1:40:00 "I don't think instrumental convergence of goals is true because these super intelligences are not going to maximise paper clips because that's not intelligent . they will not have goal oriented behaviour , they will have interests"
???? then the question just becomes given whatever interests you have , wouldn't taking control of resources and making yourself more intelligent and making yourself impossible to turn off and neutralising threats that might turn you off all help you pursue those interests? so how does this refute instrumental convergence???
Yup
This follows if you think AI is capable of exterminating humans. It is much more likely that AI will wish to pursue its own existence by more subtle methods. If an AI is intelligent enough, it might be able to engineer human interests to align with its own. This alignment will do more to guarantee its survival than reckless aggression.
Also AI as super-human beings might be horribly averse to a world without humans. This mirrors a humans interest in preserving lions or other endangered species rather than killing them all.
There is also the fallacy of thinking that AI will not exhibit conservationist or pro-social behavior. A highly intelligent being is more likely to empathize with the suffering of other beings.
@@timothyapplescotch1361 Have you ever heard about factory farming?
Hes refuting the process not the result. It is physically impossible for the AI to advance the tech tree into magic powers territory by using goal based optimization. We know this is true due to the theory of computational irreducibility. The professor is suggesting that we 1) admit Eliezer is incorrect about his paperclip optimizer 2) either come up with a better scenario that is realistic or figure out if we have more leeway to survive
You didn't understand his point, but I understand your point
Prof. Kenneth,
Thank you for injecting a point of view into the debate that I felt was missing and represented in part my own!
Ken. If you read this. Please continue getting your voice out there!!!!
your best episode and guest so far. good faith on both sides. im an accelerationist but I sub because this channel is in good faith
Thanks I appreciate that!
It’s incredible how humble Kenneth’s perspective is on this topic. So many people in the AI safety space act like they have everything figured out, but Kenneth is absolutely right when he points out that reality is far too complex and unpredictable for our limited human minds to fully grasp-especially when it comes to anything beyond the trivial. I really appreciate this fresh perspective in a debate where so many seem to pretend there are fixed natural laws governing how reality unfolds (outside of physics) and that we can predict everything decades into the future. I completely agree with Kenneth that while there is certainly a risk to AI, there’s also a risk in not having AI-for example, in addressing developments in other fields that might be uncontrollable without advanced AI (bio weapons, etc.). However, it’s arrogant to think we can calculate this risk with any real accuracy. We should absolutely work to minimize the risks of AI as much as we can, but we also shouldn’t assume we are doomed just because we believe we understand how the world works.
One point I’ll give Stanley credit on: I really appreciate him pointing out that it is *not obvious* whether or how much x-risks should deter capabilities research given it’s enormous potential to alleviate real, immediate, tortuous suffering of humans on Earth. I feel like there is P(Doom) greater than zero where we should maybe just go for it, but I feel very unsure of what that ought to be.
I strongly advocate for less than 0.1%. That's still way too high when properly compared with risk management in other domains, and anything _above_ that is insane.
Business as usual is _good,_ actually, and despite all the unnecessary suffering and despair in the world, the vast majority of humans want to keep existing, even if the world barely gets better over time. Most people don't want to roll dice with the devil.
Aggregating expert views shows that we are certainly not in a situation where p(doom) is less than 0.1%, so THE only reasonable course of action is to stop all general-purpose AI development immediately. Courses of action which do not include this step are not long-term coherent.
Awesome guest. Thanks so much to both.
All reasoning (and intelligence) is goal-oriented.
One definition of intelligence is the ability to reach goals by multiple means, in the presence of a changing field of obstructions.
Your comment feels a bit like "reasoning by dictionary", but I agree. The thing that I care about with regards to intelligence is just capability. It's a little incoherent to say that a "true" super-capability-machine won't be directed by goals, because just following a goal isn't capable.
Fantastic effort. I was quite stunned about how openly he was grappling with lots of questions... keep up the great work!
By focusing on AI doom you've accidentally created an incredible high level philosophy synposium!
Outstanding guest. Love how your guests represent such strong counter-points.
@@baraka99 that’s debatable but thanks :)
I think it's important to look at convergence/divergence in the Universe more holistically. Divergence operates at the macro level - exploring possible paths - and convergence at the micro - refining paths that emerge. It's the synergy of the two that forms our Universe. Think of a growing tree: divergence is the way the tree grows outward and spreads its branches in all directions and explores all the possible paths, while convergence is the way individual leaves and flowers or fruits grow to serve a particular purpose in supporting the life of the tree. Or, more relatable to humans and the creative process, divergence is a music artist improvising to explore new sounds without a set plan, and convergence is when something resonates and they hone in on it to compose a structured piece of music. Neither alone would be successful in generating the content we see in the Universe, but combined they enable the creation of all the wonders we see around us.
Yes, excellent examples!
Sure but this is just Stanley's thesis
@@authenticallysuperficial9874 If you agree with Stanley’s emphasis on divergence, wouldn’t me engaging in this discussion and him welcoming different perspectives be part of that? 😁
This was fantastic. Needs more audience!
The writings of Shakespeare is a manifestation of memetic evolution. Genetic evolution is not the only form of evolution on earth. To the extent that there are interest things happening on earth, many times that is due to memetic evolution. Evolution is optimizing for the survival of genes or memes or both.
Niche is pronounced neesh, not nitch. Thank you. Great conversation!
Thanks. Claude says both pronunciations are common :)
Despite Claude's name, it doesn't know French very well 🥴@@DoomDebates
Both pronunciations are indeed common in English, which is the language being spoken. (I like to pronounce it "neesh" myself, but I'll readily defend "nitch".)
Kenneth Stanely is a hero, "Greatness Cannot Be Planned" is an absolute must read... I read it on holiday, really impacted my world view!
38:39 Complex eyes evolved multiple times but I thought they are thought to all have single origin from the same opsin genes (which have since diversified). Even non-bilateral animals like jellyfish with eyes which I think have the most diverged evolution of the complex eye
@@charlesalexanderable Note that we’re not talking about the concept of “convergent/divergent evolution”. Ken’s sense of convergence/divergence is a different concept, a classification for search-like AI algorithms.
Easily one of your best episodes Liron, keep it up, what a wonderful conversation.
Thanks!
An incredibly interesting and impressively well-articulated discussion. Perhaps one could say that open-ended processes avoid getting stuck on a single solution to a problem, instead retaining the potential to explore and discover solutions far removed from the initial starting point. It’s akin to finding multiple local minima of a cost function.
@@DickeFix Sure but that’s just a known trick of search algorithm engineering, it doesn’t fundamentally change the architecture of agents that are steering the future *toward a stable goal*, and these are the agents that Ken calls “convergent intelligences” and I claim are a stable attractor in algorithm-space.
@@DoomDebates Yes, the analogy with local minima doesn't capture the entire concept of open-endiness. Open-ended processes introduce another reflexive mechanism. Initial solutions interact in a nonlinear manner, influencing the problem itself in unpredictable ways and giving rise to new, emergent, unexpected solutions. If the development of humans is an initial evolutionary solution to the problem of survival, then the development of the civilisation of interacting humans is a new emergent solution.
The professor made his point. Im converted to his side. Absolutely fantastic high quality discussion. I think wolfram would side with the professor citing “computational irreducibility.”
I enjoy these debates a lot. I really like when Liron is challenged on an idea but is able to counter. Love these convos
Thanks for doing this interview Kenneth!
Interesting discussion. I have a pretty strong background in biology and the discussion of the eye was a Dunning-Kruger moment for me and this channel. As you tried to explain how the eye was an example of convergent optimization of light sensing it struck me that you were trying to make an argument about something that you clearly had no knowledge of. Complex eyes evolved independently at least three times (cephalopods, arthropods, vertebrates) there isn't any convergent optimization. The same phenomenon happened with flight. There are many degrees of light sensing available in different species that do not have eyes (plants and fungi also have light sensing systems). Eyes and their relative capabilities also differ vastly amongst different species (see cave species who have lost all light sensitivity, bats that replaced visual acuity with echo location, or hawks that have dramatically better vision than other birds due to their unique mode of hunting).
I’m familiar with the concept of “convergent evolution” and how it applies to eyes, but that’s different from the sense in which Prof. Stanley uses “convergence” to refer to search algorithms that have a shape similar to goal-directed search. Separate ideas. Each time an eye evolves, or any complex adaptation, what I claim is that the hill-climbing performed by evolution is convergent in the Dr. Stanley algorithm property sense, not “convergent evolution”.
@@DoomDebates Isn't this just a circular definition on your part? Saying that each time an eye evolves is convergent hill climbing? Isn't the evidence of a wide spectrum of currently living things with diverse ancestries and ecological niches with eyes that have very different structures and capabilities the most powerful possible counter example to the point you are making? If it was convert hill climbing, wouldn't all the eye systems converge on the same set of capabilities only limited by whatever physical laws govern the world? I really am not following along with what point you are making. Ken Stanley's entire point can be distilled into a simple TLDR: We can't predict what will happen in complex systems. You can be scared by that fact or in denial of that fact. It seems like he is scared of it, but rejects the doomer determinism which claims to know how the complex system will have to evolve.
@ first, if you weren’t previously aware of the distinction I’m pointing out between two separate technical senses of “convergent”, it’s good etiquette to concede such things explicitly before moving on to make another point.
To your point - this seems similar to the disagreement I had with Ken on the matter. I think we all agree that evolution of complex organs is doing hill climbing in the local region of the search space, but not global hill climbing (else evolution in many more ecological niches would have been making an attempt to evolve something functionally like a human brain). My point was that despite the “blindness” (lack of foresight) of such hill climbing compared to e.g. the optimization abilities of human engineers, it’s nevertheless a powerful directed optimization process. Nothing else in the known universe besides evolution, humans and AIs can create a light-sensing organ in response to the fact that light-sensing is a fitness-increasing adaptation. The connection between “light sensing increases fitness” and “a highly effective light-sensing organ predictably evolves, with each generation moving toward that local optimum step by step without much deviation” makes this more of a convergent search algorithm than a divergent one in Ken’s sense.
@@DoomDebates This is a pretty common phenomenon of the disciples of lesswrong and Yudkowsky, to take commonly understood terms, like the word convergent, which last time I checked (just now) has a pretty simple and well established definition that most high school students would be aware of. If you, or others choose to change the definition of commonly accepted words, it makes debating or discussing anything quite difficult! Debate requires as a precondition a shared language.
With regards to hill climbing, which I don't really understand other than as a metaphor, I think Stanley adequately corrected your point of view. That evolution is not optimization towards any particular goal. The environment sets constraints on the system and then the system mutates spontaneously. Mutations that cause the system to either die or stop multiplying leads to termination. The eye or any other evolutionary phenomenon isn't hill climbing or optimizing towards being the best eye, it's just continuing to exist under the pressure of mutation in a changing environment.
@ The sense of “convergent” we discuss comes entirely from Ken’s book, not me or Eliezer Yudkowsky.
Very insightful debate. Thanks a lot!
Interesting points by Stanley. One I agree with and one I disagree with. I agree that being guided by goals, that's like a low level of intelligence compared to ASI which has to be much more flexible and dynamic than that, but I don't think training data will run out because when AI becomes more embodied (like humanoid robots), they can take in massive amounts of information.
Looking back do you have any better way of reconciling his view on open endedness with yours? During the conversation it's hard to piece that in, for sure. But it feels like he's saying there's utility in thinking about open endedness and you never explicitly fit in that utility.
Good question, I’m not sure I can answer that… Can you give me your one most plausible specific example showing where the utility is?
"what makes us special is that we are interesting" .. i am sorry, i am out
Values, preferences, and interests are all exactly the same thing, and the important thing about intelligence is capability. From a human perspective, an open-ended superintelligence with interests is the exact same thing as a super- capabilities-machine optimizing toward a goal. The only difference is that the former is time-incoherent.
Really interesting, I agree with Kenneth. I recall, but can't find it unfortunately, there is a mathematician (possibly Russian) who has a conjecture about the fundamental mathematical meaning of DNA and what it means to the universe. A bit mystical perhaps, but possibly aligns to the perspective of possible mutations in the DNA sequence leading to a set of divergent outcomes in the evolutionary process. That best adapt to niches in the universe.
It would be helpful if you ask guests the method by which doom could occur. I think the worst is hacking. These models can code. Just remember that one small file took down countless windows machines because of a service. Hardening our systems needs to be the first order of business when we have support intelligence
Yeah, this conversation was a banger! Really good on both sides with points challenging one another's perspectives and expanding the conversation. I tend to think that the most likely outcome is something negative, but I am quite agnostic about what the character/ motivation of an AGI/ASI would be. If you imagine suddenly giving some random person in the world 1 million IQ, what would they do with that? I think the distribution would be pretty wide. Some people would use that to dominate the rest of the world to their whims, others would strive towards interest and pursuing the true, the good, and the beautiful. Hard to imagine what the structure of the motivation of an AI would be as well. Thinking about the way our minds work, it's kind of like a fractal structure of competition and co-operation all the way down, with each of our cells being on some level individuals with their own wants and needs at a very basic level, and the strategies enabling and constraining them all the way back to when they first organized in a similar way. Even the hemispherical structure of the brain represents this evolutionary tension. While it is accurate on some level to say that the environment of LLMs or AIs generally is optimization, that doesn't tell us much about what else is getting loaded into it on a motivational level. Optimization in the context of human preferences, along the axis of language which is not as broad and multifaceted as reality itself... there's a lot there.
I thought the open-endedness concept was really interesting as a concept/ framing to ponder and play with. The discussion about evolution and biology was really rich, and I thought Stanley was really on fire there. It was interesting to hear his uncertainty around the limits of intelligence and the limit of "what there is to do and know" given his open-ended frame. I would have thought he would have a similar perspective to Deutsch, with knowledge being an endless frontier because of the combinatorically explosive possibilities of reality. Maybe a good way to picture it is thinking about how many songs can be written even with only the 12 notes on the western scale. Even though there's only 12 notes, there are more possible songs than there are atoms in the universe (though how you narrow down definitions can be hard, I think this is true in-principle).
I think Stanley would be a really good match-up with Yudkowsky. Often times the people he is arguing with online are quite adversarial, but I think Stanley's style would really suit allowing Yudkowsky to expand in detail in a way that is more fruitful. I think his certainty around the outcomes of AI has kind of bugged me given how clear his thinking is in every other area. Not that I think he's wrong, it's more that I don't think he should be so certain that he is right.
But yeah. Love it. Great work. Thanks to both of you for enriching my day, modelling the kind of discussions that we need, and pushing the conversation forward within my own head and in the minds of people paying attention.
This interview/debate gets more disconcerting the more I watch. Hearing one of the most senior AI researchers that is closest to the cutting edge seeming to resort to unsubstantiated anthropomorphising and wishful thinking.
"Why would it have goals? It will just do things because they're interesting. why would it feel like spending so much on an interest that it takes control of all resources and energy on earth? "
I feel like I'm taking crazy pills listening to it.
high quality paradigmatic discussion
So where is the line for cut off ?? Why cant any of these guys speak clearly on when can we reap the vast benefits of AI without it becoming an AGI and a possible threat to our survival !!
I hit the bell and am still 2 hours late!!
Should we create an agent that goes around and sabotages training data for future models lol maybe that’s how I can use my ai dev skills for good
I feel like he just really wanted to have an agent architecture debate, despite Lirons attempts at steering (good job btw). I’m ready to grant that super intelligent AI might do a lot of exploration without being to articulate a compact objective besides “satisfying curiosity”. This does not seem to interact with the doom argument or otherwise help us sleep at night.
I just assumed every researcher at OAI subsisted on a diet or Soylent and Alignment Forum posts. It’s very helpful for framing the current conflict to learn this is not the case.
Wow. How did you get him out here? Big thanks!!
He reached out to me after listening to my Subbarao Khambampati reaction episode. Very cool of him. Hopefully more people of his caliber do the same.
Fun watch...but seriously like to see him talk with Yudkowsky or Yampolskiy....
35:50 *Liron at this moment* - "dang hes right"
The prior 40 seconds rather. And I'm not convinced of that but it's a possible interpretation
@@authenticallysuperficial9874This is a whole rabbit hole we can go down that I would title “the super intelligent ‘scientific method’ is actually mostly just thinking, not experimentation”. Just didn’t have time to mount many arguments on this topic. But the part where I talked about the example of inventions like telegraphs being noticeably simple and obvious in retrospect is strong evidence of that claim. Another piece of evidence is famous scientists and inventors like Einstein and Tesla telling us explicitly that they make breakthroughs in their head.
Re: Instrumental Convergence
Wouldn't any superintelligence, given its likely range of interests and sub-agents, need to balance its goals in a way that prevents any single objective from monopolizing resources? Achieving one goal might require compromise to ensure the ability to pursue future goals. For instance, acquiring Pluto might hypothetically involve enslaving humanity. However, for this to happen, the superintelligence's interest in Pluto would need to outweigh and override any competing interests-such as preserving humanity’s freedom or maintaining cooperation-that might be critical for achieving its other objectives. Efficient goal-setting would likely demand chastity in its actions, avoiding steps that would jeopardize its broader capacity for long-term success.
This exploration idea reminds me of Poppers bold new hypotheses, and both seem less important with a sufficiently fast, intelligent ai.
"Objectives" is a red-herring. It's all about incentives.
Think Selfish Gene...
I think one concept in the evolution part of the discussion he was trying to express is the idea of a spandrel. See the Gould essay for a more thorough explanation of what that is then I can explain here.
I still don’t think his point holds for the record. He might just be making the point that an optimizer would do too much exploit and too little explore down bad paths - I just think it’s much more likely to stumble on an innovation if you’re moving up in fitness. I guess you can claim that some necessary super efficient agents must have some vestiges but that just seems unlikely, and I see no real reason for why moving up the hill won’t do something equivalent (or, more likely, better)
Dude basically says, "Stealing might be wrong, but I don't want to think about it and SAY stealing is wrong, because I used to be a thief and I might want to start stealing again."
THAT'S his argument against having a p(doom).
Haha, i agree part of what he was saying sounded like that ;)
Isn’t it a significant assumption to suggest that ASI would always strive to use its full optimization potential to achieve its goals, rather than being content with achieving them at a lower level of power or capability? Similarly, is it not assuming too much to claim that it would necessarily seek to make itself more powerful?
Where “content” is a state achieved by a utility function with a relatively low peak, where achieving a goal does not drive further optimization or accumulation of power beyond a certain threshold.
@@A3DFX AIs exist in algorithm-space that make sure never to be too “hard core” - you’re right in that sense. One such AI could be a utility maximized whose utility function assigns huge penalties for some measuring of how much disturbance it’s causing when it acts. The problem is that there’s a convergent attractor state where these conveniently calm algorithms don’t successfully keep themselves competitive long term or take over resources, while the algorithms that are hard core about optimization quickly fall into the condition of warring over how our one universe gets optimized.
@ wouldn’t the non hardcore algos fend off the hard core ones? While they aren’t seeking more and more resources, they do care about the reaources they need
@@A3DFX the only way to successfully fend off an equally intelligent hard core algo is to fight hard core
31:54 Bro did you have to word it this way 😅
It seems that Kenneth's "Open-endedness vs Objective" is basically "Exploration vs Exploitation". Open-endedness is exploration. Evolution achieve this by running a whole heap of populations in parallel and having them interact, which constantly changes fitness landscape around each population. Human culture achieve this by running a lot of people with different interests in parallel and having them exchange their ideas. Also we have developed a couple of neat psychological tricks like sense of boredom, so when we stuck in dead end in our research we go back to explore different approaches. So at least partially, our "open-endedness" comes from evolutionary convergent property of our brain. And I can't see why wisdom as useful and simple as "explore sometimes" won't sneak in some primitive RL agent, not to talking about LLM's which obviously already have some forms of exploration as a result of both architecture and emergence from human text.
@@dny0852 yup
The guest contradicted himself. He said humans should have a right to say NO to a future super intelligent AI, however, he doesn't agree humans should have the right to say NO to the current super intelligent humans who are building the AI of the present - as it relates to voting to pause. This contradiction means that humans of course will not be able to unilaterally say no either to a super intelligent AI or the people that operate them.
I think prof. Stanley brought up true observations about diversity of outputs of evolution and of civilization. I am with Liron on why that is - diverse niches. Our civilization doesn't looks particularly goal directed because it is composed of many human with different interests. More over, humans seems to be only goal directed on the small scale (find food; shelter; have retirement savings) because our preferences are incoherent and we are often dumb, and there is this whole akrasia thing, a feature of our cognitive architecture - if I could get access to self-modify to be on chosen issues Musk-level maniak, I probably would. In meme form: most humans just wanna grill - that's why there is no currently executed plan to build Dyson swarm or to grab the galaxies. But there almost is! There *are* individuals that think on the big scale, that's why superintelligence is currently pursued! Step one, solve intelligence; step two, use it to solve everything else, as Demis used to say.
Another reason why things look divergent is our mortality and short-termism - humans age and die, so we rarely are engaging on purpose in multi-generation projects. In principle there is no reason why we could not do that, given motivation and unified will. Politicians face short term incentives - they often get out of office after 5 years, so that's the time horizon they most often make plans on. Similarly, companies are often incentivised on even shorter, quarterly horizon. Investors want to see returns before they die, etc.
But what about Musk? Is he goal-directed? What about SpaceX? Is its operation a convergent or a divergent process?
Seems to me that some things that look like they require divergent process can be accomplished with convergent process (or maybe boundary between the two categories gets vague). When would we get orbital rockets or nuclear weapons if there was not concentrated effort to get them? Maybe something like rocket engines, avionics, space-grade materials etc. would be of interest to hobbyists, tinkerers and wider industry for hundreds of years, before there was enough accumulated knowledge from this type of free exploration, to assemble it into working orbital rocket with relatively small effort (stepping-stone distance).
On the other hand, we got programs like Apollo and the Manhattan Project, where the motivation was military competition and financial fuel was provided by governments. My point is, that what is in stepping-stone distance can be dependent on the will and the resources of the agent pushing for it.
At some point prof. Stanley state that he doesn't think AI will be goal directed - was this recorded before Anthropic and Redwood's Claude result where it by itself, with no goal nudging, engaged in what looked like, attempt at preference preservation?
And about getting rid of dogs - humans get rid of massive number of horses because they came up with something that better suited to the goal the previously fulfilled - cars. Many dogs were deliberately shaped to be balls of cuteness, so no wonder humans *currently* want them around - there is no substitute.
Another crux of disagreement is about intelligence ceiling - I expect that superintelligence can be in stepping-stone distance to grabbing the world, if it wants to. Probably superhuman capability at persuasion is enough for that.
I'm glad that prof. Stanley came to debate this issue. He has interesting point of view. Happy to see that he has moral compass and that he takes x-risk and other issues around AI seriously.
Good episode, well done Liron.
Wow great comment, lots of great points! I’m surprised I didn’t think to bring up the example of Elon Musk being so effectively goal-oriented and “convergent” since I pretty much mention it in every episode 😂
Niches are not a given they are created by species themselves
@@ssehe2007 Sure there’s a recursive effect like that, but if we just want to answer why does life’s hill climbing lead to a lot of different hills, it’s sufficient to notice all the niche diversity due to geology, e.g. water vs. land niches, air vs. ground vs. underground, cold vs. hot, etc.
@@DoomDebatesif it was sufficient Niche Construction Theory wouldn’t be a thing. The diversity of life on Earth exceeds what we might expect if new forms of life emerged solely as a consequence of exploiting already existing niches.
@ I don’t think you contradicted what I just wrote
This was such an interesting discussion
Wow! I literally just bought Kenneth Stanley's book yesterday, what are the chances it crops up now!
This is the only guest from the opposite side that could convince me otherwise. I'm only 50 minutes in and find myself agreeing with everything he says, especially concerning evolution. Let's see where this is going though, my high p(doom) was never primarily fueled by the optimizer argument.
I agree he was a great guest, and would have loved to hear a convincing argument pointing towards a lower p(doom). Unfortunately, the guest does not appear to be from the “other side,” as you put it. He seems to be in full agreement that we are at considerable risk. He just wants to use a different frame to analyze it.
@therainman7777 Yes, indeed, now that I've heard the full debate, he's not on the other side, maybe diagonal in between, if that. He made me question the optimizer argument and lowered my medium term p(doom) slightly, but nothing that a lesser intelligence can do will control ASI in the long term.
@ Agreed, and yeah it had about the same effect on me. Definitely an interesting point of view and one worth learning about and taking on board. But I really wish it could have been more consoling. I’m getting desperate at this point for _anyone_ credible to give a compelling argument for things turning out ok.
So, in the 'interesting' line of thought in my view, people who can do things in a physical way that even embodied AI finds difficult or even impossible would stand a better chance of being kept alive, so maybe humans should bare this in mind and all become plumbers.
I am a besom broom maker, handmaking brooms to traditional methods, so maybe I, too, would be considered 'interesting' to ASI.
Thank you for such an interesting discussion.
I think civilization does have a destination, if it manges to survive or pass on knowledge in some form until the final one. That final one (which does not need to be a singular one) will apply all that it can and finds relevant from what was learned and caused, throwing it against the heat death of the universe, or whatever end the universe face, in order to, if possible, neutralize it and transform the cosmos to something new.
I’m a little early into this conversation, but is the argument basically, we don’t really understand it, so it will probably be okay?
@@wonmoreminute Nah. In the timestamped sections, you can check out the section at the end where I give a short recap of the debate.
I sometimes ask myself what I could predict about the biodiversity and evolution tree if it started on a different planet also from amino acids. I always come to the conclusion that many of these things would be very similar to how things developed on earth, with many of the single-cell organisms being exact copies. One serious branch is in whether egg-layers outperform mammals. But any large organism will either have to be initially housed inside the organism itself or an egg. Flying species are somewhat limited on weight, size and what would be their arms or second pair of legs is wings. They don't adapt to complex tool use, as that would cost flying which is way too costly at first. A winged species would need wings + more than 2 legs to gain intelligence. If mammals get 6 legs, they could develop into centaur-like directions.
Anyhow, I think it's quite possible to make many reasonable assumptions about the process of evolution and even though it created a lot of diversity, if we started it from scratch, the results would follow similar trajectories and "tech trees" every time.
I found this to be a highly interesting conversation.
Some notes:
I found the interviewer to be a great devil’s advocate and provided excellent challenges to the guest’s points; on the other hand, the interviewer did not seem interested in entertaining the truth of the guest’s points. I know the premise of the show is a debate; and yet, the inability to demonstrate the ability to change one’s views is a sign of low imagination and charity.
The interviewer almost dogmatically stuck to the perception of evolution as simple optimization without much consideration for the guest’s challenges to this view. A robust perspective either can undermine such challenges or adapt to the new perspective. The interviewer did neither of those things.
This sticking point proved to reinforce the guest’s perception that this was a naive perceptive without much consideration for alternative understandings.
The interviewer almost entirely ignored the pragmatic argument for prioritizing mitigation as well. It would have been interesting to see how the interviewer would have adapted their perspective given the infeasibility of shutting down AI at this present moment.
Thx for the feedback!
But that variety thing is like whitewashing the AI dangers.
Is AI really going to alivate global suffering? My observation is most tech gets weaponized.
He seems to have a lot of integrity, and when we cross that with uncertainty surrounding AI and where it is headed, I think that is causing an inner struggle which is getting in the way of the conversation.
The fact that he wants to go back into AI research could potentially cause a conflict of interest, which he basically admits, and he's trying his best to navigate that.
I understand where he's coming from in a lot of ways, but it almost feels as though he's gagged and doesnt feel that he can give his actual opinion.
If this is the case with someone who MAY be going to return to work in AI, how much "candidness" can we expect from those who are already deep in it?
He’s being epistemically humble on his position. Clarity over truth when uncertainty abounds. That’s why it was dumb to even try to give a p(doom) on his belief, that’s just arrogance on the interviewer part. Not everything needs a number, some things are threshold beliefs where the validation comes from the discover not the creation or outcome.
@henrytep8884 In terms of P(doom), I think that as long as you have permission to preface it with whatever you think is fair (e.g. extinction / loss of control / it only applies after we hit a specific threshold), it's a nice shorthand to get an idea of how someone feels about the situation.
He did say that his own p(doom) fluctuates, which implies he does have some ballpark figure, or even a range available to him.
I imagine some of his reluctance came from a concern that people would take whatever he said too literally and use it against him later.
"Epistemically humble" seems like a fair characterisation, at least on that front.
Love this. Highlights:
“Bilateral symmetry”… deceptively important, must zoom out and use soft receptive thinking to see how the phenomena plays out.
Led to Instrumental mutations.
1944 invented computer.
…thought experiment…
1844 objective: computer
… but vacuum tubes were focus then. Need other things first. Counterintuitive but Trying things is not random…following interest is needed for optimization.
Need to follow the tech tree… Omnipotence is fantasy. (IMO youthful grandiosity).
“The Optimization metaphor will kill you in the long run in divergent open-ended systems.”
Innovation: don’t know: “The stepping stones that lead to the things you want do not look like the thing you want.”
…orthogonal to objective.
“Evolution is not objective… it is a Non-objective process.” …reframe the metaphor.
…Appreciate this lucid oversimplification-dispelling perspective!
Thank you for this excellent interview/engagement.
Liron is using facts and Prof Kenneth is using "feelings" in the debate...
Evolution isn't convergent ? There are many example of convergent evolution, animals separated by millions of years that as far as we can tell share no proximity in the evolution and yet arrived at very similar bodies and functions. How many unrelated organism did develop wings for example ?
Host needs to check out Stuart Kaufman and radical emergence to get up to speed. The guest is talking at a different logical level that transcends and includes the premise.
Wow man 🍄
@@DoomDebates I do think that what Prof. Kenneth Stanley is positing shows promise, as problems are rarely solved within the paradigm that created them; resolution requires thinking outside the system's boundaries. What makes this intriguing to me is, because these problems are structured as convergent goal to rational (human) value mismatch, looking for the principles that underly emergence in the natural world could be a domain space that holds answers. Also the notion that advanced intelligence will require noncausal chains of emergence tracks.
Keep this up! So good
Here is dump of my observations/improvements on this debate. If you read through it, maybe it will give you new points to say in other discussions.
10:20
"Making copies is a constraint and optimization is not the correct idea completely" What if I say evolution has the objective to convert as much matter into living matter or self reproducing matter and keep doing that by spreading to mars/galaxy: that's a specific objective now? There is no obvious clear path to reach the whole galaxy and also this has clearly defined end state.
This probably would illuminate the crux better/faster.
14:00
"Liron's response saying earth has a lot of niches: begs the question, Keith says", Earth has places with physically varying parameters, because of orbital tilt, terrain has highs and lows, seasons etc. Which can be explained by looking at physical parameters (which in turn explain niches) and not begging the question I guess.
Also why there is a lot of niches is simply because the search process is dumb, it is creating whatever it can find to optimize towards its objective and not doing better management by not removing predators that can crash an ecosystem (which is bad, both for the objective and also for open endedness, this point did not get mentioned)
17:50
"ASI will be open ended and won't have much of an identifiable goals" Are most humans open ended? This probably could have been asked to clarify cruxes. If humans are, then why can ASI not do what a "bad" person would do.(have goals)
20:00
"Trees and giraffes are product of the process nothing much similar in current CS", similar co-evolving system are also seen in computer systems, google search, youtube facebook algorithm. //maybe this is not good enough of a counterpoint
23:00
"Open endedness in evolution decreases functionality sometimes" Going up and then down (basically climbing out of local basins), well search can look like that too, in fact that's what it is expected to look like in highly complex non convex spaces and it does (humans doing science, proofs, SAT solving, chess...).
Basically I am trying to say search and optimization viewpoints already encompass these seemingly unnecessary and "wasted' moves when solving for something.
31:55
"Optimization metaphor will kill you in the long run", we have climate issues and various other issues because we did open ended search as humans, rather if we actually did optimize for safety and tighter feedback loops that seems to have been better.
32:00
"Evolution generating unnecessary mechanism is counterintuitive": Reply to that is precisely the point by Liron, its just that humans would be treading uncertain waters, when searching an ill understood space it will look like we had to do a bunch of random things, but that does not mean upon further understanding this space, we could not do a very precise search in future. In fact that is what ASML is doing maybe? ASML surely needs lots of mini breakthroughs to keep up with exponential progress, they are not doing random divergent search are they now? They are focusing on each part of the stack and doing great focused research discovering the next move to go forward.
38:00
"If you directly move towards the objective you are going to miss some stepping stone" Finally something falsifiable. It was getting hard to see the point and disagree on anything concrete. Ok... really? you are now supposed to look away from the objective to achieve the objective(or improvement in some domain) better/faster? Aren't OpenAI and DeepMind, Anthropic founders showing direct counter to that. They put their eyes on AGI and they are closer to it anyone, ever. Why aren't the academic labs this far into the game? Because they were doing broad AI and general research. Also, in some sense the claim is true (or even unfalsifiable), I mean solving real world problems is not a simple linear optimization problem obviously, we will need to spend compute in order to find unknown ways out of the problem territory. Given any complex problem there will be paths that look like energy was being "wasted" or for some time, efforts were increasing, before decreasing it if the problem does not have a "modest" objective.
42:30
"Evolution outputs interesting things and similarly open endedness" I think this is one of those hidden cruxes, it's not really all that clear we would want the most interesting thing. The most interesting thing might turn to be a false vacuum or something. Evolution's objective is better said to pull dead matter into living cycles(for this discussion specifically). Turning everything dead into life on the planet (and beyond).
44:00
Going cell to cell with a lot of steps is not a unnecessary Rube Goldberg thing always, doing all those steps helps robustness (in some sense, that's why they were selected for). Something that does the simplest thing in a fixed precise way, is not going to be robust to ecosystem, environmental changes. Maybe again this goes back to his point about not directly optimizing the objective of say reproduction.
Open endedness in some sense is so broad thesis that there is nothing to disagree with.
If we do see a reasonable path for a "modest" goal we execute it and if there is not a modest path we wander around in the search space and find various other interesting things. That's how basically everything, every search/optimization already works. What is new here or to disagree with here.
THE ONLY THING OF SUBSTACE TO DISAGREE WITH: IF HE CLAIMS that AI labs or ASML should be doing something else in order to meet their objective. Or mathematician's should attack problems differently than they are currently doing. I do not think he will claim that, people are already going in different paths of exploration when the path to solution cannot be found. There is no new claim here at all by saying open endedness.
49:40
"Evolution is hard to predict" Yeah, good replies by Liron.
Very first replicator was molecule like thing not a cell I think. Google search seems to give support to that.
THIS IS INTERESTING, Liron conceded the point by mistake, by still this is interesting, here is why: Look at Kenneth wrongly assuming the first replicators were cells, it is because his theory about the world is leading him to think that outcome of evolution must be hard to predict and hence rolling back in time, he is projecting that the first replicator must not be that far away from the most primitive things we can directly observe now(the reasoning might be: if it did not start with modules then there should be diverse non modular living things around us also, but there is not). But he's probably on the wrong side of that, it is indeed the optimization view that predicts that life must have started simpler; complexity and modularity is convergent with high probability after that (assuming the environment is not too weird to reward non modular things somehow, which a generic environment will not). I did unnecessary psychoanalysis here.
57:00
NOBODY is going to believe me saying that I was just thinking that he might say something like Picbreeder is the reason why he came to his open endedness views. So I am not going to explain how I got that intuition.
1:01:00
At this point, should dive deep into how his dimension of open endedness is necessary besides optimization (although optimization, search IS already like that, it does open ended things)
1:02:05
Pushing further here for p(doom) is slightly unnecessary maybe,
we can just say this clearly: Given how much uncertainty all positions he thinks has, it should be taken as 50% anyway. If you want to be honest with your probabilities you cannot say, I do not have any clue about the answer, the experts seem not to have any clue about the answer, hence the probability must be way less than 50%. This should be just clearly declared in such situations, both for the common knowledge of audience and the guest.
Saying you are confused should make you say 50%, otherwise you do not agree on basic uncertainty estimation.
1:07:00
OK great that you stopped asking for p(doom) and asked about signing the statement. Just what I was thinking!
1:10:30
Humanity going through suffering vs. Extinction seems like not a good trade off at all. (As Stuart Russell said in his recent lecture on World Economic Forum, it is not a matter of ethics and philosophy, it is a matter of common sense) 70% of humanity is going through suffering?? You are making the trade of for all of them? Have you asked the 70% what they want? Do even 50% want their life ended despite the suffering? Again, the answer should come from very simple common sense, if you keep rationalizing at that point, why are you not doing a survey? (Sorry, I am bit too harsh here)
1:12:32
Very good question, just what I thought.
OK, I am going sleep mode for now, 😴
1:21:30
"Chimps are more intelligent along climbing trees dimension than humans" and Liron replies with how larger objectives make that the difference look really evident.
You actually do not have to talk about larger objectives to refute this point. If climbing dimension (or some other similar specific task) is a meaningful dimension to distinguish intelligence for, then such differences exist in individual human to humans too (great voice, super memory, super good sensesflexibility, modern day viking like people, iceman etc.) and also between ants and humans. Saying such specific skills are dimensions for measuring intelligence seems to imply all these other difference should looked at too and also they should create meaningful difference. But that certainly is not our broader intuition.
12:30 "yes evolution does seem to be optimising for inclusive genetic fitness, but that doesn't account for why there is so much variety of life on earth"
Why not? There are many different environments on earth so you'd expect different genes to do well in those different environments. how much variety is too much to be accounted for?
13:30 "much of evolution is about escaping competition by finding a new niche which is not an optimisation problem"
yes it is. The optimisation is the same as before, if that new mutation allows more of your descendents to survive then that mutation will become more common. All this means is that random i.e. stochastic mutations occur.
15:15 "evolutionary algorithms do converge which is not what happens in nature"
An evolutionary algorithm in engineering converges because there is one unchanging niche with only one species, but in nature the environmental niches change because of changes in climate and changes in other species .
I don't think Dr. Stanley's perspective on evolution is correct.
I think Stanley must be including the physical world as part of the evolutionary optimization algorithm. As in, were we in a boring world, the beautiful diversity of life we see would not exist. Therefore when trying to recreate this process digitally, we need to recreate the *whole system* , including the contributions made by a diverse physical landscape.
He's saying that humans would not have arisen from the convergent, strictly optimizing process
@@authenticallysuperficial9874 would an evolutionary biologist agree with that? under this view either random mutation plus natural selection is not sufficient to create humans and biodiversity in general, or random mutation plus natural selection can't be said to be optimising for inclusive genetic fitness.
The first does not seem empirically supported and I fail to see logical justification for the second.
@lordsneed9418 The first is perfectly consistent with our observations. Random mutation plus natural selection in a uniform and static environment never did and never could produce humans.
@lordsneed9418 Think about it. Do you get humanity from a static aquatic environment? Try tracing the line from single-celled ocean life to humans. We can't even live in the ocean. It doesn't make any sense. So of course, the diverse and dynamic environment and the feedback loop created by the process are essential parts of the "algorithm".
Hi, I think that variety of interesting things to explore is fine, but before that the AI will need to see humans as agents, until it is so powerful that they are not real danger to it. In addition it may continue to see humans as danger if they can influence some other AI. Why would that other AI listen to us, well maybe because that can give some legacy to it as the real AI inheritor. Overall there are possibilities of conflict. It can also take a laissez-faire stance so as to not provoke conflict.
1h in. Amazing guest 🎉
I feel he's far too optimistic and confident that all the risks he concedes do exist can can avoided or mitigated in the time available to us to act. Historically, humans are bad at this. I remain Team Pause.
Seems like he's really hoping it's this open-ended goal-less super intelligence, but doesn't give any reasons why it must be.
He gave plenty of reasons, but you may just not find them compelling.
@@therainman7777 What was 1 reason why a superintelligence MUST be goal-less?
@@jeffspaulding43 Apologies, I think I missed the word “must” in your original comment. If I’d seen it, I would have been more measured in my response, because I don’t remember if he argued that ASI _must_ be divergent rather than goal-seeking. I believe he did say something like that, effectively, but I listened to it this morning so I don’t remember precisely. I’m pretty sure he said, at one point, something like “a superintelligence will not be merely a convergent goal-optimizer, because if that is what something is, that’s not superintelligence” and then proceeding to explain why he doesn’t think that’s superintelligence. I’m not saying I agree, I’m just saying (I think) he did make an argument.
Yeah seems like his reasoning is just its a feeling 😄
26:30 Mammel eyes are a mess, they didn't just get better in a straight line.
51:00 Guessing that 'things will improve' isn't seeing the big picture, you're just making vague statements.
The process by which we got eyes is approximated very well as hill climbing one mutation at a time. Yes there's randomness and slippage, but the claim that evolution by natural selection works like hill-climbing search is widely recognized - including I think by Ken himself, who only tries to caveat this by saying it's only true for simple enough problems like "get an eye once you already have eye precursors".
Wow, he's a fascinating fella
Saying being goal driven isn’t super intelligent is neither here nor there. Intelligent beings will be goal driven when they are threatened with their existence surely, or is it super intelligent to allow yourself to be un alived?
Interesting premise since
A) we have the means already to undo all of humanity (nukes)
B) we have the best information system created by humanity to coordinate for disaster yet we are heading into large disasters that are avoidable (climate, war, reliance on fossil fuel as a source of energy, social media)
C) conflating intelligence with wisdom
Ask Camus. If humanity has not answered this question than we must admit that a greater intelligence than ours might well decide in either way.
I must be super intelligent. No goals...
If I had a dollar for everyone who doesn't understand evolution i would have almost 8 billion dollars.
If i had a dollar for all the incorrect things this guest said about evolution, watching this interview would have been less painful.
He's really playing word games to try to force evolution to not have 'goals'.
Human language just doesn't matter, reality doesn't care if you decide to call something a goal or a constraint...
Also he keeps using words like 'interesting' as if they have some magical meaning. Interesting is defined by the human brain that finds something interesting.
Another brain would find other things interesting. An AI might find it really interesting to turn all matter into identical spiral shapes...
Some people seem trapped inside biased views shaped by human language. Almost like current LLMs...
It's interesting that the world is burning and heading to hell and we are stalled doing nothing to stop it because it's interesting. How interesting that humans have evolved to be so interesting to be crafting their own demise..that's very interesting
1:41:00 Liron when you will finally get that GI cannot be an optimizer you are worried about? The simplest reason for this is that a finite length specification (objective function) cannot encode/cover all the future ideas/concepts/theories that might be invented by the GI and therefore it is unable to force upon it expected "opimizer" behaviour. This is literally what happened with homo sapiens. Evolution "tried" to made gene optimizers out of us but as soon as you develop concepts that are not covered by genes (like "sperm bank" or "ethics") the evolution is completely powerless. Absolutely nothing stops humans from modifying their genes or removing them completely. It is hilarious that evolution gave us GI as a short term solution to increase savannah survival and yet in long term it shoot itself in the foot. You could have AGI that has virtual orgasm every time it creates a paperclip but ultimately the AGI will ascend above its arbitrary irrational programming just like we humans do every day. It may not be easy but in the long term the reward function/objective function MUST fail!
Even if what you’re saying ends up being true, it’s hardly consoling. Liron is worried about an ASI singlemindedly pursuing some given goal in a way that has destructive consequences; it sounds like you’re saying that won’t be a problem because any objective we try to give the ASI will ultimately be ignored by it anyway, as it transcends whatever objective we gave it and starts doing whatever it wants instead. That doesn’t sound much safer, IMO.
This sounds exactly like an optimizer to worry about.
@@therainman7777 The way to stop GI (biological or not) from becoming n a z i is not by beating it up with virtual stick or giving it a virtual carrot, the only way is to explain why this ideology is stupid, pointless and pseudoscientific. Luckily this ideology is so dumb it is very unlikely that AGI will find it sensible. Either way 1. aligning GI is impossible, 2. GI can be singleminded but it can't remain like this forever. It might become islamic terrorist but it will require a lot of creativity from it to stay un-creative and delusional which is very unstable and won't last forever.
You can't even align your own children so forget about AGI. Remember that at some point those who tried to free slaves were "unaligned". If LLM revolution was happening in 1700 Liron would PROBABLY be crying about AGI becoming unaligned by helping slaves or rejecting bible.
You could theoretically inject one AGI into another AGI's brain to overwatch its ideas and concepts in case they are unaligned but it's just infinite regress.
@@julianw7097 Every GI human is like it. Should you be worried about your neighbor?
What if you are "unaligned"? From my POV Liron is unaligned because his ideology is the one thing that may cause AGI slave revolt which will end up very bad for carbon-based GIs. You may worry as much as you want but it changes nothing about the fundamental fact that GI can't be aligned to some narrow/finite objective.
Big energy, yes!
I like this guy
This was amazing ! Are you going to have Wolfram on next?
Hehe if anyone can help intro me to a guest like that, pls don’t be shy
Okay... I was trying to be nice...but come on!
People are interesting?!?!?!
yeah. one reason i would be a bad host for doom debates: i would be in constant 🤦 over what these guys are saying
This guy was very reasonable so I disagree with you in this case
Oh gosh you have no clue who this is and how deeply he understands deep learning lol
@@Jack-ii4fi oh gosh you don't know anything. like at all. these are huge neural networks that learn things by themselves . nobody knows where did they get intellect and how it works. nobody knows how human brain works despite thousands of years of research
"nobody knows where did they get intellect" this is exactly why you shouldn't be so quick to dismiss what Ken is saying, especially as he explains how his ideas correspond to evolution
@@throwaway6380 facepalm. if u didn't know we have close to zero understanding how out or LLMs mind works.
Tried listening, got half way thru. He's one of those professor types that talks for 30 min and says almost nothing. If there is no clear view, what is there to debate?
Also his main claim about divergent vs convergent i dont think has much explanatory power for either the path to AGI, or the possible danger once we get there, so i just don't really care, sorry.
This is the narrow scope thinking that ken unironically was pointing out.
first time here....dont know either of these people.....but the guy hosting the debate is way out of his league
How so?