วีดีโอ

"Formal verification is overrated" says the guy making a living from almost impossible to formally verify systems 🤷‍♀

This is a really interesting framing but all the more reason to slow down or pause, if both risks are similar, human caused risks are still closer to coming into fruition

Wow, this is cool

Interesting & informative?

Finally, you argue that "'tool AI' is unstable + uncompetitive". What I think Max Tegmark and others are saying here is that we don't need AI systems with *agency*. That is, they don't need their own goals and the ability to act in the world to achieve those goals. Any problem that humans want solved, can be just as well solved by a powerful AI which only accepts formal problem specs and provides solutions and formal proofs of those solutions. We certainly don't want to eliminate machine learning! We just want to eliminate AIs running rampant over the human infrastructure. You suggest that it would be "unstable" in the sense that it's trivial to convert a powerful tool into a powerful agent. I totally agree and have been giving various talks on how to deal with that. It's a simpler problem if powerful AIs require trillion dollar data centers which can be constrained by government regulation. Unfortunately, recent results like Microsoft's "rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking" arxiv.org/abs/2501.04519 seem to suggest that even tiny models that can run on cellphones may be powerful enough to cause great harm. I believe we need to create a trustable infrastructure which cannot be subverted by powerful AI (using techniques I describe in my talks). But right now we're in a race between using AI for capabilities and using it for safety. Formal methods are a critical tool for creating a trustable infrastructure but it looks like the time for effective action is very short now.

Your second point is that "reality is complicated + unknown". I agree with that in general, but here we are talking about *designing* systems in which have high confidence of safety. Every engineering discipline already deals with that and has developed techniques and models which simplify the problems to the point where humans can solve them. Semiconductor specialists create design rules which provide strong guarantees that their chips behave as desired. For security, we need a model of the adversary. If the adversary can only attack through the designed digital channels, we only need formal proofs of security at the software level. Physical attacks require models of the physical device and proofs of security against a class of adversary which goes beyond today's common practice. You mentioned our example of DNA synthesis machines. We certainly weren't proposing some kind of automatic system to determine which DNA sequences are safe for humans. Rather, we assume a "whitelist" of allowed safe sequences (determined as they are today by a biology safety board). The safety challenge we intended was the much simpler task of ensuring that a rogue AI could not synthesize any sequence not on the list. Today's synthesis machines allow anybody to synthesize anything. We argue that as AI becomes extremely powerful, that is inadequate and that we need a "gatekeeper" between the AI and the synthesis machine which checks for allowed sequences. Doing that in a way which can't be physically subverted is challenging but my video above describes a powerful class of techniques for that.

This seems great but also like it could be really hard in the complex models

This was about AGI?

This was very informative

It was nearly over my head but she explained things pretty well

The AI revolution is a humongous hot mess, and most people on the planet will suffer because of it.

AGI is weird ! If only one can understand the latent space better.

I don't like this 5 min things :/ should be at least 10, better if 15. With like examples and stuff to make me want to get interested in your beautiful research

Evan's high points of vocal pitch are up there on a par with Charlie Day, or Justin Roiland as Morty Smith.

WHY are SO many of the cool people into SAEs these days???

The idea of focusing on second best solutions is really striking to me, as it does seem to me that a lot of our current messes in society is around that focus on going to the first best solution all the time.

For those wondering, they already published the video where they talk about that last example: th-cam.com/video/9eXV64O2Xp8/w-d-xo.html

# Richard Ngo - Reframing AGI Threat Models [Alignment Workshop] ## Key Takeaways ### Main Argument - Traditional distinction between AI misuse and misalignment may not be useful - Misuse: Humans using AI for harmful purposes - Misalignment: AI autonomously deciding to do harmful things - As AI systems become more agent-like, the distinction becomes less meaningful ### Technical Perspective - Technical solutions for preventing misuse and misalignment often overlap: - Monitoring systems (AI behavior vs. user behavior/interactions) - Detecting behavioral changes (deceptive alignment vs. backdoors) - Communication concerns (AI steganography vs. jailbreaking) ### Governance Perspective - Both misuse and misalignment risks vary significantly based on the actors involved - Example of bioweapons: - Often discussed as a terrorist misuse problem - Most dangerous bioweapons typically come from state-level actors - Different actors require different mitigation strategies ### Proposed Framework - Introduction of "misaligned coalitions" concept: - Groups of humans and AIs attempting to grab power illegitimately - Can include various actors (terrorist groups, lawbreaking corporations) - Leadership within coalition (human vs. AI) may be unclear or irrelevant ### Risk Spectrum - Risks range from decentralization to centralization: - Small-scale actors (decentralization risks) - Large-scale actors (centralization risks) - Political dimension noted: - Democrats tend to focus more on centralization risks - Republicans tend to focus more on decentralization risks ### Call to Action - Need to avoid splitting AI safety community along these divisions - Importance of developing unified frameworks that address both types of risks Note: The provided subtitle content indicates it was incomplete, but this summary covers the main points from the available material.

Is it just me or this is a really big deal? The examples were fascinating! I will make sure to look more into it

31:40 An example I thought here is that if you feel lonely and are very online, the AI might create online friends to interact with you and support you, without you knowing that they are AI

Isn't this inherently going to have a limit and not work for an AGI? Wouldn't a smart enough model just be able to game all of these and any other test that we come up with?

E

L bill bill bill bill bill bill l bill bill bill l bill an m an ch m am an n an bill l am an m an ch ch k ch n bgm k bgm bill m ch l

J cup chn bgm chn chn bgm chn chn chn chn by chn by chn chn bgm chn n co am an m an m am an n an n am bill bill an m am an bgm m am an m am m am an ch n an m an n an mk an bgm m am

😊😊

Vbhhhhb❤❤❤❤❤❤❤❤bcc

Misaligned audience.

😊😊😊 kl😊😊😊p😊000😊p😊

😅

Hi sir or madam

Nice talk! I disagree that adversarial robustness has only one attack and differs from other computer security in that way. Once the simple PGD attack is solved in a tight epsilon ball, you still can’t say there is no adversarial image that breaks the model. Enumerating all possible attacks is still very difficult/ impossible for now.

my fav boy 😅

explain the notion cleanly, thanks for helping me understand!

thanks for uploading!

FAR•AI, great video it was really entertaining

was the no Spieltheorie in this talks

The kind of safety he advocates is impossible to achieve. Imagine a lever that can be put in either of two positions, i.e. we have 2 posisble actions. It is impossible to say if either of these actions are safe or unsafe without knowing all the secondary effects. That is you need to look at the system the lever is connected to to determine if either of its actions is safe or unsafe. Same with models. Unless you know the full system it is not possible to know if a models response is safe or not. This paper is a load of hogwash.

Why are doomers so afraid of intelligence? That's quite telling

If it’s superhuman and robust would that make it omnipotent? 😅

17:00-17:08 🙃

The blue triangles appear to be white

Ilya please stop working for a second and do another talk like this one

Mainstream? There are very few actually training models (first token movers?), with a large open-source pool around them (each wanting to have their own flappy bird moment). Perhaps he is hinting that "guardrails" has been a downstram process up until this point? *also an ontological failing to suggest ML alignment has anything to do with AGI alignment, but whatever.

um.. "SA" is generally understood to be about an awareness of the physical space around the body (1st person view; can be you in a room or you behind the wheel of a car, etc). Just within the first 2 minutes, it's being described as consciousness with access to how that consciousness functions... so.. artificial superintelligence... from an ML model.... *sigh*

Future in the making...right here. Its happening right here. We r greatful witnesses.

Great talk…

The dynamic you are describing is an arms-race. Trying to stay out ahead of the AI is a losing battle; all of these clever hacks are not going to get you anything more than a bigger crash when the house of cards collapses--unless you actually fix the problem.

All caucasian researchers on this topic, amazing!

Most important part: good people may sparks the bad usage of AI, not to say the evil people.

Why alignement is not trivial --> We will solve AGI alignement in 4 years