Superintelligence
Note: The not ELI5 version is Nick Bostrum's Superintelligence, a lot of what follows derives from my idiosyncratic understanding of Tim Urban's (waitbutwhy) summary of the situation [0]. I think his explanation is much better than mine, but doubtless longer. There are some humans who are a lot smarter than a lot of other humans. For example, the mathematician Ramanujan could do many complicated infinite sums in his head and instantly factor taxi-cab license plates. von Neumann pioneered many different fields and was considered by many of his already-smart buddies to be the smartest. So we…
This article brings up an important source of bias that tech people risk - that we overuse models from programming when thinking about other aspects of the world. We should be learning alternative models from other subjects like economics, philosophy, sociology, etc so that we can improve our mental toolbox and avoid thinking everything works like a software system. I'd say that another related source of bias is that we are surrounded by people who think like us. It's a shame though, that the article dismisses without much explicit justification risk from artificial intelligence and the…
This post is basically a repackaging of Nick Bostrom's book SuperIntelligence, a work suspended somewhere between the sci-fi and non-fiction aisles. As a philosopher of the future, Bostrom has successfully combined the obscurantism of Continental philosophy, the license of futurism and the jargon of technology to build a tower from which he foresees events that may or may not occur for centuries to come. Nostradamus in a hoody. Read this sentence: "It looks quite difficult to design a seed AI such that its preferences, if fully implemented, would be consistent with the survival of humans…
Notice that, in Robin’s scenario, the present epoch of the universe is extremely special: it’s when civilizations are just forming, when perhaps a few of them will achieve technological liftoff, but before one or more of the civilizations has remade the whole of creation for its own purposes. Now is the time when the early intelligent beings like us can still look out and see quadrillions of stars shining to no apparent purpose, just wasting all that nuclear fuel in a near-empty cosmos, waiting for someone to come along and put the energy to good use. This presumes that The Most…
Then, Sir, I encourage you to get thee quickly to this Partially Examined Life podcast episode[1] with Nick Bostrom[2]. Only came out on the 6th of January and it makes for super interesting listening. It's based off of the text with his 2014 book in which he's gone to great lengths to think philosophically about this very existential threat -- in his opinion, man's most pressing. Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making…
I think I am in quite a similar position to you: after secondary school here in Ireland, I didn't consider any universities outside of my home town (pretty much because I didn't know a single person who was considering bigger and better options, so it genuinely didn't cross my mind to apply to Stanford, CMU, MIT etc.) so I ended up going to a fairly average and not very well known university to study electronic engineering for my undergrad. I went on an exchange for a year to UCLA and this was when I started to feel something similar to the sentiment you're expressing here. I'm now in my…
I read "Superintelligence" by Nick Bostrom, essentially on the recommendation of Elon Musk (he tweeted about it). It talks about the dangers of strong AI and possible paths to it, and how humans can mitigate its effects. The only reason I read past the beginning is because in the preface he says: "This book is likely to be seriously mistaken in a number of ways". So at least he's intellectually honest. I believe he's building 300 pages of argument and analysis on a flawed premise. As far as I can tell, the entire discussion rests on what he calls "instrumental goals" vs. "final goals".…
Bravo Tunesmith and Diego: Absolutely on the mark! Bostrom and Yudkowsky certainly miss this crucial theme of a plurality of non-convergent AGI cultures. Without adding this key consideration all discussion of an AGI pause of 6 or 600 month is unrooted in reality. I find Bostrom’s Superintelligence almost quaintly out of date. Yudkowsky is almost unreadable polemic. Bostrom was written before Trump, Putin, and Xi made the clash of cultural assumptions and notions of one capitalized Truth so glaringly wrong. It was already obviously wrong to cultural anthropologists, but many of us in our…
Well the most pressing question is whether it will kill us all. There are good reasons to suspect that; Nick Bostrom's Superintelligence: Paths, Dangers, Strategies (2014) remains my favorite introduction to this thorny problem, especially the chapter called "Is the Default Outcome Doom?" Whether LLMs are sufficient for artificial superintelligence (ASI) is of course also an open question; I'm actually inclined to say no, but there probably isn't much left to get to yes. A lot of smart people, including myself, find the argument convincing, and have tried all manner of approaches to avoid…
Too late to edit, so I'll post just a few examples here: >The only way out of this mess is to design a moral fixed point, so that even through thousands and thousands of cycles of self-improvement the AI's value system remains stable, and its values are things like 'help people', 'don't kill anybody', 'listen to what people want'. Bostrom absolutely did not say that the only way to inhibit a cataclysmic future for humans post-SAI was to design a "moral fixed point". In fact, many chapters of the book are dedicated to exploring the possibilities of ingraining desirable values in an AI, and…