Anticipating Superintelligence with Nick Bostrom – TWiML Talk #181

Hello and welcome to another episode of Twimble Talk, the podcast why interview interesting people, doing interesting things in machine learning and artificial intelligence. I’m your host Sam Charrington. In today’s episode we’re joined by Nick Bostrom, professor in the faculty of philosophy at the University of Oxford, where he also heads the future of Humanity Institute. A multidisciplinary institute focused on answering big picture questions for humanity with regards to AI safety and ethics. Nick is of course also the author of the book Super Intelligence, Paths Danger Strategies. In our conversation we discussed the risks associated with artificial general intelligence and the more advanced AI systems Nick refers to as super intelligence. We also discussed Nick’s writings on the topic of openness in AI development. In particular the advantages and costs of opening closed development on the part of nations and AI research organizations. Finally, we take a look at what good safety precautions might look like and how we can create an effective ethics framework for super intelligent systems. Before we dive in, a final announcement for Wednesday’s America’s online meetup. At 5pm Pacific time, David Clement will present the paper Deep Mimic, example guided deep reinforcement learning of physics based character skills by researchers from UC Berkeley, including former guests Peter Rebeel and Sergei Levine. For more info or to register, visit twimmelai.com slash meetup. Okay, enjoy this show. Alright everyone, I am on the line with Nick Bostrom. Nick is professor in the faculty of philosophy at the University of Oxford as well as being director of the Future of Humanity Institute and director of the Governance of Artificial Intelligence Program. Nick, welcome to this week in machine learning and AI. Thanks for having me. Nick from what I read, you had four undergraduate majors, eventually settling on physics and neuroscience for your graduate work and now you’re a philosopher. Can you start us off by telling us a little bit about your journey and your background and what you’re focused on nowadays? Well, I started a bunch of different things as an undergraduate that seemed to me to possibly be useful for eventually trying to understand big important things. I didn’t know exactly which things were important to understand but philosophy seemed to be one area, physics, another neuroscience and AI also useful to have. So I kind of just started around a lot of different things to make a long story short. The thing that I’m now running the Future of Humanity Institute is a multidisciplinary research center at Oxford University. We have people from a number of different disciplines, mainly mathematics and computer science but also some philosophers, political scientists, trying to think hard about big picture questions for human civilization, things that could affect the trajectory of Earth originating intelligent life and AI has been a big focus, I’d almost say, obsession of ours for a number of years now. So we have both one group that is doing technical work on AI safety, AI alignment and another group that is looking at AI governance issues. And did you study AI as part of your graduate work? I did a little bit. Yeah, this was back in the 90s. I took some AI courses back then. I then wrote a master’s thesis in computational neuroscience. And then I kind of drifted away from the field a little bit but came back to it as a focus area when I began working on my book, The Book Superintelligence. So this would have been maybe 10 years ago. However, those are not familiar with the book. What were the major themes in superintelligence? The book tries to think through what happens if AI succeeds one day at its original goal which has all along been to produce general intelligence in machine substrate, not just to substitute for human cognition in specific tasks but to figure out ways, I think, of achieving the same powerful general learning ability, planning ability, reasoning ability that make us humans unique. So at least when I was beginning working on that book, there had been a surprisingly small amount of attention paid to what would happen if the ultimate goal were achieved. There’s a lot of work trying to make AI slightly more capable but a little thought to what would happen if we achieved human level general artificial intelligence one day. I argue in the book that that probably would be followed within relatively short order by superhuman levels of machine intelligence. And I then, to bulk of the book, then it explores different scenarios and tries to introduce different concepts and analytic tools that one can use to try to begin to think more systematically about the issues that will arise in that kind of radically transformative context. Yeah, I’m wondering, do you consider yourself or do you think of yourself as either pessimistic or optimistic about this, the direction that AI is taking? I’m full of both hopes and fears. I think sometimes my public persona is often believed to be more on the negative side than I actually am. That is I’m actually very excited about the myriad beneficial applications both in the near term and also over a longer period of time that one could hope to get from AI. But I think because the book had a significant focus on what could go wrong if we failed to get this transition to machine superintelligence right. Because the book spent quite a number of pages trying to get the more granular understanding of exactly where the pitfalls are there so that we could avoid them. I think then say journalists often come to me to get the negative scoop. And once you’re sort of in that pigeonhole, the journalist might ask you a bunch of questions. You say various things, some positive, some negative, they cut out all the positive things you say. And then you appear at saying this negative sound bite and then other journalists hear that and it kind of gets self-reinforcing loop. But to answer your question, no, I wouldn’t consider myself as either an optimist or pessimist but I’m trying to just better understand what this spectrum of possibilities are and most importantly how actions people take or can take here and now will affect likely long-term outcomes for humanity. When you think about the kind of spectrum of outcomes and the things that could go wrong, then how do you categorize them? Is there kind of a high-level categorization that you’ve developed of the different ways that you think about the safety issues that we’re trying to safeguard? Well, you can carve it up in different ways. If you want a very high-level division, you could distinguish risks that come from the AI itself. That is ways that AI could harm humans. So there we have in the context of machine super-intelligence, the possibility of a failure to solve the alignment problem, failure to get this hypothetical future super-intelligence system to do what we wanted to do. We can go into that in more detail later if you want. But that’s kind of one category of risk. Then another would be the risk that not that the AI would do something bad to humans, but that humans would use the AI to do bad things to other humans. So use it unwisely, recklessly or maliciously. The way we’ve used a lot of other technologies throughout human history, not just for beneficial purposes, but to which war or to oppress one another. Then I actually think there is a third category of concern, which has received a lot less attention, but this would be risks of us humans doing bad things to the AI. I think at some point these digital minds that we’re building might come to process various degrees of moral status, just like many non-human animals have degrees of moral status. And there is then a risk that we will maltreat, say an AI system that is capable of some form of sentience, more of the relevant interests, capable of suffering. At least if we draw a logical scheme, that should be in there as one category of potential harm that could occur. It strikes me that in order to fully wrap our heads around this issue, it’s fundamentally a long-term, broad time horizon issue. There’s a lot of the response that you see to these issues being raised is that we’re nowhere near AGI, we have no idea how we’re going to get there. When you think about the timeframe or time horizon of your research, do you put a number on that? Do you have a sense as to what’s the timeframe that we need to be worried about this kind of thing? There is a lot of uncertainty about the timeline. In organizing one’s thinking, it’s sometimes better not to do it with reference to say calendar years, but rather relative to some set of capabilities. Different way we reach this level of technical capability than these social issues will arise. I think a lot of the confusion and some of the controversies surrounding these questions come from conflating to different contexts. The context of long-term, radically transformative machine super-intelligence or human level intelligence on the one hand and on the other hand, the near-term context of what we can do now or we’ll be able to do over the next few years that will impact, say, national security or impact the economy. I think both of these contexts are important, are legitimate things to have conversations about, but they are very different and why needs to keep them apart. Otherwise, I think one simultaneously tends to over-hype the present and under-hype the longer-term future. And so is your research particularly focused on the longer term? Yeah, that’s what I’m most interested in because I think ultimately that will matter more and make a bigger difference in the world. Do you work on both these near-term issues as well as the longer-term issues? Well, a little bit on the near-term, but really our heart is in the longer term, trying to figure out whether there are things at some point that might affect the trajectory of human civilization as opposed to just be bumps in the road. I think it’s also more neglected. There are more groups interested in near-term issues, so our relative ability to add value there is smaller than I think on the more neglected longer-term issues. I’m curious what your experience is talking to folks about these longer-term issues. Given the challenges that we have with things like global warming and humans impact on, the habitability of the Earth, which seems to be more present than AGI, how do you get people to care, I guess, is the question? There’s a bit of a huge amount of interest actually in our work in general, but on AI in particular. So maybe it’s a pricing, but yeah, that hasn’t been a problem. I think the key challenge now is not so much to create a greater level of interest on AI and long-term AI, but rather to try to channel the existing level of interest and concern in constructive directions. So a lot of people have this general sense, well, AI, maybe it could be really big, it could be good or bad or scary, we don’t even know how to think about it. So how do you take that and then use that amount of activation energy to produce actually constructive work in the world? Like say, research that will give us better tools for scalable control or advances in our ability to politically organize or to think of governance arrangements that could result in a better outcome down the road. I want to take a step back. You mentioned that along this thinking about the timeline that it’s often better to think about it in terms of capability. What are their specific pivot points or inflection points in the capability timeline that are notable? Achieving AGI, I imagine is one and achieving superintelligence is another. Are there inflection points between where we are today and AGI that you think are interesting milestones? It’s actually quite hard to think of what would be a reliable indicator that AGI is going to happen, say, five years down the line. And I think it’s quite possible that it’s that what will happen is it will look like we’re lost in a thick forest for some unknown period of time and then maybe we stumble on a clearing and the finish line is just a few words ahead of us. That is, we shouldn’t have a great deal of confidence in our ability to be able to see a long time in advance that AGI will occur. There are some milestones that maybe if you had asked people 20 years ago, they would have thought would be pretty impressive, say, the successes with AlphaGo. Now that we passed them, I mean, there is a risk, I guess, of just gradually taking for granted things that really X and D were hugely impressive and at our expectation level just adjust so that there would be no point maybe at which we will be more shocked and odd than we were with AlphaGo until we’re all most all the way there and you already have weak forms of AGI running around and doing and at that point it’s kind of a little bit late and today it’s to start thinking about these safety issues. I think we want to use the time we have available now to put ourselves in the best possible position for the coming transition to the era of machine superintelligence. And so you mentioned a big part of your work is trying to come up with these concrete strategies or agendas that folks should be taking up. What are some of those things that we should be doing to prepare? Well, one is this research field of AISafety, which when the book was being written hardly existed and might have been 10 people in the world who were doing that and it was very, very far from mainstream. So now there is a research community working on this with groups in a number of different places. There’s a group at Berkeley, some working Montreal. We are doing a joint technical research seminars with DeepMind who also has an AISafety group, OpenAI. So it’s kind of become a small little research field in its own right and that seems constructive and probably there should be more of that kind of work. And I think we’re still at an earlier stage with respect to the governance challenges. Maybe we are with respect to the governance challenges where we were with respect to AI safety work five years ago. Like there is some sense that it’s important that somebody should work on it but not yet a very clear conception about just what kind of work would be helpful. So maybe a few years down the road, we will have a clearer sense of what kind of work on governance would be productive. So the AISafety is a technical challenge, ultimately the more people hammering away at it, the better, at worst they produce nothing and at best they produce some insight that could actually be useful. When it comes to political problems it’s not always obvious that having more people work on it will produce a better outcome or producing more insight or knowledge or shared understanding will always produce a better outcome. We also have to worry there about things like arms race dynamics and so forth. So it gets more strategically complicated to figure out what would actually be a helpful intervention when we are talking about things that are more in the political domain. Maybe as an example of that you wrote a paper, I guess it was last year, some time on the implications of openness in AI development. With the cursory view of the paper it was, your results were a little bit counterintuitive. Can you talk a little bit about that paper and what led to it? There is this widespread view that openness in AI development is a good thing. Openness is almost one of these words that freedom or democracy or fairness, that is almost like just in a plus light. It sounds good, we should have more of it. I think it’s not that obvious that ultimately that is what we want to have more of with AI. I think the short term impact of more openness or positive that is more people more quickly get access to state of the art techniques and can use them more widely and I think on balance that is positive. But if we are thinking about this hypothetical future strategic context where we are getting close to developing machine super intelligence and you think maybe there will be several groups or countries or firms competing to try to get there first. In that context openness could be extremely dangerous. You would want, it seems to me, whoever develops super intelligence first to have the ability at the end of that development process to pass for six months, let us say, or a year to test their system very carefully, double check all their safety mechanisms, maybe to slowly boost its intelligence through the human range and into the super intelligence range. But if you have, say, 20 different research groups running next to neck with almost indistinguishable technology, then if any one of them decides to take it slow and be careful, they will just be surpassed by one of their competitors. So it seems in an extreme race condition, the race would go to whoever takes the fewest precautions. The least cautious, that seems to be a risk-increasing situation. So you’d be looking for ways to maybe increase the lead of whichever AI developer isn’t the lead at the time when you’re getting close to super intelligence. So that’s the backdrop. Now think about what openness does. It kind of equalizes one variable that could cause dispersion in AI capabilities. So if you are open about the general science, well, then everyone has access to the same general scientific ideas. If you’re open about your source code, let’s say, then you equalize the software base that different developers have. That would mean that any remaining dispersion would have to come from, say, difference in hardware or different data sets or something like that. But one less source of dispersion and capability. So that would tend to equalize the race, make it more tightly competitive and therefore tend to reduce the lead time that the lead developer has in the end to go slow for the sake of caution. You know, you are in this paper proposing that there are some costs to open this in that they accelerate AI development and more specifically eliminate the opportunity to put in checks and balances kind of in the end game. I’m also wondering if there’s a cost to lack of transparency or closeness that you factored into the analysis that strikes me as mostly around the danger of not knowing where AGI is. If AGI is much closer but it’s closed and you don’t see it and it’s potentially in a more advanced state in your competitors, how do you factor that into the analysis here? So the full analysis, there are a number of other important considerations as well. Besides the one I mentioned concerning the racing dynamic, you might think about whether say an open development context tends to say attract a different kind of participant, maybe with better motives than products done in secret. But in terms of being able to know the capabilities even of different products, let’s set aside their actual algorithms or ideas but even just knowing how far along different product is. In at least one simple model we have, this is in an earlier paper with a couple of colleagues of mine called racing to the precipice. You actually get the higher level of overall risk taking if competitors can see more precisely how far along each other is. The intuition being roughly that in this very simple gap theoretic model, if you have a winner takes all dynamic, that if you see that you are behind, you sort of know for sure that you will lose and you’d be willing to take any extra amount of risk if that helps you have at least some chance of catching up. Whereas if you are unsure about your relative position then that would be a limit to the amount of risk you would take because you might be ahead and you wouldn’t want to then take more risk just to get farther ahead if that then means you’d be likely to destroy the world if she succeeded. Now one can construct different kinds of models of this type of situation and get different outcome. But I think there are some general lessons that seem relatively robust. One is that the greater degree to which there is a commonality of purpose between different competitors. That is the greater the degree to which it wouldn’t matter trying to body who got their first, the more investment in safety you’re likely to get. In the limit where it is completely indifferent who gets their first and you don’t have a race. You’d be quite happy to drop out of the race if that allows another competitor to spend more time and be more cautious in the relevant stages. So that fostering, co-operation, fostering a kind of commitment to the common good would seem to not just be good from a fairness point of view, making sure everybody gets the slice of the upside but also be good from an AI risk point of view in taking some of the pressure off this possibility of a racing dynamic. And it’s not an all or nothing thing. But the more you can kind of ingrain early on incredible commitment to use AI for the common good of all of humanity rather than for narrow, factional purposes to just enrich one company or strengthen the military of one nation. I think the more one can reduce this competition dynamic and obviate some of the problems associated with that. So does that present a paradigm of sorts in this particular research area that a more collaborative environment reduces this competitive race but also tends towards openness which increases the competitive nature of it or the risks associated with it? Yeah, that could be some trade off there. I think you might want to make a distinction between collaboration and cooperation. So if collaboration means actively working together on one and the same product side by side that would be maybe one way of cooperating but you could also modern at least theoretically the possibility of having entirely separate products that have no communication but are both committed to helping each other out or to sharing the spoils. So that might be a highly operative development regime but one that lacks actual active collaboration. And what it seems that what ultimately primarily want here is cooperation and whether that involves collaboration as well is a more tactical question of what seems feasible at the given time. Okay. Returning to the work in an AI safety which is a bit further along. What would you say are some of the interesting directions there and any early noteworthy results that folks have had? Well, I think maybe it’s obvious but it took a little while for it to become kind of common knowledge that the approach, best illustrated by loss of robotics. The idea that you would handcraft a few principles to guide what AI is allowed to do does not seem to scale and therefore seems entirely unpromising as an approach to solving super intelligence alignment and that what instead you’ll need to do somehow is to leverage the intellectual capability of these hypothetical future systems to help us solve the alignment problem maybe by having AI that learn human preferences from human behavior or brain directing with us or that otherwise leverage their intelligence to help us figure out what it is that we are trying to get them to do. So now the question then becomes how can you actually do that? And then there is a number of different ideas for how to go about researching that with different researchers having different judgments and intrusions about which of these is more promising. Some of these research avenues are more continuous with current research that is of interest quite independently of any application in a future context of super intelligence. So you have, say, inverse reinforcement learning and human preference modeling that is a few even if what you’re trying to do is to get like say a recommender system to work better. You want to see various customers have bought these different books and films and rated them thus what other books and films are they most likely to use. That is a simple example of how you try to build an AI system that can infer what humans want. But if you’re trying to move that technology in a direction that could also work in this context of super intelligence then there are some distinct challenges that arise that you could try to do work on. But there is a bunch of other ideas as well of research that seems useful to give us a better understanding of the possible safety challenges that could arise when you have kind of super human systems that you need to control. So just to give you one flavor of that. So one thing that a human level system could do is to engage in strategic behavior. It could like humans do it can anticipate what other humans do. It could be deceptive. A super intelligent system could do that presumably to a super human degree of competence. So once you have a sufficiently capable system you might no longer be able to just assume that you could easily test its capabilities. It might kind of pretend to be less competent than it really is. If it predicts that that will then result in a certain behavior on the part of the programmer. So it’s human keepers. It might conceal its true goals if it perceives that there is a strategic rationale for doing so. So that kind of qualitatively different behavior that you wouldn’t necessarily expect to see in some human artificial intelligence systems. Also when you have a super intelligent system there might be a little part of the system. Some internal optimization process that might itself be highly capable and maybe smarter than human. And you need to think about whether there could be some agent processes emerging from a larger system that you hadn’t built in there by design. So there are some ways in which the control problem looks qualitatively different when one is thinking about the challenge of controlling a super intelligence as opposed to the challenge that we currently have of getting more limited AI systems to perform transportation. How as a AI safety researcher how would you attack those types of problems in the absence of the actual super intelligence? What are some of the approaches folks are taking to kind of define and make progress in these areas? Well, so again, it comes down part of to taste the subjective judgment. So you could on the one hand try to do more theoretical work, try to think through more from first principles what are some of the issues that could arise with very powerful systems. Or to a different personality you might prefer to try to do things that kind of build on existing techniques and develop them in a direction that could seem to be useful. So a lot of these examples of the latter would be do all you say better techniques for understanding what is going on inside a deep neural network interpretability tools. It seems like that would be useful not just for making faster progress today. If you are a researcher like you want to see exactly what’s going on in your network and why it’s doing what it’s doing, but it also seems like that could lead into over time things that would be useful for AI safety in the longer term. If you could kind of have better tools for monitoring the cognitive processes occurring inside the system. So there is like on one hand, you had these things like transparency, trying to better understand adversarial examples and countermeasures to that, better ways of doing reinforcement learning or imitation learning. And on the other hand, these more kind of conceptual or purely mathematical studies of hypothetical systems that we can’t currently build, but that you can nevertheless reason about a little bit like you would about some mathematical object that you can’t actively compute, but you could have some theoretical understanding about. I guess I’m wondering how, you know, when we reason about these systems mathematically in that way, is there a clear mapping from those more theoretical from the work we’re doing today, theoretically, to, you know, what might need to be done far off in the future in the case of these systems coming into existence. Or is it, you know, more an issue of hay work exploring these different areas, theoretically, and it’s providing a foundation for future iterations of work in the same area that build on each other with the hopes that we keep pace with the advances that are happening on the AI side itself? It’s more the latter. So AI safety, I think, should still be regarded as a pre-powered,igmatic science that it’s not clear what the best or most relevant way to address these problem is. It’s not even generally agreed exactly what the problem is. So you have different smart people trying out different things coming up with problem formulations or concepts or sub-products that should be explored. Some of them might turn out to be useful. It’s possible none of them would be really useful, but what would be useful is to have built up a research community that is kind of continuously engaging with these questions and gradually over the years refining their insights and having them kind of then being able to apply that skill to the systems that are eventually built. I think it’s kind of the sense that this is important enough that we should try our best to make progress on it. And here are some cool ideas for how to make progress. We don’t yet know whether or not those will actually turn out to be useful. Are there some things that folks that are working in the broader AI field should be thinking about, meaning that folks that aren’t working specifically in AI safety or AI governance but are developing machine learning in AI systems today? How do you advise those folks to contribute to this broader issue of AI safety without being fully dedicated to this kind of research? Well, I mean, it’s like how you contribute to any cost in the world that you’re interested in. You could try to work directly there or you could try to support other people working there. You could recommend to talented friends or young who wants to go into this to actually do so, donate money or give prestige by legitimizing it and so forth. Sure, I guess I was wondering if there are things that you wish every person working in AI was thinking about or something like that. It does seem to me that what would be quite robustly valuable across a wide range of different scenarios is to have a more cooperative approach to AI, to try to as far as possible and bring into the community a commitment to the common goods principle that AI is super intelligence. If it actually were one day achieved, it should be for the benefit of all of humanity. And in the service of why we shared ethical ideals. And I think the more that that kind of becomes embedded within the machine learning community, the greater the chance that we’ll actually ultimately happen as well. I think the research community has a non-trivial amount of power. Certainly today, if you want to do cutting edge AI research, say, your firm and you want to be able to hire from the first tier of talent, it helps a lot if you can credibly claim that you’re going to use this for ethical acceptable purposes. If you want to do something nasty, like figure out a better way to generate spam or something like that, or do some kind of highly unpoliteable military application, chances are you’re going to have a much harder time to recruit the best. You might have to go to the second or third tier talent. So this kind of big commitment to idealism and cosmopolitan values. I think could exert some shaping influence over how AI is developed and used. And so I think anybody in the community can do their part to strengthen that. And I think that cumulative could be quite valuable. There is a sort of more esoteric issue as well, which relates to this idea of digital minds again, that as our AI systems become more and more complex and capable, and at some point maybe have the same behavioral repertoire of capabilities as animals like a mouse, say, or a dog. At that point, I think the question of the moral status of these digital minds becomes increasingly relevant. And it’s a hard thing to discuss. It still feels a little bit like one of those silly things that is kind of, you know, you can’t say with a strange face. But that’s where AI safety was five years ago. It was also this fringe thing that a few people on the internet talked about. And you couldn’t really, and to some extent, these current conversations about AI governance and the wider impact of society. That’s also been moving from a fringe science fiction saying people on the internet to something that people who think of themselves as serious people now acknowledge that’s a legitimate thing to work on. And I think that the moral status of digital minds needs to start making this migration as well from suitable topic for the philosophy seminar room into the kind of thing that you can talk about in some mainstream forum with different views on it, but without it being a silly thing or something that you kind of snicker about. And again, yeah, as you can contribute to that by just not being afraid of talking about that if the topic comes up with your friends or colleagues. One of the interesting things that I came across in some of your writing was this notion of how ethics itself is this dynamic force over time, this dynamic picture over time. And we need to, I forget the specific construct, but it was part of the way we approach AI safety is to build in some notion of ethics. It’s almost like we need to go back in time to being in, you know, ancient Greece or something like that and trying to build a system that could map itself from that, you know, ethical system to our current ethical system, which is, you know, pretty dramatically different. And that’s kind of the way we need to think about building a system today. Can you kind of elaborate on that? Did I give you enough to spark some recognition, which we’re actually saying? Yeah, so I think it would be a mistake if one thinks about how one would use super intelligence to try to list all our current object level ideas and conceptures and moral principles and try to hardwire those into the AI to then forever characterize how the future should pan out. We should recognize that just as every other earlier age that we can now look back on by our lives were severely misguided, had huge moral blind spots in terms of, I don’t know, you could go through the list, like slavery, status of women, how it did animal, social inequalities, causes for war. Presumably, we have now not yet attained full moral enlightenment. And it’s quite likely that if there are leader stages of human history that looks back on 2018, they might also shudder to think about the atrocities that we were committing right now. And so we want to leave open the possibility for moral growth. And not just moral growth narrowly conceived, but for in general, there to be development in how we think about human values and what life can involve and how we can organize society. And so that one perhaps attractive vision for how super intelligence would be used would be to enable a deeper deliberation on the part of humanity informed, say, by super intelligent advice. And maybe AI could help us increase human intelligence and safeguard us while we were doing this, but some kind of deliberative process that could ponder these things for a long time before we made any irrevocable decisions about exactly what kind of post human future we would want to create and move into. To kind of summarize, it’s tempting for us to say, hey, in order to ensure the ethical behavior of these systems, let’s bacon our ethics. But in the future, our ethics will undoubtedly be proven to be inferior. And so we need to. Yeah. And navigate, build almost an ethical calculus maybe and build systems that can navigate kind of changing ethical standards. Well, okay. Yeah. So I don’t think an ethical calculus in the sense that we would lay down some moral axioms and then the AI would compute things from those. But more that there is some process whereby we do moral thinking or in general do thinking about what we want in life. And if the AI could learn to do that same kind of thinking or to maybe learn to extrapolate our thinking that might be one way of getting indirectly at this thing that we really want as opposed to the thing we would say we want if we had to make up some answer of the cuff. So I think you’re right for two reasons. One is that you wouldn’t want to just call in our current misconceptions and superficial misunderstandings. You would want this possibility of learning and developing and doing something better that we could do at this per if the moment. Also, I think this harkens back to the earlier idea of a commitment to the common good that the more you can conceive of this as a very inclusive process, we’re not just one person or one country. So one country’s values would achieve total dominance, but something that could incorporate many different interests, many different value perspectives. It would be not an absolute degree, but a widely generous and inclusive degree. I think that would also make it possible to conceive of a future that is more widely appealing and that would reduce some of these incentives to race to the precipice that would arise if every participant in the development process were just have been done on imposing their own idiosyncratic views on the entire future. So I think that two reasons for doing it more indirectly and in this more inclusive way to the extent that it’s possible. And that, again, I think is something that anybody who’s a member of this community in small ways, not so much in the specific work that do day to day, but in terms of being ethical members of this community that can sometimes talk about and express views about how their little contributions should be used and the overall purpose for which this technology should be developed, where there’s space for a lot of voices to cumulative to make a big difference. For folks that have listened to our conversation and want to learn more or dig deeper or begin to even contribute or support some of the work happening in this area, what are good places to start in terms of resources or organizations to follow? Well, I have maybe an obvious bias. I point to the book that we talked about earlier, like maybe best expresses my perspective on this, but there is a kind of overlapping set of communities that are interested in this. So one, quite interesting one is the effective altruism community. It’s concerned about the wider range of costs areas, but AI also increasing being one of them. It’s a community of people who are trying to take care of how you can have the greatest possible positive impact on the world in terms of what you do with your career or in terms of where you donate your charitable giving. So checking out their resources, they have a career guide, they have various blogs, would we want a good place to start? If you want to do technical AI safety, then you can Google technical AI research agendas and you’ll find different ideas. Or you can see some of the latest publications coming out from the BIND AI Safety or Open AI or FHI or MIRI, the group at Berksey, to see the concrete examples. So I think people are working on it. Well, Nick, thank you so much for taking the time to chat with me. Is there anything else that you’d like to leave folks with? Now, I think the bottom line is, we don’t know how long it will take, but if we ever do get to create super intelligence, it’s such a big thing that our bottom line must be the sense of enormous humility that we are just in way over our heads, but we have no choice but to make our best effort to get through this somehow in a responsible, wise and generous way. You’re great. Thank you. Alright, everyone, that’s our show for today. For more information on Nick or any of the topics covered in this episode, visit twimmelai.com slash talk slash 181. If you’re a fan of this podcast, we’d like to encourage you to visit your Apple or Google podcast app and leave us a five star rating and review. Your reviews help inspire us to create more and better content and they help new listeners find the show. As always, thanks so much for listening and catch you next time.