If there is one factor that contributed most to society’s progress, it is intelligence. We have progressed from living in caves to living in electrified houses with clean water and a quality of life that would have seemed like a sheer utopia if our ancestors had been able to even imagine it. At the same time, intelligence contributes to the biggest risks to humanity – without intelligence, we would have no nuclear energy but also no nuclear bombs. 

Transformative AI could amplify both the societal benefits and risks of intelligence. But contrary to the emergence of human intelligence, we have an opportunity to shape AI’s development: What goals could society achieve with AI? What institutions already exist for this purpose? Can we learn from their successes and shortcomings to build institutions better designed to leverage transformative AI for good?  

To explore these questions, Foresight Institute and the Future of Life Institute gathered leading researchers in AI, policy, law, economics, and related fields for a two-day event in February 2024: the Existential Hope Transformative AI Institution Design Hackathon. The goal was to design institutions that can guide the development of Transformative AI for the benefit of humanity. Institutional proposals were judged based on how well they would work, how realistic they were, and the positive impact they could have. If you are curious to learn more about the hackathon procedures and outcomes you can learn all about it in our detailed hackathon report.

Here are the three winning institutions:

1. Can AI Make Us Happier? The Flourishing Foundation’s Proposal for a Human-Centered Future (Hackathon Winner)

The Flourishing Foundation is an independent innovation lab that tackles the question: can AI make us happier? Their mission is to ensure that powerful new technologies like AI benefit humanity and the planet.

This interdisciplinary group of scientists, designers, engineers, and artists believe technology should strengthen our connections – to ourselves, each other, and the natural world. They advocate for “life-giving” economic systems, arguing that deploying AI within current economic structures won’t necessarily improve well-being.

The Flourishing Foundation takes a systems-thinking and life-centric design approach. Here’s how:

1. Operationalize Research: Translate interdisciplinary research into knowledge frameworks that better guide conscious technology creation: e.g. alternative well-being based success metric for consumer tech products and services other than “engagement”
2. Incubate Products: Provide holistic and hands-on support for innovators to design and run experiments with a focus on conscious/humane use of transformative technologies: e.g. AI-enabled solutions for elderly care and family connection
3. Build Movement: Build awareness by mobilizing innovator communities to channel their creative energy towards conscious tech creation: e.g. weekly meetups, quarterly build days, and symposiums.

2. AI for Democracy? The Global Deliberation Coordinator Aims to Revolutionize Global Decision-Making (Shared Second Place)

The Global Deliberation Coordinator (GDC) is a new approach to global decision-making through “Global Deliberation as a Service” (GDaaS). GDC is a coordinating body that works with partners around the world to convene a representative microcosm of the planet — and equip them with the structure and resources needed for high-quality deliberation.

These global deliberations can be utilized by international organizations, governments, and companies to prove input or make critical decisions that put “humanity in the loop”. Through an advanced market commitment, pilot projects, and integration of cutting-edge AI and deliberative technology, the GDC seeks to demonstrate the feasibility and impact of this new model.

By making global deliberative processes more accessible and impactful, the GDC aims to strengthen humanity’s collective decision-making capabilities in the face of planetary challenges like artificial intelligence development and climate change. GDaaS offers a powerful new tool for incorporating the considered will of the people into how we navigate the crucial choices ahead.

3. Preparing for the Unexpected: Transformative Simulations Research Institute (Shared Second Place)

The Transformative Simulations Research Institute (TSR) is a new organization dedicated to rigorously modeling how individuals, groups, and societies may respond to the emergence of transformative artificial intelligence (TAI) capabilities. As TAI systems grow more powerful, there are risks of misaligned or adversarial development that could destabilize or threaten humanity.

To help mitigate these risks, TSR employs cutting-edge simulation techniques like wargaming exercises, computational games, and human-led scenario roleplays to systematically investigate potential TAI trajectories from multiple perspectives. By developing an empirically-grounded, multidisciplinary understanding of the cognitive patterns, social dynamics, and ethical issues that may arise when advanced AI intersects with human actors, TSR aims to equip policymakers and technologists with crucial foresight.

TSR’s goal is to steer transformative AI development toward robustly beneficial outcomes that safeguard human flourishing over the long term. TSR’s simulations chart the vast possibility space of TAI-enabled events and human decision pathways, identifying potential pitfalls but also constructive governance frameworks. The institute pioneers novel experiential modeling approaches to reality-test our assumptions and future-proof society against catastrophic AI failure modes as this powerful technology advances.

A hackathon can excite institutional prototypes but the real work lies in realizing them. To incentivize and support the continuation of the work initiated during the hackathon, the winning team was awarded $10,000, and the two teams that were selected as runners-up, each received $5,000. To put these ideas into action, each proposal – The Flourishing Foundation, the Global Deliberation Coordinator, and the Transformative Simulations Research Institute – are currently being incubated into real-world institutions. We look forward to following their evolution. 

To provide efforts like this with the support they need, The Future of Life Institute launched its Futures program: This program aims to steer humanity toward the beneficial uses of transformative technologies, including offering funding opportunities for research on safe AI applications to improve the world. 

To offer a recurring forum for envisioning beneficial AI worlds, The Foresight Institute launched its Existential Hope Worldbuilding Course: This course focuses on exploring AI in various future scenarios, promoting optimistic visions of AI solving global challenges.

We extend our gratitude to all who contributed – from our hackathon teams, to the judges, mentors, and the Future of Life Institute. Stay tuned for further updates on the implementation of these ideas!

Resources

• TAI Institution Design Hackathon Report
• FLI Futures Program
• Foresight Institute’s Existential Hope Program
• Foresight Institute’s Existential Hope Newsletter

Although FLI did not develop this statement, we strongly support it, and believe the progress in regulating nuclear technology and synthetic biology is instructive for mitigating AI risk. FLI therefore recommends immediate action to implement the following recommendations.

Recommendations:

• Akin to the Nuclear Non-Proliferation Treaty (NPT) and the Biological Weapons Convention (BWC), develop and institute international agreements to limit particularly high-risk AI proliferation and mitigate the risks of advanced AI, including track 1 diplomatic engagements between nations leading AI development, and significant contributions from non-proliferating nations that unduly bear risks of technology being developed elsewhere.
• Develop intergovernmental organizations, akin to the International Atomic Energy Agency (IAEA), to promote peaceful uses of AI while mitigating risk and ensuring guardrails are enforced.
• At the national level, establish rigorous auditing and licensing regimes, applicable to the most powerful AI systems, that place the burden of proving suitability for deployment on the developers of the system. Specifically:
  • Require pre-training auditing and documentation of a developer’s sociotechnical safety and security protocols prior to conducting large training runs, akin to the biocontainment precautions established for research and development that could pose a risk to biosafety.
  • Similar to the Food and Drug Administration’s (FDA) approval process for the introduction of new pharmaceuticals to the market, require the developer of an AI system above a specified capability threshold to obtain prior approval for the deployment of that system by providing evidence sufficient to demonstrate that the system does not present an undue risk to the wellbeing of individuals, communities, or society, and that the expected benefits of deployment outweigh risks and harmful side effects.
  • After approval and deployment, require continued monitoring of potential safety, security, and ethical risks to identify and correct emerging and unforeseen risks throughout the lifetime of the AI system, similar to pharmacovigilance requirements imposed by the FDA.
• Prohibit the open-source publication of the most powerful AI systems unless particularly rigorous safety and ethics requirements are met, akin to constraints on the publication of “dual-use research of concern” in biological sciences and nuclear domains.
• Pause the development of extremely powerful AI systems that significantly exceed the current state-of-the-art for large, general-purpose AI systems.

The success of these actions is neither impossible nor unprecedented: the last decades have seen successful projects at the national and international levels to avert major risks presented by nuclear technology and synthetic biology, all without stifling the innovative spirit and progress of academia and industry. International cooperation has led to, among other things, adoption of the NPT and establishment of the IAEA, which have mitigated the development and proliferation of dangerous nuclear weapons and encouraged more equitable distribution of peaceful nuclear technology.  Both of these achievements came during the height of the Cold War, when the United States, the USSR, and many others prudently recognized that geopolitical competition should not be prioritized over humanity’s continued existence.  

Only five years after the NPT went into effect, the BWC came into force, similarly establishing strong international norms against the development and use of biological weapons, encouraging peaceful innovation in bioengineering, and ensuring international cooperation in responding to dangers resulting from violation of those norms.  Domestically, the United States adopted federal regulations requiring extreme caution in the conduct of research and when storing or transporting materials that pose considerable risk to biosafety.  The Centers for Disease Control and Prevention (CDC) also published detailed guidance establishing biocontainment precautions commensurate to different levels of biosafety risk.  These precautions are monitored and enforced at a range of levels, including through internal institutional review processes and supplementary state and local laws.  Analogous regulations have been adopted by nations around the world.

Not since the dawn of the nuclear age has a new technology so profoundly elevated the risk of global catastrophe.  FLI’s own letter called on “all AI labs to immediately pause for at least six months the training of AI systems more powerful than GPT-4.”  It also stated that “If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”  

Now, two months later – despite discussions at the White House, Senate hearings, widespread calls for regulation, public opinion strongly in favor of a pause, and an explicit agreement by the leaders of most advanced AI efforts that AI can pose an existential risk – there has been no hint of a pause, or even a slowdown.  If anything, the breakneck pace of these efforts has accelerated and competition has intensified.

The governments of the world must recognize the gravity of this moment, and treat advanced AI with the care and caution it deserves. AI, if properly controlled, can usher in a very long age of abundance and human flourishing. It would be foolhardy to jeopardize this promising future by charging recklessly ahead without allowing the time necessary to keep AI safe and beneficial.

• New at IAFF: An Untrollable Mathematician
• New at AI Impacts: 2015 FLOPS Prices
• We presented “Incorrigibility in the CIRL Framework” at the AAAI/ACM Conference on AI, Ethics, and Society.
• From MIRI researcher Scott Garrabrant: Sources of Intuitions and Data on AGI

News and links

• In “Adversarial Spheres,” Gilmer et al. investigate the tradeoff between test error and vulnerability to adversarial perturbations in many-dimensional spaces.
• Recent posts on Less Wrong: Critch on “Taking AI Risk Seriously” and Ben Pace’s background model for assessing AI x-risk plans.
• “Solving the AI Race“: GoodAI is offering prizes for proposed responses to the problem that “key stakeholders, including  developers, may ignore or underestimate safety procedures, or agreements, in favor of faster utilization”.
• The Open Philanthropy Project is hiring research analysts in AI alignment, forecasting, and strategy, along with generalist researchers and operations staff.

This newsletter was originally posted on MIRI’s website.

• News  — What MIRI’s researchers have been working on lately, and more.
• Goals — We plan to grow our research team 2x in 2018–2019. If we raise $850k this month, we think we can do that without dipping below a 1.5-year runway.
• Actual goals — A bigger-picture outline of what we think is the likeliest sequence of events that could lead to good global outcomes.

Our funding drive will be running until December 31st.

Research updates

• New at IAFF: Reward Learning Summary; Reflective Oracles as a Solution to the Converse Lawvere Problem; Policy Selection Solves Most Problems
• We ran a workshop on Paul Christiano’s research agenda.
• We’ve hired the first members of our new engineering team, including math PhD Jesse Liptrap and former Quixey lead architect Nick Tarleton! If you’d like to join the team, apply here!
• I’m also happy to announce that Blake Borgeson has come on in an advisory role to help establish our engineering program. Blake is a Nature-published computational biologist who co-founded Recursion Pharmaceuticals, where he leads the biotech company’s machine learning work.

General updates

• “Security Mindset and Ordinary Paranoia”: a new dialogue from Eliezer Yudkowsky. See also part 2.
• The Open Philanthropy Project has awarded MIRI a three-year, $3.75 million grant!
• We received $48,132 in donations during Facebook’s Giving Tuesday event, of which $11,371 — from speedy donors who made the event’s 85-second cutoff! — will be matched by the Bill and Melinda Gates Foundation.
• MIRI Staff Writer Matthew Graves has published an article in Skeptic magazine: “Why We Should Be Concerned About Artificial Superintelligence.”
• Yudkowsky’s new book Inadequate Equilibria is out! See other recent discussion of modest epistemology and inadequacy analysis by Scott Aaronson, Robin Hanson, Abram Demski, Gregory Lewis, and Scott Alexander.

News and links

• New from DeepMind researchers: “Interpreting Deep Neural Networks Using Cognitive Psychology“
• New from OpenAI researchers: “Corrigibility“
• A collaboration between DeepMind and OpenAI: “Learning from Human Preferences“
• Recent progress in deep learning: “Self-Normalizing Neural Networks“
• From Ian Goodfellow and Nicolas Papernot: “The Challenge of Verification and Testing of Machine Learning“
• From 80,000 Hours: a guide to working in AI policy and strategy and a related interview with Miles Brundage of the Future of Humanity Institute.

In the second 3 months of 2017, FHI has continued its work as before exploring crucial considerations for the long-run flourishing of humanity in our four research focus areas:

• Macrostrategy – understanding which crucial considerations shape what is at stake for the future of humanity.
• AI safety – researching computer science techniques for building safer artificially intelligent systems.
• AI strategy – understanding how geopolitics, governance structures, and strategic trends will affect the development of advanced artificial intelligence.
• Biorisk – working with institutions around the world to reduce risk from especially dangerous pathogens.

We have been adapting FHI to our growing size. We’ve secured 50% more office space, which will be shared with the proposed Institute for Effective Altruism. We are developing plans to restructure to make our research management more modular and to streamline our operations team.

We have gained two staff in the last quarter. Tanya Singh is joining us as a temporary administrator, coming from a background in tech start-ups. Laura Pomarius has joined us as a Web Officer with a background in design and project management. Two of our staff will be leaving in this quarter. Kathryn Mecrow is continuing her excellent work at the Centre for Effective Altruism where she will be their Office Manager. Sebastian Farquhar will be leaving to do a DPhil at Oxford but expects to continue close collaboration. We thank them for their contributions and wish them both the best!

Key outputs you can read

• When will AI exceed human performance? Evidence from AI experts

A number of co-authors including FHI researchers Katja Grace and Owain Evans surveyed hundreds of researchers to understand their expectations about AI performance trajectories. They found significant uncertainty, but the aggregate subjective probability estimate suggested a 50% chance of high-level AI within 45 years. Of course, the estimates are subjective and expert surveys like this are not necessarily accurate forecasts, though they do reflect the current state of opinion. The survey was widely covered in the press.

• Guide to working in AI policy and strategy

An earlier overview of funding in the AI safety field by Sebastian Farquhar highlighted slow growth in AI strategy work. Miles Brundage’s latest piece, released via 80,000 Hours, aims to expand the pipeline of workers for AI strategy by suggesting practical paths for people interested in the area.

• That is not dead which can eternal lie: the aestivation hypothesis for resolving Fermi’s paradox

Anders Sandberg, Stuart Armstrong, and their co-author Milan Cirkovic published a paper outlining a potential strategy for advanced civilizations to postpone computation until the universe is much colder, and thereby producing up to a 10³⁰ multiplier of achievable computation. This might explain the Fermi paradox, although a future paper from FHI suggests there may be no paradox to explain.

Individual research updates

Macrostrategy and AI Strategy

Nick Bostrom has continued work on AI strategy and the foundations of macrostrategy and is investing in advising some key actors in AI policy. He gave a speech at the G30 in London and presented to CEOs of leading Chinese technology firms in addition to a number of other lectures.

Miles Brundage wrote a career guide for AI policy and strategy, published by 80,000 Hours. He ran a scenario planning workshop on uncertainty in AI futures. He began a paper on verifiable and enforceable agreements in AI safety while a review paper on deep reinforcement learning he co-authored was accepted. He spoke at Newspeak House and participated in a RAND workshop on AI and nuclear security.

Owen Cotton-Barratt organised and led a workshop to explore potential quick-to-implement responses to a hypothetical scenario where AI capabilities grow much faster than the median expected case.

Sebastian Farquhar continued work with the Finnish government on pandemic preparedness, existential risk awareness, and geoengineering. They are currently drafting a white paper in three working groups on those subjects. He is contributing to a technical report on AI and security.

Carrick Flynn began working on structuredly transparent crime detection using AI and encryption and attended EAG Boston.

Clare Lyle has joined as a research intern and has been working with Miles Brundage on AI strategy issues including a workshop report on AI and security.

Toby Ord has continued work on a book on existential risk, worked to recruit two research assistants, ran a forecasting exercise on AI timelines and continues his collaboration with DeepMind on AI safety.

Anders Sandberg is beginning preparation for a book on ‘grand futures’.  A paper by him and co-authors on the aestivation hypothesis was published in the Journal of the British Interplanetary Society. He contributed a report on the statistical distribution of great power war to a Yale workshop, spoke at a workshop on AI at the Johns Hopkins Applied Physics Lab, and at the AI For Good summit in Geneva, among many other workshop and conference contributions. Among many media appearances, he can be found in episodes 2-6 of National Geographic’s series Year Million.

AI Safety

Stuart Armstrong has made progress on a paper on oracle designs and low impact AI, a paper on value learning in collaboration with Jan Leike, and several other collaborations including those with DeepMind researchers. A paper on the aestivation hypothesis co-authored with Anders Sandberg was published.

Eric Drexler has been engaged in a technical collaboration addressing the adversarial example problem in machine learning and has been making progress toward a publication that reframes the AI safety landscape in terms of AI services, structured systems, and path-dependencies in AI research and development.

Owain Evans and his co-authors released their survey of AI researchers on their expectations of future trends in AI. It was covered in the New Scientist, MIT Technology Review, and leading newspapers and is under review for publication. Owain’s team completed a paper on using human intervention to help RL systems avoid catastrophe. Owain and his colleagues further promoted their online textbook on modelling agents.

Jan Leike and his co-authors released a paper on universal reinforcement learning, which makes fewer assumptions about its environment than most reinforcement learners. Jan is a research associate at FHI while working at DeepMind.

Girish Sastry, William Saunders, and Neal Jean have joined as interns and have been helping Owain Evans with research and engineering on the prevention of catastrophes during training of reinforcement learning agents.

Biosecurity

Piers Millett has been collaborating with Andrew Snyder-Beattie on a paper on the cost-effectiveness of interventions in biorisk, and the links between catastrophic biorisks and traditional biosecurity. Piers worked with biorisk organisations including the US National Academies of Science, the global technical synthetic biology meeting (SB7), and training for those overseeing Ebola samples among others.

Funding

FHI is currently in a healthy financial position, although we continue to accept donations. We expect to spend approximately £1.3m over the course of 2017. Including three new hires but no further growth, our current funds plus pledged income should last us until early 2020. Additional funding would likely be used to add to our research capacity in machine learning, technical AI safety and AI strategy. If you are interested in discussing ways to further support FHI, please contact Niel Bowerman.

Recruitment

Over the coming months we expect to recruit for a number of positions. At the moment, we are interested in applications for internships from talented individuals with a machine learning background to work in AI safety. We especially encourage applications from demographic groups currently under-represented at FHI.

News and links

• From MIT Technology Review: “Why Google’s CEO Is Excited About Automating Artificial Intelligence.”
• A new alignment paper from researchers at Australian National University and DeepMind: “Reinforcement Learning with a Corrupted Reward Channel.”
• New from OpenAI: Baselines, a tool for reproducing reinforcement learning algorithms.
• The Future of Humanity Institute and Centre for the Future of Intelligence join the Partnership on AI alongside twenty other groups.
• New AI safety job postings include research roles at the Future of Humanity Institute and the Center for Human-Compatible AI, as well as a UCLA PULSE fellowship for studying AI’s potential large-scale consequences and appropriate preparations and responses.

News and links

• “The Off-Switch Game“: Dylan Hadfield-Manell, Anca Dragan, Pieter Abbeel, and Stuart Russell show that an AI agent’s corrigibility is closely tied to the uncertainty it has about its utility function.
• Russell and Allan Dafoe critique an inaccurate summary by Oren Etzioni of a new survey of AI experts on superintelligence.
• Sam Harris interviews Russell on the basics of AI risk (video). See also Russell’s new Q&A on the future of AI.
• Future of Life Institute co-founder Viktoriya Krakovna and FHI researcher Jan Leike join Google DeepMind’s safety team.
• GoodAI sponsors a challenge to “accelerate the search for general artificial intelligence”.
• OpenAI releases Universe, “a software platform for measuring and training an AI’s general intelligence across the world’s supply of games”. Meanwhile, DeepMind has open-sourced their own platform for general AI research, DeepMind Lab.
• Staff at GiveWell and the Centre for Effective Altruism, along with others in the effective altruism community, explain where they’re donating this year.
• FHI is seeking AI safety interns, researchers, and admins: jobs page.

This newsletter was originally posted here.

Very simple robots with very constrained tasks do not need goals or values at all. Although the Roomba’s designers know you want a clean floor, Roomba doesn’t: it simply executes a procedure that the Roomba’s designers predict will work—most of the time. If your kitten leaves a messy pile on the carpet, Roomba will dutifully smear it all over the living room. If we keep programming smarter and smarter robots, then by the late 2020s, you may be able to ask your wonderful domestic robot to cook a tasty, high-protein dinner. But if you forgot to buy any meat, you may come home to a hot meal but find the aforementioned cat has mysteriously vanished. The robot, designed for chores, doesn’t understand that the sentimental value of the cat exceeds its nutritional value.

AI and King Midas

Stuart Russell, a renowned AI researcher, compares the challenge of defining a robot’s objective to the King Midas myth. “The robot,” says Russell, “has some objective and pursues it brilliantly to the destruction of mankind. And it’s because it’s the wrong objective. It’s the old King Midas problem.”

This is one of the big problems in AI safety that Russell is trying to solve. “We’ve got to get the right objective,” he explains, “and since we don’t seem to know how to program it, the right answer seems to be that the robot should learn – from interacting with and watching humans – what it is humans care about.”

Russell works from the assumption that the robot will solve whatever formal problem we define. Rather than assuming that the robot should optimize a given objective, Russell defines the problem as a two-player game (“game” as used by economists, meaning a decision problem with multiple agents) called cooperative inverse reinforcement learning (CIRL).

A CIRL game includes a person and a robot: the robot’s only purpose is to make the person happy, but it doesn’t know what the person wants. Fortunately, it can learn more about what the person wants by observing her behavior. For example, if a robot observed the human’s morning routine, it should discover how important coffee is—not to itself, of course (we don’t want robots drinking coffee), but to the human. Then, it will make coffee for the person without being asked.

What about the example of King Midas, who thought he wanted as much gold as possible? How does the approach deal with someone stating their wishes incorrectly?

“We don’t have to know what we want. That’s the whole point. It’s hard to know what we want, and what we ask for is evidence for what we really want, but it’s not ground truth,” says Russell.

The robot should learn what people want by watching their actions. Then, if someone asks for something that’s technically incorrect, the robot will understand their underlying wish. Russell adds, “The robot—if it had been there—would have told Midas that he didn’t really want everything turned to gold, maybe just a few choice objects that he might point at from time to time and say the magic word.”

AI Off-Switch

Russell and his Berkeley colleagues also recently announced further progress toward ensuring safe AI, with a paper on ensuring an AI’s off-switch is always accessible. An off-switch is often suggested to guarantee a robot doesn’t harm humans – if the robot goes bad, just turn it off! But what if the AI anticipates that and disables its off switch?

Russell and his team propose solving this by providing a level of uncertainty in the robot’s objective. They created another CIRL game in which the robot chooses to announce its intentions, which gives the human a chance to switch it off, or to plunge ahead with its plan, bypassing human oversight.

They showed that a robot with a fixed objective will take the second choice and disable its off-switch. But if there is uncertainty about the objective, the robot will reason that a human decision to switch the robot off would imply the announced intention is undesirable. Because its goal is to make the human happy, the robot accepts being switched off. The robot has a positive incentive not to disable the off-switch, and that incentive is directly tied to the robot’s uncertainty about the human’s true objective.

Ensuring AI Safety

In addition to his research, Russell is also one of the most vocal and active AI safety researchers concerned with ensuring a stronger public understanding of the potential issues surrounding AI development.

He recently co-authored a rebuttal to an article in the MIT Technology Review, which claimed that real AI scientists weren’t worried about the existential threat of AI. Russell and his co-author summed up why it’s better to be cautious and careful than just assume all will turn out for the best:

“Our experience with Chernobyl suggests it may be unwise to claim that a powerful technology entails no risks. It may also be unwise to claim that a powerful technology will never come to fruition. On September 11, 1933, Lord Rutherford, perhaps the world’s most eminent nuclear physicist, described the prospect of extracting energy from atoms as nothing but “moonshine.” Less than 24 hours later, Leo Szilard invented the neutron-induced nuclear chain reaction; detailed designs for nuclear reactors and nuclear weapons followed a few years later. Surely it is better to anticipate human ingenuity than to underestimate it, better to acknowledge the risks than to deny them. … he risk arises from the unpredictability and potential irreversibility of deploying an optimization process more intelligent than the humans who specified its objectives.”

This summer, Russell received a grant of over $5.5 million from the Open Philanthropy Project for a new research center, the Center for Human-Compatible Artificial Intelligence, in Berkeley. Among the primary objectives of the Center will be to study this problem of value alignment, to continue his efforts toward provably beneficial AI, and to ensure we don’t make the same mistakes as King Midas.

“Look,” he says, “if you were King Midas, would you want your robot to say, ‘Everything turns to gold? OK, boss, you got it.’ No! You’d want it to say, ‘Are you sure? Including your food, drink, and relatives? I’m pretty sure you wouldn’t like that. How about this: you point to something and say ‘Abracadabra Aurificio’ or something, and then I’ll turn it to gold, OK?’”

This article is part of a Future of Life series on the AI safety research grants, which were funded by generous donations from Elon Musk and the Open Philanthropy Project.

Unfettered Science Essential to Decision Making; the Science Community Will Be Watching

WASHINGTON (November 30, 2016)—More than 2300 scientists from all fifty states, including 22 Nobel Prize recipients, released an open letter urging the Trump administration and Congress to set a high bar for integrity, transparency and independence in using science to inform federal policies. Some notable signers have advised Republican and Democratic presidents, from Richard Nixon to Barack Obama.

“Americans recognize that science is critical to improving our quality of life, and when science is ignored or politically corrupted, it’s the American people who suffer,” said physicist Lewis Branscomb, professor at the University of California, San Diego School of Global Policy and Strategy, who served as vice president and chief scientist at IBM and as director of the National Bureau of Standards under President Nixon. “Respect for science in policymaking should be a prerequisite for any cabinet position.”

The letter lays out several expectations from the science community for the Trump administration, including that he appoint a cabinet with a track record of supporting independent science and diversity; independence for federal science advisors; and sufficient funding for scientific data collection. It also outlines basic standards to ensure that federal policy is fully informed by the best available science.

For example, federal scientists should be able to: conduct their work without political or private-sector interference; freely communicate their findings to Congress, the public and their scientific peers; and expose and challenge misrepresentation, censorship or other abuses of science without fear of retaliation.

“A thriving federal scientific enterprise has enormous benefits to the public,” said Nobel Laureate Carol Greider, director of molecular biology and genetics at Johns Hopkins University. “Experts at federal agencies prevent the spread of diseases, ensure the safety of our food and water, protect consumers from harmful medical devices, and so much more. The new administration must ensure that federal agencies can continue to use science to serve the public interest.”

The letter also calls on the Trump administration and Congress to resist attempts to weaken the scientific foundation of laws such as the Clean Air Act and Endangered Species Act. Congress is expected to reintroduce several harmful legislative proposals—such as the REINS Act and the Secret Science Reform Act—that would increase political control over the ability of federal agency experts to use science to protect public health and the environment.

The signers encouraged their fellow scientists to engage with the executive and legislative branches, but also to monitor the activities of the White House and Congress closely. “Scientists will pay close attention to how the Trump administration governs, and are prepared to fight any attempts to undermine of the role of science in protecting public health and the environment,” said James McCarthy, professor of biological oceanography at Harvard University and former president of the American Association for the Advancement of Science. “We will hold them to a high standard from day one.”

Post-fundraiser update: Donors rallied late last month to get us most of the way to our first fundraiser goal, but we ultimately fell short. This means that we’ll need to make up the remaining $160k gap over the next month if we’re going to move forward on our 2017 plans. We’re in a good position to expand our research staff and trial a number of potential hires, but only if we feel confident about our funding prospects over the next few years.Since we don’t have an official end-of-the-year fundraiser planned this time around, we’ll be relying more on word-of-mouth to reach new donors. To help us with our expansion plans, donate at https://intelligence.org/donate/ — and spread the word!

Research updates

• Critch gave an introductory talk on logical induction (video) for a grad student seminar, going into more detail than our previous talk.
• New at IAFF: Logical Inductor Limts Are Dense Under Pointwise Convergence; Bias-Detecting Online Learners; Index of Some Decision Theory Posts
• We ran a second machine learning workshop.

General updates

• We ran an “Ask MIRI Anything” Q&A on the Effective Altruism forum.
• We posted the final videos from our Colloquium Series on Robust and Beneficial AI, including Armstrong on “Reduced Impact AI” (video) and Critch on “Robust Cooperation of Bounded Agents” (video).
• We attended OpenAI’s first unconference; see Viktoriya Krakovna’s recap.
• Eliezer Yudkowsky spoke on fundamental difficulties in aligning advanced AI at NYU’s “Ethics of AI” conference.
• A major development: Barack Obama and a recent White House report discuss intelligence explosion, Nick Bostrom’s Superintelligence, open problems in AI safety, and key questions for forecasting general AI. See also the submissions to the White House from MIRI, OpenAI, Google Inc., AAAI, and other parties.

News and links

• The UK Parliament cites recent AI safety work in a report on AI and robotics.
• The Open Philanthropy Project discusses methods for improving individuals’ forecasting abilities.
• Paul Christiano argues that AI safety will require that we align a variety of AI capacities with our interests, not just learning — e.g., Bayesian inference and search.
• See also new posts from Christiano on reliability amplification, reflective oracles, imitation + reinforcement learning, and the case for expecting most alignment problems to arise first as security problems.
• The Leverhulme Centre for the Future of Intelligence has officially launched, and is hiring postdoctoral researchers: details.

The following newsletter was originally posted on MIRI’s website.

Our big announcement this month is our paper “Logical Induction,” introducing an algorithm that learns to assign reasonable probabilities to mathematical, empirical, and self-referential claims in a way that outpaces deduction. MIRI’s 2016 fundraiser is also live, and runs through the end of October.

Research updates

• Shtetl-Optimized and n-Category Café discuss the “Logical Induction” paper.
• New at IAFF: Universal Inductors; Logical Inductors That Trust Their Limits; Variations of the Garrabrant-inductor; The Set of Logical Inductors Is Not Convex
• New at AI Impacts: What If You Turned the World’s Hardware into AI Minds?; Tom Griffiths on Cognitive Science and AI; and a Superintelligence excerpt on sources of advantage for digital intelligence.

General updates

• We wrote up a more detailed fundraiser post for the Effective Altruism Forum, outlining our research methodology and the basic case for MIRI.
• We’ll be running an “Ask MIRI Anything” on the EA Forum this Wednesday, Oct. 12.
• The Open Philanthropy Project has awarded MIRI a one-year $500,000 grant to expand our research program. See also Holden Karnofsky’s account of how his views on EA and AI have changed.

News and links

• Sam Altman’s Manifest Destiny: a profile by The New Yorker.
• In a promising development, Amazon, Facebook, Google, IBM, and Microsoft team up to launch a Partnership on AI to Benefit People and Society aimed at developing industry best practices.
• Alex Tabarrok vs. Tyler Cowen: “Will Machines Take Our Jobs?” (video)
• Google Brain makes major strides in machine translation.
• A Sam Harris TED talk: “Can we build AI without losing control of it?” (video)
• A number of updates from the Future of Humanity Institute.
• The Centre for the Study of Existential Risk is accepting abstracts (due Oct. 18) for its first conference, on such topics as “creating a community for beneficial AI.”
• Andrew Critch: Interested in AI alignment? Apply to Berkeley.

How many systems today rely on the Internet to run smoothly? If the Internet were to go down, that would disrupt work, government, financial transactions, communications, shipments, travel, entertainment – nearly every aspect of modern life could be brought to a halt. If someone were able to intentionally take down the Internet, how much damage could they cause?

Cybersecurity expert, Bruce Schneier, recently wrote a post, Someone is Learning to Take Down the Internet, which highlights the increasing number of attacks focused on “probing the defenses of the companies that run critical pieces of the Internet.”

In his post, Schneier explains that someone — he suspects a large nation state like Russia or China — has been systematically testing and probing various large and important Internet companies for weaknesses. He says companies like Verisign, which registers many major web and email addresses, have seen increasing, large-scale attacks against their systems. The attackers are forcing the companies to mount various defenses in response, giving the attackers a better idea of what defense capabilities the companies have and where their defenses may be weak.

Schneier writes, “Someone is extensively testing the core defensive capabilities of the companies that provide critical Internet services. […] It feels like a nation’s military cybercommand trying to calibrate its weaponry in the case of cyberwar. It reminds me of the US’s Cold War program of flying high-altitude planes over the Soviet Union to force their air-defense systems to turn on, to map their capabilities.”

At the moment, there doesn’t appear to be much that can be done about these attacks, but at the very least, as Schneier says, “people should know.”

Read Schneier’s full article here.

News and links

• The Open Philanthropy Project awards $5.6 million to Stuart Russell to launch an academic AI safety research institute: the Center for Human-Compatible AI.
• “Who Should Control Our Thinking Machines?“: Jack Clark interviews DeepMind’s Demis Hassabis.
• Elon Musk explains: “I think the biggest risk is not that the AI will develop a will of its own, but rather that it will follow the will of people that establish its utility function, or its optimization function. And that optimization function, if it is not well-thought-out — even if its intent is benign, it could have quite a bad outcome.”
• Modeling Intelligence as a Project-Specific Factor of Production: Ben Hoffman compares different AI takeoff scenarios.
• Clopen AI: Viktoriya Krakovna weighs the advantages of closed vs. open AI.
• Google X director Astro Teller expresses optimism about the future of AI in a Medium post announcing the first report of the Stanford AI100 study.
• Buzzfeed reports on efforts to prevent the development of lethal autonomous weapons systems.
• In controlled settings, researchers find ways to detect keystrokes via distortions in WiFi signals and jump air-gaps using hard drive actuator noises.
• Solid discussions on the EA Forum: Should Donors Make Commitments About Future Donations? and Should You Switch Away From Earning to Give?

See the original newsletter on MIRI’s website.

The new center will be funded, primarily, by a generous grant from the Open Philanthropy Project for $5,555,550. The center will focus on research around value alignment, in which AI systems and robots will be trained using novel methods to understand what a human really wants, rather than just relying on initial programming.

Russell is most well known as the co-author of Artificial Intelligence: A Modern Approach, which has become the standard textbook for AI students. However, in recent years, Russell has also become an increasingly strong advocate for AI safety research and ensuring that the goals of artificial intelligence align with the goals of humans.

In a statement to FLI, Russell (who also sits on the FLI Science Advisory Board) said:

“I’m thrilled to have the opportunity to launch a serious attack on what is — as Nick Bostrom has called it — ‘the essential task of our age.’ It’s obviously in the very early stages but our work (funded previously by FLI) is already leading to some surprising new ideas for what safe AI systems might look like. We hope to find some excellent PhD students and postdocs and to start training the researchers who will take this forward.”

An example of this type of research can be seen in a paper published this month by Russell and other researchers on Cooperative Inverse Reinforcement Learning (CIRL). In inverse reinforcement learning, the AI system or robot has to learn a human’s goals by observing the human in a real-world or simulated environment, and CIRL is a potentially more effective method for teaching the AI to achieve this. In a press release about the new center, the Open Philanthropy Project listed other possible research avenues, such as:

• “Value alignment through, e.g., inverse reinforcement learning from multiple sources (such as text and video).
• “Value functions defined by partially observable and partially defined terms (e.g. ‘health,’ ‘death’).
• “The structure of human value systems, and the implications of computational limitations and human inconsistency.
• “Conceptual questions including the properties of ideal value systems, tradeoffs among humans and long-term stability of values.”

Other funders include the Future of Life Institute and the Defense Advanced Research Projects Agency, and other co-PIs and collaborators include:

• Pieter Abbeel, Associate Professor of Computer Science, UC Berkeley
• Anca Dragan, Assistant Professor of Computer Science, UC Berkeley
• Tom Griffiths, Professor of Psychology and Cognitive Science, UC Berkeley
• Bart Selman, Professor of Computer Science, Cornell University
• Joseph Halpern, Professor of Computer Science, Cornell University
• Michael Wellman, Professor of Computer Science, University of Michigan
• Satinder Singh Baveja, Professor of Computer Science, University of Michigan

In their press release, the Open Philanthropy Project added:

“We also believe that supporting Professor Russell’s work in general is likely to be beneficial. He appears to us to be more focused on reducing potential risks of advanced artificial intelligence (particularly the specific risks we are most focused on) than any comparably senior, mainstream academic of whom we are aware. We also see him as an effective communicator with a good reputation throughout the field.”

News and links

• Complexity No Bar to AI: Gwern Branwen argues that computational complexity theory provides little reason to doubt that AI can surpass human intelligence.
• Bill Nordhaus, the world’s leading climate change economist, writes a paper on the economics of singularity scenarios.
• The Open Philanthropy Project has awarded Robin Hanson a three-year $265,000 grant to study multipolar AI scenarios. See also Hanson’s new argument for expecting a long era of whole-brain emulations prior to the development of AI with superhuman reasoning abilities.
• “Superintelligence Cannot Be Contained” discusses computability-theoretic limits to AI verification.
• The Financial Times runs a good profile of Nick Bostrom.
• DeepMind software reduces Google’s data center cooling bill by 40%.
• In a promising development, US federal regulators argue for the swift development and deployment of self-driving cars to reduce automobile accidents: “We cannot wait for perfect. We lose too many lives waiting for perfect.”

See the original newsletter on MIRI’s website.

Chinese  

The following article was originally posted on Futurism.com.

Recent advances in artificial intelligence have made it clear that our computers need to have a moral code. Disagree? Consider this: A car is driving down the road when a child on a bicycle suddenly swerves in front of it. Does the car swerve into an oncoming lane, hitting another car that is already there? Does the car swerve off the road and hit a tree? Does it continue forward and hit the child?

Each solution comes with a problem: It could result in death.

It’s an unfortunate scenario, but humans face such scenarios every day, and if an autonomous car is the one in control, it needs to be able to make this choice. And that means that we need to figure out how to program morality into our computers.

Vincent Conitzer, a Professor of Computer Science at Duke University, recently received a grant from the Future of Life Institute in order to try and figure out just how we can make an advanced AI that is able to make moral judgments…and act on them.

MAKING MORALITY

At first glance, the goal seems simple enough—make an AI that behaves in a way that is ethically responsible; however, it’s far more complicated than it initially seems, as there are an amazing amount of factors that come into play. As Conitzer’s project outlines, “moral judgments are affected by rights (such as privacy), roles (such as in families), past actions (such as promises), motives and intentions, and other morally relevant features. These diverse factors have not yet been built into AI systems.”

That’s what we’re trying to do now.

In a recent interview with Futurism, Conitzer clarified that, while the public may be concerned about ensuring that rogue AI don’t decide to wipe-out humanity, such a thing really isn’t a viable threat at the present time (and it won’t be for a long, long time). As a result, his team isn’t concerned with preventing a global-robotic-apocalypse by making selfless AI that adore humanity. Rather, on a much more basic level, they are focused on ensuring that our artificial intelligence systems are able to make the hard, moral choices that humans make on a daily basis.

So, how do you make an AI that is able to make a difficult moral decision?

Conitzer explains that, to reach their goal, the team is following a two path process: Having people make ethical choices in order to find patterns and then figuring out how that can be translated into an artificial intelligence. He clarifies, “what we’re working on right now is actually having people make ethical decisions, or state what decision they would make in a given situation, and then we use machine learning to try to identify what the general pattern is and determine the extent that we could reproduce those kind of decisions.”

In short, the team is trying to find the patterns in our moral choices and translate this pattern into AI systems. Conitzer notes that, on a basic level, it’s all about making predictions regarding what a human would do in a given situation, “if we can become very good at predicting what kind of decisions people make in these kind of ethical circumstances, well then, we could make those decisions ourselves in the form of the computer program.”

However, one major problem with this is, of course, that morality is not objective — it’s neither timeless nor universal.

Conitzer articulates the problem by looking to previous decades, “if we did the same ethical tests a hundred years ago, the decisions that we would get from people would be much more racist, sexist, and all kinds of other things that we wouldn’t see as ‘good’ now. Similarly, right now, maybe our moral development hasn’t come to its apex, and a hundred years from now people might feel that some of the things we do right now, like how we treat animals, is completely immoral. So there’s kind of a risk of bias and with getting stuck at whatever our current level of moral development is.”

And of course, there is the aforementioned problem regarding how complex morality is. “Pure altruism, that’s very easy to address in game theory, but maybe you feel like you owe me something based on previous actions. That’s missing from the game theory literature, and so that’s something that we’re also thinking about a lot—how can you make this, what game theory calls ‘Solutions Concept’—sensible? How can you compute these things?”

To solve these problems, and to help figure out exactly how morality functions and can (hopefully) be programmed into an AI, the team is combining the methods from computer science, philosophy, and psychology “That’s, in a nutshell, what our project is about,” Conitzer asserts.

But what about those sentient AI? When will we need to start worrying about them and discussing how they should be regulated?

THE HUMAN-LIKE AI

According to Conitzer, human-like artificial intelligence won’t be around for some time yet (so yay! No Terminator-styled apocalypse…at least for the next few years).

“Recently, there have been a number of steps towards such a system, and I think there have been a lot of surprising advances….but I think having something like a ‘true AI,’ one that’s really as flexible, able to abstract, and do all these things that humans do so easily, I think we’re still quite far away from that,” Conitzer asserts.

True, we can program systems to do a lot of things that humans do well, but there are some things that are exceedingly complex and hard to translate into a pattern that computers can recognize and learn from (which is ultimately the basis of all AI).

“What came out of early AI research, the first couple decades of AI research, was the fact that certain things that we had thought of as being real benchmarks for intelligence, like being able to play chess well, were actually quite accessible to computers. It was not easy to write and create a chess-playing program, but it was doable.”

Indeed, today, we have computers that are able to beat the best players in the world in a host of games—Chess and Alpha Go, for example.

But Conitzer clarifies that, as it turns out, playing games isn’t exactly a good measure of human-like intelligence. Or at least, there is a lot more to the human mind. “Meanwhile, we learned that other problems that were very simple for people were actually quite hard for computers, or to program computers to do. For example, recognizing your grandmother in a crowd. You could do that quite easily, but it’s actually very difficult to program a computer to recognize things that well.”

Since the early days of AI research, we have made computers that are able to recognize and identify specific images. However, to sum the main point, it is remarkably difficult to program a system that is able to do all of the things that humans can do, which is why it will be some time before we have a ‘true AI.’

Yet, Conitzer asserts that now is the time to start considering what the rules we will use to govern such intelligences. “It may be quite a bit further out, but to computer scientists, that means maybe just on the order of decades, and it definitely makes sense to try to think about these things a little bit ahead.” And he notes that, even though we don’t have any human-like robots just yet, our intelligence systems are already making moral choices and could, potentially, save or end lives.

“Very often, many of these decisions that they make do impact people and we may need to make decisions that we will typically be considered to be a morally loaded decision. And a standard example is a self-driving car that has to decide to either go straight and crash into the car ahead of it or veer off and maybe hurt some pedestrian. How do you make those trade-offs? And that I think is something we can really make some progress on. This doesn’t require superintelligent AI, this can just be programs that make these kind of trade-offs in various ways.”

But of course, knowing what decision to make will first require knowing exactly how our morality operates (or at least having a fairly good idea). From there, we can begin to program it, and that’s what Conitzer and his team are hoping to do.

So welcome to the dawn of moral robots.

This interview has been edited for brevity and clarity. 

Research updates

• A new paper: “A Formal Solution to the Grain of Truth Problem.” The paper was presented at UAI-16, and describes the first general reduction of game-theoretic reasoning to expected utility maximization.
• Participants in MIRI’s recently-concluded Colloquium Series on Robust and Beneficial AI (CSRBAI) have put together AI safety environments for the OpenAI Reinforcement Learning Gym. Help is welcome creating more safety environments and conducting experiments on the current set. Questions can be directed to rafael.cosman@gmail.com.

General updates

• We attended the White House’s Workshop on Safety and Control in AI.
• Our 2016 MIRI Summer Fellows Program recently drew to a close. The program, run by the Center for Applied Rationality, aims to train AI scientists’ and mathematicians’ research and decision-making skills.
• “Why Ain’t You Rich?“: Nate Soares discusses decision theory in Dawn or Doom. See “Toward Idealized Decision Theory” for context.
• Numerai, an anonymized distributed hedge fund for machine learning researchers, has added an option for donating earnings to MIRI “as a hedge against things going horribly right” in the field of AI.

News and links

• The White House is requesting information on “safety and control issues for AI,” among other questions. Public submissions will be accepted through July 22.
• “Concrete Problems in AI Safety“: Researchers from Google Brain, OpenAI, and academia propose a very promising new AI safety research agenda. The proposal is showcased on the Google Research Blog and the OpenAI Blog, as well as the Open Philanthropy Blog, and has received press coverage from Bloomberg, The Verge, and MIT Technology Review.
• After criticizing the thinking behind OpenAI earlier in the month, Alphabet executive chairman Eric Schmidt comes out in favor of AI safety research:

  Do we worry about the doomsday scenarios? We believe it’s worth thoughtful consideration. Today’s AI only thrives in narrow, repetitive tasks where it is trained on many examples. But no researchers or technologists want to be part of some Hollywood science-fiction dystopia. The right course is not to panic—it’s to get to work. Google, alongside many other companies, is doing rigorous research on AI safety, such as how to ensure people can interrupt an AI system whenever needed, and how to make such systems robust to cyberattacks.

• Dylan Hadfield-Mennell, Anca Dragan, Pieter Abbeel, and Stuart Russell propose a formal definition of the value alignment problem as “Cooperative Inverse Reinforcement Learning,” a two-player game where a human and robot are both “rewarded according to the human’s reward function, but the robot does not initially know what this is.” In a CSRBAI talk (slides), Hadfield-Mennell discusses applications for AI corrigibility.
• Jaan Tallinn brings his AI risk focus to the Bulletin of Atomic Scientists.
• Stephen Hawking weighs in on intelligence explosion (video). Sam Harris and Neil DeGrasse Tyson debate the idea at greater length (audio, at 1:22:37).
• Ethereum developer Vitalik Buterin discusses the implications of value complexity and fragility and other AI safety concepts for cryptoeconomics.
• Wired covers a “demonically clever” backdoor based on chips’ analog properties.
• CNET interviews MIRI and a who’s who of AI scientists for a pair of articles: “AI, Frankenstein? Not So Fast, Experts Say” and “When Hollywood Does AI, It’s Fun But Farfetched.”
• Next month’s Effective Altruism Global conference is accepting applicants.

• New paper: “Safely Interruptible Agents.” The paper will be presented at UAI-16, and is a collaboration between Laurent Orseau of Google DeepMind and Stuart Armstrong of the Future of Humanity Institute (FHI) and MIRI; see FHI’s press release. The paper has received (often hyperbolic) coverage from a number of press outlets, including Business Insider,Motherboard, Newsweek, Gizmodo, BBC News, eWeek, and Computerworld.
• New at IAFF: All Mathematicians are Trollable: Divergence of Naturalistic Logical Updates;Two Problems with Causal-Counterfactual Utility Indifference
• New at AI Impacts: Metasurvey: Predict the Predictors; Error in Armstrong and Sotala 2012
• Marcus Hutter’s research group has released a new paper based on results from a MIRIx workshop: “Self-Modification of Policy and Utility Function in Rational Agents.” Hutter’s team is presenting several other AI alignment papers at AGI-16 next month: “Death and Suicide in Universal Artificial Intelligence” and “Avoiding Wireheading with Value Reinforcement Learning.”
• “Asymptotic Logical Uncertainty and The Benford Test” has been accepted to AGI-16.

General updates

• MIRI and FHI’s Colloquium Series on Robust and Beneficial AI (talk abstracts and slides now up) has kicked off with opening talks by Stuart Russell, Francesca Rossi, Tom Dietterich, and Alan Fern.
• We visited FHI to discuss new results in logical uncertainty, our new machine-learning-oriented research program, and a range of other topics.

News and links

• Following an increase in US spending on autonomous weapons, The New York Times reports that the Pentagon is turning to Silicon Valley for an edge.
• IARPA director Jason Matheny, a former researcher at FHI, discusses forecasting and risk from emerging technologies (video).
• FHI Research Fellow Owen Cotton-Barratt gives oral evidence to the UK Parliament on the need for robust and transparent AI systems.
• Google reveals a hidden reason for AlphaGo’s exceptional performance against Lee Se-dol: a new integrated circuit design that can speed up machine learning applications by an order of magnitude.
• Elon Musk answers questions about SpaceX, Tesla, OpenAI, and more (video).
• Why worry about advanced AI? Stuart Russell (in Scientific American), George Dvorsky (inGizmodo), and SETI director Seth Shostak (in Tech Times) explain.
• Olle Häggeström’s new book, Here Be Dragons, serves as an unusually thoughtful and thorough introduction to existential risk and future technological development, including a lucid discussion of artificial superintelligence.
• Robin Hanson examines the implications of widespread whole-brain emulation in his new book, The Age of Em: Work, Love, and Life when Robots Rule the Earth.
• Bill Gates highly recommends Nick Bostrom’s Superintelligence. The paperback edition is now out, with a newly added afterword.
• FHI Research Associate Paul Christiano has joined OpenAI as an intern. Christiano has also written new posts on AI alignment: Efficient and Safely Scalable, Learning with Catastrophes,Red Teams, and The Reward Engineering Problem.