This is a short list of resources that explain the major risks from AI, with a focus on the risk of human extinction. This is meant as an introduction and is by no means exhaustive.

The basics – How AI could kill us all

• AI experts are increasingly afraid of what they’re creating by Kelsey Piper at Vox (2022) — A very accessible introduction to why AI “might kill us all”.
• The ‘Don’t Look Up’ Thinking That Could Doom Us With AI by Max Tegmark for TIME (2023) — An easy-to-read response to common objections, using references to the 2021 film.

Deeper dives into the extinction risks

• FAQ on Catastrophic AI Risk by Yoshua Bengio (2023) — One of the “godfathers of AI” addresses AI risks in a Q&A format.
• Most Important Century Series by Holden Karnofsky (2022) — Karnofsky argues the far future will look radically unfamiliar, and may be determined by the AI we develop this century. Highlights include AI Could Defeat All Of Us Combined and Why Would AI “Aim” To Defeat Humanity?
• The Need For Work On Technical AI Alignment by Daniel Eth (2023) — A semi-technical explanation of “the alignment problem”, how it could be catastrophic for humanity, and how we can solve it.

Academic papers

• Joseph Carlsmith (2022) — Is Power-Seeking AI an Existential Risk?
• Richard Ngo, Lawrence Chan, Sören Mindermann (2022) — The alignment problem from a deep learning perspective
• Karina Vold and Daniel R. Harris (2021) — How Does Artificial Intelligence Pose an Existential Risk?
• Benjamin S. Bucknall and Shiri Dori-Hacohen (2022) — Current and Near-Term AI as a Potential Existential Risk Factor
• Chan et al. (2023) — Harms from Increasingly Agentic Algorithmic Systems
• Acemoglu and Lensman (2023) — Regulating Transformative Technologies
• Dan Hendrycks, Mantas Mazeika and Thomas Woodside (2023) — An Overview of Catastrophic AI Risks

Videos and podcasts

• Why would AI want to do bad things? — Robert Miles (2018)
• How do we prevent the AIs from killing us? — Paul Cristiano on Bankless (2023)
• Pausing the AI Revolution? — Jaan Tallinn on The Cognitive Revolution (2023)
• The Case for Halting AI Development — Max Tegmark on the Lex Friedman Podcast (2023)
• Don’t Look Up – The Documentary: The Case For AI As An Existential Threat — DaganOnAI (2023)

Books

• The Alignment Problem by Brian Christian (2020)
• Life 3.0 by Max Tegmark (2017)
• Human Compatible: Artificial Intelligence and the Problem of Control by Stuart Russell (2019)
• Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World by Darren McKee (2023)

Additional AI risk areas – Other than extinction

• AI Now Institute research areas — resources on present AI harms, including accountability, climate, labour, privacy, biometric risks, large-scale AI models and more.
• Algorithmic Justice League education page — articles on AI issues like facial recognition, racial discrimination and social justice.
• Stepping back from the brink: Why multilateral regulation of autonomy in weapons systems is difficult, yet imperative and feasible — Frank Sauer for the IRRC (2020)
• The Risks of Autonomous Weapons Systems for Crisis Stability and Conflict Escalation in Future U.S.-Russia Confrontations — Burgess Laird for The RAND Blog (2020)
• ICRC Position Paper on Autonomous Weapons Systems — The International Committee of the Red Cross (2021)

Artificial intelligence (AI) holds great economic, social, medical, security, and environmental promise. AI systems can help people acquire new skills and training, democratize services, design and deliver faster production times and quicker iteration cycles, reduce energy usage, provide real-time environmental monitoring for pollution and air quality, enhance cybersecurity defenses, boost national output, reduce healthcare inefficiencies, create new kinds of enjoyable experiences and interactions for people, and improve real-time translation services to connect people around the world. For all of these reasons and many more researchers are thrilled with the potential uses of AI systems to help manage some of the world’s hardest problems and improve countless lives.

But in order to realize this potential, the challenges associated with AI development have to be addressed. This page highlights four complementary resources to help decision makers navigate AI policy: A dashboard that helps analyse the current documents published on the OECD website, a global landscape of national and international AI strategies; a list of prominent AI policy challenges and key recommendations that have been made to address them; and a list of AI policy resources for those hoping to learn more.

1. National Strategy Radar

The Future of Life Institute has partnered with PricewaterhouseCoopers in the development of an initiative to analyze the soft and hard law efforts to govern artificial intelligence (AI). The dashboard below was created with the help of a natural language processing tool that categorized documents downloaded from the OECD’s AI governance database in February of 2022. Further background information on this initiative is available in this blog post and users can expect periodic updates to this resource.

Summary View

This dashboard summarizes the distribution of AI documents published by governments and sorted by geography, year, and topic. The fact that a country lacks a bubble does not mean it lacks documents relevant to artificial intelligence. Rather, it indicates that they are not available within the OECD database.

Document View

This view gives an in depth look at all the documents individually, organized by their country of origin and the topics identified via a natural language processing-based dashboard developed by PwC.

2. AI Policy Challenges

This page is intended as an introduction to the major challenges that society faces when attempting to govern Artificial Intelligence (AI). FLI acknowledges that this list is not comprehensive, but rather a sample of the issues we believe are consequential.

Here are ten areas of particular concern for the safe and beneficial development of AI in the near- and far-future. These should be prioritised by policymakers seeking to prepare for and mitigate the risks of AI, as well as harness its benefits.

3. AI Policy Resources

The evolution of AI systems has proved to be so rapid and continuous that society now expects novel methods and applications everyday. To keep up, stakeholders in the public, private, and nonprofit worlds are responding with a variety of instruments, from soft to hard law, and academic or grey literature. As a result, the resources that describe and respond to the policy challenges generated by AI are always in flux. This page contains a few excellent resources to help you stay up to date.

The viral disease known as COVID-19 has claimed hundreds of thousands of lives, devastated the global economy and brought much of the world to a virtual standstill. Sluggish and insufficient government responses have hugely exacerbated the damage. And yet for researchers and modelers, the outbreak of a pandemic — as well as our underpreparedness for such an event — came as no surprise. Given the potential consequences, why didn’t we plan ahead?

As we begin to rebuild, it’s important to consider the failures and mistakes that helped fuel this crisis. What can we learn from COVID-19 about pandemic preparedness, and preparedness for other catastrophic risks? We asked experts from a variety of fields — risk management, medicine, biology, engineering and more — what they had to say. Browse their answers below.

Click on a name to read each expert’s full response:

• Yi Zeng | Chinese Academy of Sciences/BAAI

• Anthony Aguirre | UC Santa Cruz

• Emilia Javorsky | FLI

• Huw Price | University of Cambridge/CSER

• Jason Matheny | CSET

• Matthew Meselson | Harvard University

• Clarissa Rios Rojas | CSER

• Stuart Russell | UC Berkeley

• Alan Robock | Rutgers University

• Toby Ord | FHI

• Jaan Tallinn | Skype

Background information

The Disease

COVID-19 is an infectious disease caused by the SARS-CoV-2 virus (severe acute respiratory syndrome coronavirus 2), a novel coronavirus. Coronaviruses are a large family of viruses that occur in humans and animals, and it is believed that the SARS-CoV-2 virus originated in bats.

The most common symptoms of COVID-19 are fever, tiredness, and dry cough. Other symptoms may include aches and pains, nasal congestion, runny nose, sore throat and diarrhea. Roughly 80% of people with COVID-19 recover without needing special treatment. Around 1 in 6 people becomes seriously ill and develops difficulty breathing. Older people, as well as those with pre-existing medical conditions such as high blood pressure, diabetes or heart problems, are more likely to develop serious symptoms. 

To date, research suggests that SARS-CoV-2 is transmitted primarily through respiratory droplets from the nose or mouth of an infected person. It is possible to contract the virus by directly inhaling these droplets, or by touching a surface where droplets have landed and then touching the eyes, nose or mouth.  

Sources:

https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/summary.html#background

https://www.who.int/news-room/q-a-detail/q-a-coronaviruses

The Pandemic

The current COVID-19 outbreak began in the city of Wuhan, China — a transportation hub with a population of 11 million. By the end of December 2019, doctors had seen dozens of patients with a form of viral pneumonia that did not respond to treatment. By this point, estimates suggest that as many as 1,000 people were already infected.

In the first 3 weeks of January, before travel was restricted, roughly 7 million people left Wuhan. On January 20th, the first cases outside China appeared in Japan, Thailand and South Korea. On January 21st, the first U.S. case was reported in Washington State. On January 24th, the first European cases were reported in France. By the end of January, when Wuhan was placed under lockdown, outbreaks were growing in 30 cities in 26 countries. 

By March 1st, Italy, Iran and South Korea were each reporting thousands of cases. On March 11th, the World Health Organization declared COVID-19 a pandemic. By March 17, all 50 U.S. states were reporting cases. 

Sources:

https://www.nytimes.com/interactive/2020/03/22/world/coronavirus-spread.html

https://www.npr.org/sections/goatsandsoda/2020/03/30/822491838/coronavirus-world-map-tracking-the-spread-of-the-outbreak

Further Resources

Additional COVID-19 materials from sources we trust:

Health Security Headlines Newsletter | John Hopkins University Center for Health and Security

National Coronavirus Response: A Roadmap to Reopening | American Enterprise Institute

Ready or Not 2020: Protecting the Public’s Health from Diseases, Disasters and Bioterrorism | Trust for America’s Health

COVID-19 Resource Center | Center for Infectious Disease Research and Policy

Coronavirus Disease Resources | World Health Organization

Coronavirus Series | ProPublica

Coronavirus Tech Report Newsletter | MIT Technology Review

Coronavirus Outbreak Resources | The New York Times

Coronavirus Forecasts | Metaculus

Make sure to check back here, as we’ll be updating this page throughout the month.

Rasha Abdul Rahim

Deputy Director of Amnesty Tech, Amnesty International

“[W]hen people around the world and civil society can think of a potent idea that’s worth fighting for, and stick at the concept however long it may take, and develop the proposal to get traction from political leaders, we really can make a difference.”

Elizabeth Barnes

Safety Team Member, OpenAI

“You can probably learn things much faster than you expect. It’s easy to think that learning some new skill will be impossibly hard. I’ve been surprised a lot of times how quickly things go from being totally overwhelming and incomprehensible to pretty alright.”

Rebecca Boehm

Economist, Union of Concerned Scientists

“The recent elevation of conversations about the importance of racial equity and inclusion makes me very hopeful for our future. I believe solving the big food and agricultural issues we are facing will require not only the voices, but the leadership of a diverse set of people.”

Liv Boeree

Co-founder, Raising for Effective Giving (REG) | Ambassador, www.effectivegiving.org

“I’ve found the [existential risk] community extremely welcoming and respectful, so I’d recommend it highly to any woman who is interested in pursuing work in this area.”

Astrid Caldas

Senior Climate Scientist, Union of Concerned Scientists

“Learn as much as you can not only from academic institutions or NGOs, but from people on the frontlines and those who are being the most impacted by climate change. Attend events, visit places if you can, to see first hand how people are dealing with the issues, and find out how you can help them become more resilient. Sometimes it is as simple as showing them a website they didn’t know about, or telling them about grants and other resources to protect their homes from floods.”

Rosie Campbell

Assistant Director, Center for Human-Compatible AI (CHAI) at UC Berkeley

“I’m a big advocate for diversity. We’re trying to solve big, important problems, and it’s worrying to think we could be missing out on important perspectives. I’d love to see more women in AI safety!”

Sonia Cassidy

Director Of Operations, Alliance to Feed the Earth in Disasters (ALLFED)

“Do not ever underestimate yourself and what women bring into the world, this field or any other. Our talents and skills are needed, and so are you!”

Carla Zoe Cremer

Research Affiliate, Centre for the Study of Existential Risk | Researcher, Leverhulme Centre for the Future of Intelligence

“My ideas are taken seriously and my work is appreciated. The problems in existential risk are hard, unsolved and numerous — which means that everyone welcomes your initiative and contributions and will not hold you back if you try something new.”

Kristina Dahl

Senior Climate Scientist, Union of Concerned Scientists

“Climate change is at this incredible nexus of science, culture, policy, and the environment. To do this job well, one has to bring to the table a love of the environment, a willingness to identify and fight for the policies needed to protect it, a sensitivity to the diverse range of decisions people make in their daily lives, and a fascination with the nitty-gritty bits of the science.”

Jeanne Dietsch

State Senator, NH | Founder and CEO of multiple tech startups

“Entrepreneurs: make sure that your company is competitive, that you have innovative processes and/or products. And I will paraphrase Michael Bloomberg: ‘Hire honest people who are smarter than yourself.'”

Anca Dragan

Assistant Professor, UC Berkeley

“I’m hopeful that progress in intelligence and AI tools can lead to freeing up more people to spend more time on education and creative pursuits — I think that would make for a wonderful future for us.”

Photo: Human-Machine Interaction / Anca Dragan / Photos Copyright Noah Berger / 2016

Beatrice Fihn

Executive Director, ICAN

“Don’t be too intimidated or impressed by senior people and ‘important’ people. Most of them don’t actually know as much as they come across as knowing.”

Danit Gal

Project Assistant Professor, Cyber Civilization Research Center, Keio University

“If you find something that moves you — be it further developments in an established field, a way to combine existing fields to create new ones, or something that’s entirely off the beaten path — pursue it. The act of pursuing the things that fascinate you is the real experience you need. If you can combine this with something that’s useful and beneficial to this world, you’ve won the game.”

Paula Garcia

Energy analyst, Union of Concerned Scientists

“Read, talk to others that work in the renewable energy industry, identify where in the value chain you want to contribute, and go for it!”

Rose Hadshar

Project Manager, Research Scholars Programme, Future of Humanity Institute

“[S]o many extremely able people are trying to make [the future] good.”

Emilia Javorsky

Director, Scientists Against Inhumane Weapons

“I’m a pretty optimistic person at baseline, but particularly so after getting to know the incredible people that compose the x-risk community. They care so deeply about engineering a positive future for humanity — I feel tremendously grateful to have the opportunity to work with them!”

Natalie Jones

PhD Student, University of Cambridge | Research Affiliate, CSER

“The other people working in this field are so fiercely intelligent and capable. It’s hard not to have a conversation which leaves you with a perspective or idea you hadn’t thought of before. This, and the knowledge that one is doing useful and important work, combine to make it very rewarding.”

Jade Leung

Head of Research and Partnerships, Center for the Governance of Artificial Intelligence, University of Oxford

“It is a rare, sometimes overwhelming, always humbling privilege to be in a position to work directly on a challenge which I believe is one of the most important facing us this century.”

Cassidy Nelson

Research Scholar, Future of Humanity Institute, University of Oxford

“I feel I’m surrounded by people who care deeply about life and addressing large and complex risks. I feel this field’s focus, while grim on its own, is also intrinsically coupled with the desire and hope that the future can go well. I remain hopeful that if we can navigate the next century safely, a better existence awaits us and our descendants. I am inspired by what could be possible for conscious life and I hope that my career can help ensure no catastrophic event occurs before our future is secured.”

Charlie Oliver

Founder/CEO, TECH 2025 (Served Fresh Media)

“Don’t allow other people to define your dreams and don’t allow them to place limits on what you can do. And just as important, if not more so, don’t limit your own potential with soul-crushing self-doubt. A little self-doubt is okay and quite normal. But when it begins to keep you from taking big risks necessary to discover your strengths and path, you have to fix that right away or that type of thinking will fester.”

Marie-Therese Png

PhD Student, Oxford Internet Institute

“Underrepresented perspectives — women, people of colour, and other intersectional identities — are highly valuable at this point in uncovering blindspots. Your concerns may not currently be represented in the research community, but it doesn’t mean they shouldn’t be. There is low replaceability because if you weren’t there it wouldn’t be any single person’s main focus. When you’re a minority in the room it’s even more important to overcome audience inhibition and speak up or a blindspot may persist.”

Carina Prunkl

Senior Research Scholar, Future of Humanity Institute, University of Oxford

“AI is a really exciting field to work in and there is a real need for people with diverse academic backgrounds – you don’t need to be a coder to make substantial contributions. Make use of existing women networks or write directly to women researchers if you would like to know what it is like to work at a particular organisation or with a particular team. Most of us are more than happy to help and share our experiences.”

Francesca Rossi

AI Ethics Global Leader and Distinguished Research Staff Member, IBM Research

“My advice to women is to believe in what they are and what they are passionate about, to behave according to their values and attitudes without trying to mimic anybody else, and to be fully aware that their contribution is essential for advancing AI in the most inclusive, fair, and responsible way.”

Susi Snyder

Managing Director, Don’t Bank on the Bomb, PAX & ICAN

“Find your passion, produce the research that supports your policy recommendation and demand the space to say your piece. I always think to the first US woman that ran on a major party for President- Shirley Chisholm, she said “if they don’t give you a seat, bring a folding chair”.  I think about the fact that there are (some) more seats now, and that’s amazing. There is still a long, long way to go before equity, but there are some serious efforts to move closer to that day.”

Bing Song

Vice President, Berggruen Institute | Director of the Institute’s China Center

“Women should embrace and dive into this new area of thinking about the future of humanity. Male dominance in past millennia in shaping the world and in how we approach the universe, humanity, and life needs to be questioned. More broad based, inclusive, non-confrontational and equanimous thinking, which is more typically associated with the female approach to things, is sorely needed in this world.”

Shuchi Talati

Geoengineering Research, Governance and Public Engagement Fellow, Union of Concerned Scientists

“Domestic and international dedication to addressing climate change is continuously growing. Though we are far from where we need to be, I remain optimistic that we’re on a promising path.”

Mary Wareham

Coordinator of the Campaign to Stop Killer Robots | Advocacy Director of Human Rights Watch arms division

“Study what you are passionate about and not what you think will get you a job.”

Jody Williams

Chairwoman, Nobel Women’s Initiative | Nobel Laureate

“If I have advice, it would be to be clear about who you want to be in your life and what you stand for — and then go for it.”

Bonnie Wintle

Research Fellow, School of Biosciences, University of Melbourne | Research Affiliate, Centre for the Study of Existential Risk (CSER), University of Cambridge

“Seeing the huge turnout of school kids and young people at climate change demonstrations gives me hope for the future. The next generation of leaders and decision makers seem to be proactive and genuinely interested in addressing these problems.”

Baobao Zhang

PhD Candidate, Political Science, Yale University | Research Affiliate, Center for the Governance of AI, University of Oxford

“AI policy is a nascent but rapidly growing field. I think this is a good time for women to enter the field. Sometimes women are hesitant to enter a new discipline because they don’t feel they have adequate knowledge or experience. My work has taught me that you can quickly learn on the job and that you can apply the skills and knowledge you already have to your new job.”

Meia Chita-Tegmark

Co-founder, Future of Life Institute | Postdoctoral Scholar, Tufts University

“Be brave. This is our world too, we can’t let it be shaped by men alone.”

Ariel Conn

Director of Communications/Outreach and Weapons Policy Advisor, Future of Life Institute

“Success in this job comes with much greater satisfaction than success in any other job I’ve had.”

Jessica Cussins Newman

AI Policy Specialist, Future of Life Institute | Research Fellow, UC Berkeley Center for Long-Term Cybersecurity

“Don’t discount yourself just because you think you don’t have the right background — the field is actively looking for ways to learn from other disciplines.”

Victoria Krakovna

Cofounder, Future of Life Institute | Research Scientist, DeepMind

“It’s great to see more and more talented and motivated people entering the field to work on these interesting and difficult problems.”

Nine countries currently hold approximately 14,000 nuclear weapons — the U.S. and Russia have most of these — and thousands are ready to be launched on command. The risk that nuclear weapons will be used seems to increase daily, and over a trillion dollars are scheduled for more lethal, more easy-to-use, and more destabilizing nuclear weapons.

Most of that money will come from taxes, but it will be paid to companies that produce a variety of products. In fact, for many companies, the nuclear weapons and components they create are only a small fraction of total production. By stigmatizing these companies with actions like divestment and boycotts, it can become financially beneficial for them to pull out of nuclear weapons production.

The rest of this page provides all the resources you need to learn about nuclear divestment and take action.

Are you investing in nuclear weapons?

Before you can divest from nuclear weapons, it helps to know what you’re invested in. FLI has put together a divestment/investment tool to help you figure out if you’re invested in nuclear weapons and which mutual funds are nuclear-free. Just type in the funds you have now and go:

Responsible Investing Made Easy

It also helps to know which companies are connected to nuclear weapons production. While we list the companies to avoid in the tool above, we got all of our information from Don’t Bank on the Bomb, which provides a much more comprehensive resource on who the nuclear weapons companies are:

Nuclear Weapons Producers

Even with all of this knowledge, the process of divesting can seem daunting. We suggest learning from the fossil fuel divestment campaign. Check out the resource below for tips and suggestions to turn the idea of divestment into action:

Climate Reinvestment

For Industry, Academia, Nonprofit Organizations, and State and Local Governments

Companies, universities, churches, non-profits, local governments — any organization that has investments — all have tremendous potential to influence nuclear-weapons-producing companies. These institutions usually have money in investments, pension funds, etc., that they can divest from nuclear weapons. For example, in 2016, the city council of Cambridge, MA voted to divest their $1 billion pension fund from nuclear weapons producing companies. If other organizations and institutions follow suit, that could send a powerful message to companies that build the world’s deadliest weapons.

We’ve produced a guide for city’s to divest from nuclear weapons, which was handed out to the Mayors for Peace at the US Conference of Mayors in 2017, but many of the suggestions could apply to any organization or institution.

City’s Guide to Nuclear Weapons Divestment

For tips on how local governments can create policies to divest from nuclear weapons, we recommend the Policy Checklist put together by Don’t Bank on the Bomb:

Create a Nuclear Weapons Free Investment Policy

For examples of what a nuclear divestment policy might look at, please see the Cambridge City Council’s policy order:

Cambridge City Council Policy Order

For Individuals

Nuclear weapons might seem like a daunting problem for just one person to try to address, but there’s plenty that a single individual can do to help reduce the threat of nukes. Obviously, if you’re invested in nuclear weapons, you can use the tools above to divest and reinvest your money in socially responsible funds. But if for any reason you can’t divest or if you don’t have money to invest, there’s still plenty you can do to help get money out of nuclear weapons companies.

Whether you have investments or not, you likely work with some type of financial institution or bank. Don’t Bank on the Bomb offers a great resource that walks you through the steps of convincing your financial institution to divest from nuclear weapons companies. It can be surprising how quickly an institution will divest when even just a few people reach out!

You Decided to Invest, What Now?

But don’t stop with your own financial institutions! If you want to more actively join the divestment movement, Don’t Bank on the Bomb has also put together an extensive document with great steps you can take to help the divestment campaign grow:

Campaigner Guide

Or if you prefer a shorter summary of actions you can take:

Take Action for Divestment

Divestment Works

The primary reason we encourage divestment is because we’ve seen the positive impact it’s had in the past, especially with regard to stigmatizing companies:

• From a report by The New Republic: “On tobacco, divestment helped to highlight the decades-old tactics the industry used to mislead consumers on the health impacts of smoking. The scrutiny paved way for new taxes, limits on smoking in public spaces, and other policies documented to change public behavior. The symbolic resonance of schools like Harvard, University of Michigan, and Stanford divesting was much more important than the amount of money pulled. … By financial measures, the campaign was a failure—since the ’90s, tobacco companies’ stocks have only grown, a sign that divestment didn’t hurt the industry’s long-term success. ”
• From the Stranded Assets program: ‘’’in almost every divestment campaign we reviewed from adult services to Darfur, from tobacco to South Africa, divestment campaigns were successful in lobbying for restrictive legislation affecting stigmatized firms’. For example, even though the share prices of tobacco companies have performed well throughout the divestment campaign, the industry has faced increasing restrictive legislation.”
• From the UN: “While it is unlikely that divestment by a single financial institution or government would create sufficient pressure on a company for it to end its involvement in nuclear weapons production, divestment by even a few institutions or states based on the same justification can impact a company’s strategic direction.”
• Examples: “Motorola divested from its defense business in order to stave off bad publicity in reputable news outlets. Similarly, Revlon’s decision to disinvest its South African operation was due to credible threats by customer groups to boycott Revlon products.”
• From the World Post: “US company Lockheed Martin, which describes itself as the world’s largest arms manufacturer, announced it ceased its involvement with the production of rockets, missiles or other delivery systems for cluster munitions and stated it will not accept such orders in the future. The arms manufacturer expressed the hope that its decision to cease the activities in the area of cluster munitions would enable it to be included in investors portfolios again, thereby suggesting that pressure by financial institutions had something to do with its decision.”
• From the UN: “Textron also recently announced it would stop the production of cluster munitions and commentators in financial media suggested that one reason for this decision would be to increase the ‘ownability’ of Textron shares for mainly European investors, which largely exclude cluster munitions producers from financing. This suggests that pressure by financial institutions was a contributing factor in Lockheed Martin and Textron’s decisions to end involvement with cluster munitions production.

Learn More About Nuclear Weapons

General Info

Federation of American Scientists: Nuclear Weapons

Nuclear Darkness: The Deadly Consequences of Nuclear War

Union of Concerned Scientists: Prevent Nuclear War

International Campaign to Abolish Nuclear Weapons (ICAN): The Facts

FLI: The Risk of Nuclear Weapons

Money and Nukes

MinutePhysics: Why You Should Care About Nukes

FLI: Trillion Dollar Nukes App

Nuclear close calls

FLI: Timeline of Nuclear Close Calls

Union of Concerned Scientists: I Wish I Didn’t Know That – Real-life Tales of Close Calls, Screw Ups, and Nuclear Near Misses

Union of Concerned Scientists: Close Calls with Nuclear Weapons

Nuclear weapons ban

ICAN: Why a Ban

ICAN: Status of the Treaty on the Prohibition of Nuclear Weapons

Declassified US nuclear targets

FLI: 1100 Declassified US Nuclear Targets

More Resources
There are so many great organizations working to rid the world of nuclear threats. Learn more about the what they’re doing and consider joining their efforts.

Abolition 2000: An international global network of organisations and individuals working for a global treaty to prohibit and eliminate nuclear weapons.

Acronym Institute for Disarmament Diplomacy: Develops and supports multilateral disarmament and security agreements

The Center for Arms Control and Non-Proliferation: National non-partisan, non-profit dedicated to enhancing peace and security through expert policy analysis and thought-provoking research.

Cambridge Center for Existential Risk: Multidisciplinary research center within the University of Cambridge to study and mitigate existential risks

Campaign for Nuclear Disarmament: Advocacy group that seeks nuclear disarmament in Britain and the establishment of an international treaty banning all nuclear weapons

Center for Policy on Emerging Technologies (C-PET): A wide network of knowledgeable professionals who identify, clarify, and prioritize the big questions raised in a future perspective to cultivate a context within which solutions can be developed.

Don’t Bank on the Bomb: A global effort to discourage investments in companies that help build or upgrade nuclear weapons.

Defense Threat Reduction Agency: Official U.S. combat support agency for countering weapons of mass destruction, including nuclear, biological, radiological, chemical, and high yield explosive threats.

Federation of American Scientists: Think tank focused on providing analysis and policy recommendations on national and international security issues related to applied science and technology

Future of Humanity Institute: Oxford-based research organization analyzing big-picture questions for human civilization in an attempt to help shape the future of humanity

Global Catastrophic Risk Institute: Think tank leading research, education, and professional networking on the full breadth of major global catastrophic risks

Global Zero: International movement for the elimination of nuclear weapons. Powered by a visionary group of 300 international leaders and experts who support our bold, step-by-step plan to eliminate all nuclear weapons by 2030, the relentless creativity, energy and optimism of young people and half a million citizens worldwide, Global Zero is challenging the 20th century idea of basing national security on the threat of mass destruction.

Human Survival Project: Seeks to protect humanity from nuclear holocaust and its societal and environmental aftermath; a joint initiative of the University of Sydney Centre for Peace and Conflict Studies at the and People for Nuclear Disarmament

The International Campaign to Abolish Nuclear Weapons (ICAN): Coalition of non-governmental organizations in one hundred countries promoting adherence to and implementation of the United Nations nuclear weapon ban treaty. This landmark global agreement was adopted in New York on 7 July 2017, and ICAN was awarded the Nobel Peace Prize on December 10, 2017.

Institute for Science and International Security: Analyzes and communicates to the public on the science and policy behind nuclear nonproliferation and related international security issues, with emphasis on tracking nuclear weapons programs worldwide.

James Martin Center for Nonproliferation Studies: Middlebury College-based organization focused on reducing the spread of weapons of mass destruction and training future nonproliferation experts

Leverage Research: A team of researchers, inventors, educators, and activists focused on improving the world. Their research includes reducing catastrophic risk from totalitarianism and artificial general intelligence

Lifeboat Foundation: A non-profit organization encouraging scientific advancement while reducing existential risk and other possibly harmful outcomes from emerging technologies, including nanotechnology, robotics, AI, and genetic engineering

Mayors for Peace: Close cooperation among the cities that strives to raise international public awareness regarding the need to abolish nuclear weapons. It contributes to the realization of genuine and lasting world peace by working to eliminate starvation and poverty, assist refugees fleeing local conflict, support human rights, protect the environment, and solve the other problems that threaten peaceful coexistence within the human family.

National Security Archive: Library of declassified U.S. documents related to nuclear weapons, government secrecy, and other issues based at George Washington University; also supports research and investigative journalism

Nautilus Institute: Assembles professionals and conducts research to help solve threats of nuclear war, urban and energy insecurity, and climate change with a focus on the Asia-Pacific region

Nuclear Threat Initiative: Non-profit organization focused on strengthening global security by reducing the societal risk posed by nuclear, biological, and chemical weapons

Outrider Group: With their interactive features and in-depth articles, Outrider Group seeks to educate the public about the risks of nuclear war and climate-induced catastrophe and explain how people can become part of the solution.

Pax Christi USA (PCUSA): Grounded in the Gospel and Catholic social teaching, PCUSA is a membership organization that rejects war, preparation for war, every form of violence and domination, and personal and systemic racism. As PCUSA, a section of Pax Christi International, we are a Catholic peace and justice movement that seeks to model the Peace of Christ in our witness to the mandate of the nonviolence of the Cross.

Peace Action: The nation’s largest grassroots peace network with chapters and affiliates in states across the country.

People for Nuclear Disarmament: A citizens?? group based in Sydney promoting nuclear disarmament and related issues

Ploughshares Fund: Supports analysis and advocacy towards improving global security and peace via reductions in nuclear arsenals

Parliamentarians for Nuclear Non-proliferation and Disarmament (PNND): Non-partisan forum for parliamentarians nationally and internationally to share resources and information, develop cooperative strategies and engage in nuclear nonproliferation and disarmament issues, initiatives and arenas.

Physicians for Social Responsibility (PSR): Working for more than 50 years to create a healthy, just and peaceful world for both the present and future generations. PSR advocates on the issues you care about by addressing the dangers that threaten communities, such as nuclear weapons.

Program on Science and Global Security: Research group based at Princeton University studying nuclear weapons, biosecurity, and related issues

Pugwash Conferences on Science and World Affairs : Bring together scientists, academics, and public policy specialists to discuss the latest scientific insight to help reduce threats facing global society arising from science and technology, such as nuclear weapons and other WMDs

Reaching Critical Will: A Women??s International League for Peace and Freedom program that works to increase the influence of NGOs in UN disarmament processes including for nuclear and other weapons

Skoll Global Threats Fund: Promotes research, education, governance, and outreach towards confronting global catastrophic risks

Stimson Center: Washington D.C. based think tank focused on improving global security and reducing weapons of mass destruction

The Union of Concerned Scientists: Works on developing and implementing innovative, practical solutions to some of our planet’s most pressing problems—from combating global warming and developing sustainable ways to feed, power, and transport ourselves, to fighting misinformation, advancing racial equity, and reducing the threat of nuclear war.

US Department of Defense Nuclear Posture Review: A legislatively-mandated review that establishes U.S. nuclear policy, strategy, capabilities and force posture for the next five to ten years.

US National Nuclear Security Administration : Agency within the U.S. Department of Energy charged with managing and securing U.S. nuclear weapons, nuclear nuclear nonproliferation, and naval reactors

William J. Perry Project: Created by Former US Secretary of Defense William Perry, this project seeks to stimulate an informed and broadly inclusive public conversation about the role of nuclear weapons in today’s world and to work toward a world in which nuclear weapons are never used again.

Woodrow Wilson International Center for Scholars: Washington D.C. based research center focused on bridging scholars with policy experts in a host of fields, including global environmental change, nuclear proliferation, international security, and global sustainability and resilience

World Beyond War: Global nonviolent movement to end war and establish a just and sustainable peace.

World Institute for Nuclear Security: Helps secure nuclear and radioactive materials from theft, unauthorized access, and sabotage

• Frequently Asked Questions about the Future of Artificial Intelligence

Open Letters

• AI Open Letter: An open letter on maximizing the societal benefits of AI (Jan 11, 2015).
• Digital Economy Open Letter: An open letter by a team of economists about AI’s future impact on the economy. It includes specific policy suggestions to ensure positive economic impact. (Jun 4, 2015)
• Autonomous Weapons Open Letter: An open letter from AI and robotics researchers urging a ban on offensive autonomous weapons beyond meaningful human control (Jul 28, 2015).
• Asilomar AI Principles: A set of principles to guide beneficial AI development (Jan 30, 2017).
• Open Letter to the United Nations Convention on Certain Conventional Weapons: An open letter from leaders of AI and robotics companies calling for a UN ban on lethal autonomous weapons (Aug 20, 2017).
• Pledge Against Lethal Autonomous Weapons: Companies and individuals pledge that they will not participate in nor support the development, manufacture, trade, or use of lethal autonomous weapons.
• 2018 Statement to the UN on Behalf of LAWS Open Letter Signatories: This statement was read on the floor of the United Nations during the August, 2018 CCW meeting, in which delegates discussed a possible ban on lethal autonomous weapons.
• 2019 Statement to the United Nations in Support of a Ban on LAWS: This statement was read on the floor of the United Nations during the March, 2019 CCW meeting, in which delegates discussed a possible ban on lethal autonomous weapons.

Grants Program

• Elon Musk donates $10M to keep AI beneficial: Press release for Elon Musk’s donation and grants program (Jan 15, 2015).
• Request for proposals, FAQ, Timeline: Materials for the 2015 international grants competition – now closed (Jan 22, 2015).
• New International Grants Program Jump-Starts Research to Ensure AI Remains Beneficial: Press release about FLI grant program results (Jul 1, 2015).
• 2015 Grants Recommended for Funding: FLI grant awardees listing (Jul 1, 2015).
• AI Safety Research: Profiles of AI safety researchers funded by FLI (Oct 4, 2016).
• Elon Musk donates $2M to keep AGI beneficial: Press release for the second round of AI safety grants which will focus on AGI safety (July 25, 2018).

Research Priorities

• Research priorities for robust and beneficial AI: A summary of the research areas covered by our grants program.
• A survey of research questions for robust and beneficial AI: A collection of example projects and research questions within each area.

Videos

• Prof. Marc Lipsitch: Risks and Benefits of Gain-of-Function Experiments in Potentially Pandemic Pathogens
• Cathal Garvey: Bringing Biotechnology into the Home (TEDx Talk): In this TEDx talk, Cathal Garvey reviews the benefits of biotechnology and introduces consumer biotechnologies in development; such as bioprinters, customized pets, downloadable DNA, and biochips.
• Nina Tandon: Caring for Engineered Tissue (TED Talk): In this TED talk Nina Tandon talks about simple but powerful methods of caring for artificially grown cells by copying their natural environment.
• Gregory Stock: To Upgrade is Human (TED Talk): Biotech ethicist Gregory Stock talks about new, more meaningful (and controversial) technologies, like customizable babies, whose adoption might drive human evolution.
• Richard Resnick: Welcome to the Genomic Revolution (TEDx Talk): In this accessible talk from TEDxBoston, Richard Resnick shows how cheap and fast genome sequencing is about to turn health care (and insurance, and politics) upside down.
• Ellen Jorgensen: Biohacking – You Can Do It Too (TED Talk): In this TED talk, Ellen Jorgensen introduces us to the biohacking community and reviews common misconceptions about the dangers of biohacking.

Research Papers

• Bruce Schneier – Resources on Existential Risk , p. 103

Books

• Biotechnology for Beginners: An overview of biotechnology for beginners and lay readers. Includes a wide array of biotech sciences such as: genetics, immunology, biochemistry, agronomy, food science, and animal science.
• Biotechnology and the Human Good
• Fighting for the Future of Food 
• Introduction to Biotechnology
• Our Posthuman Future: Consequences of the Biotechnology Revolution

Organizations

• Cambridge Working Group: Works to address the risks of Potential Pandemic Pathogen research through promoting public understanding, risk assessment and regulation of such research.
• etc Group: Works to address the socioeconomic and ecological issues surrounding new technologies (especially agriculture) that could have an impact on the world’s poorest and most vulnerable people.
• American Biological Safety Association (ABSA): Promotes biosafety as a scientific discipline through forums and peer-reviewed journals for biosafety professionals throughout the world.
• Biomedical Advanced Research and Development Authority (BARDA): Develops and procures needed medical countermeasures (MCMs), including vaccines, therapeutics, diagnostics, and non-pharmaceutical countermeasures, against a broad array of public health emergencies in the US.
• Biotechnology Industry Organization (BIO): A trade association representing biotechnology companies, academic institutions, state biotechnology centers, and related organizations around the world. Researches and developes innovative healthcare, agriculturual, industrial, and environmental biotechnology products.
• Center for Advanced Biotechnology and Medicine: Conducts biomedical and biotech research and development to improve human health.
• Center for Biodefense and Emerging Infectious Diseases (CBEID): Addresses and researches important issues in the protection of the US from biological threats, including bioterrorism, biological warfare, and tropical infectious diseases.
• Center for Infectious Disease Research and Policy (CIDRAP): Works to prevent illness and death from infectious disease threats through research by implementing real world practical applications, policies, and solutions.
• Center for International Security and Cooperation: Stanford University’s hub for researchers working together to tackle some of the world’s most pressing security and international cooperation issues; and to influence the policymaking agenda in the United States and abroad to help build a safer world.
• Center for Policy on Emerging Technologies (C-PET): A wide network of knowledgeable professionals who identify, clarify, and prioritize the big questions raised in a future perspective to cultivate a context within which solutions can be developed.
• Center for Strategic and International Studies (CSIS): A bipartisan organization that conducts research and analysis, and develops policy initiatives with a future outlook, to help decision-makers chart a course toward a better world.
• Centers for Disease Control and Prevention (CDC): The United States government’s center for research, analysis, and detection of new and emerging global health threats. Promotes healthy and safe behaviors, communities, and environments.
• Centre for Cellular and Molecular Platforms (C-CAMP): A research and development center in India which enables bioscience research and entrepreneurship by providing research, development, training, and service in biotechnology platforms.
• Centre for the Study of Existential Risk: A multidisciplinary research center dedicated to the study and mitigation of risks that could lead to human extinction.
• Chemical Biological Incident Response Force (CBIRF): Responds to chemical, biological, radiological, nuclear, or high yield explosive events in order to assist local, state, or federal agencies in consequence management.
• Chinese Center for Disease Control and Prevention (China CDC): A nonprofit institution that works in the fields of disease control and prevention, and public health management.
• Defence Advanced Research Projects Agency (DARPA) – Biological Technologies Office: Researches and designs breakthrough technologies for nation security.
• Defence Threat Reduction Agency (DTRA): The U.S. Department of Defense’s official Combat Support Agency for countering weapons of mass destruction.
• EcoHealth Alliance: An international organization of scientists dedicated to the conservation of biodiversity that researches the relationships between wildlife, ecosystems, and human health.
• Emerging Pandemic Threats (EPT) Program: Monitors and increases local capacities of geographic hot spots to identify the emergence of new infectious diseases in high-risk wildlife that could pose a major threat to human health.
• Federation of American Scientists (FAS): Provides a scientific analysis of and solutions to protect against catastrophic threats to national and international security.
• Foresight Institute: A think tank and public interest organization focused on transformative future technologies. Discovers and promotes the technological benefits, and helps avoid the dangers of nanotechnology, AI, biotech, and other similar life-changing developments.
• Global Catastrophic Risk Institute (GCRI): A think tank leading research, education, and professional networking on global catastrophic risk.
• Intelligence: Collects and conveys essential information the President and the members of policymaking, law enforcement, and military communities require to execute their appointed duties.
• International Federation of Biosafety Associatons (IFBA): Supports and promotes biosafety on a national and international level through collaboration among national and regional biosafety organizations worldwide.
• International Geosphere-Biosphere Programme (IGBP): Studies the global-scale and regional-scale interactions between Earth’s biological, chemical, and physical processes and their interactions with human systems.
• Lawrence Livermore National Laboratory (LLNL): Applies multidisciplinary science and technology to anticipate, innovate, and deliver responsive solutions to complex global security needs.
• Lifeboat Foundation: An organization dedicated to encouraging scientific advancements while helping humanity survive existential risks and possible misuse of increasingly powerful technologies.
• Middle East Consortium on Infectious Disease Surveillance (MECIDS): Advances the capabilities for early infectious disease and food-borne illness prevention, detection, control, and response between its member countries of Israel, Jordan, and Palestine.
• National Institute of Allergy and Infectious Diseases (NIH): Conducts and supports basic and applied research to better understand, treat, and prevent infectious, immunologic, and allergic diseases.
• New England Complex Systems Institute (NECSI): An independent academic research and educational institution that develops complex systems and its applications to study how interactions within a system lead to its behavioral patterns, and how the system interacts with its environment.
• Nuclear Threat Initiative (NTI): A nonpartisan organization that focuses on strengthening global security by reducing the risk of using and preventing the spread of nuclear, biological, and chemical weapons.
• Resilience Alliance: A research organization comprised of scientists and practitioners from many disciplines who collaborate to explore the dynamics of social-ecological systems.
• Skoll Global Threats Fund: Seeks solutions, strengthens alliances, and spurs actions needed to confront the global threats imperiling humanity.
• Society for Risk Analysis (SRA): A multidisciplinary, interdisciplinary, scholarly, international society that provides an open forum for all those who are interested in risk analysis.
• The Biosecurity Engagement Program (BEP): Engages with Life Scientists to combat biological threats worldwide by providing assistance to improve biosecurity, biosafety, pathogen surveillance, and infectious disease surveillance and response.
• The Center for Bioethics and Human Dignity (CBHD)
• The Club of Rome: An informal association of long-term thinkers interested in and concerned with the future of humanity and the planet.
• The Genome Analysis Centre (TGAC): A research institute focused on the application of genomics and bioinformatics to advance plant, animal, and microbial research to promote a sustainable bioeconomy.
• The Institute for Ethics and Emerging Technologies (IEET): A think tank which promotes ideas about how technological progress can increase freedom, happiness, and human flourishing in democratic societies.
• The Program on Science and Global Security (SGS): Carries out research, policy analysis, education, and training in nuclear arms control and nonproliferation, and provides research and administrative support to the International Panel on Fissile Materials.
• U.S. Department of Defense (DOD): Provides military forces needed to deter war and to protect the security of the United States.
• U.S. Department of Homeland Security (DHS) – National Biodefense Analysis and Countermeasures Center (NBACC): A national resource used to understand the scientific basis of the risks posed by biological threats and to attribute their use in bioterrorism or biocrime events.
• UPMC Center for Health Security: Protects people’s health from the consequences of epidemics and disasters, and ensures that communities are resilient to major challenges.
• Wilson Center: A non-partisan policy forum and a leading institution for tackling global issues through independent research and open dialogue to inform actionable ideas for Congress, the Administration, and the broader policy community.
• X-events: A research institute that works with the theory of surpize, studies extreme events, experiments different anticipation methods, and develops tools for research and decision making.
• Prevent Project: PREVENT works to reduce the risks that zoonoses—diseases that move between animals and humans—can pose to public health.

These organizations above all work on biotechnology issues, though many cover other topics as well. This list is undoubtedly incomplete; please contact us to suggest additions or corrections.

Media

• Systemic Risk and Stability in Financial Networks
• The Future Power Grid: Resilience and Systemic Risk

Research Papers

• Systemic Risk, Contagion, and Financial Networks: A Survey
• Bruce Schneier – Resources on Existential Risk, p. 133

Books

• New Directions for Understanding Systemic Risk

Organizations

• Society for Risk Analysis: An interdisciplinary, scholarly, international society that provides an open forum for all those who are interested in risk analysis.
• Singularity University: Provides educational programs, innovative partnerships and a startup accelerator to help individuals, organizations and governments understand cutting-edge technologies, and how to utilize these technologies for positive impact.
• Saving Humanity from Homo Sapiens: A philanthropic organization that supports groups working on reducing existential risk for humanity.