Recently, Sen. Wiener sent an open letter to Andreessen Horowitz and Y Combinator dispelling misinformation that has been spread about SB1047, including that it would send model developers to jail for failing to anticipate misuse and that it would stifle innovation. The letter points out that the “bill protects and encourages innovation by reducing the risk of critical harms to society that would also place in jeopardy public trust in emerging technology.” Read Sen. Wiener’s letter in full here. 

Anthony Aguirre, Executive Director of the Future of Life Institute:

“This poll is yet another example of what we’ve long known: the vast majority of the public support commonsense regulations to ensure safe AI development and strong accountability measures for the corporations and billionaires developing this technology. It is abundantly clear that there is a massive, ongoing disinformation effort to undermine public support and block this critical legislation being led by individuals and companies with a strong financial interest in ensuring there is no regulation of AI technology. However, today’s data confirms, once again, how little impact their efforts to discredit extremely popular measures have been, and how united voters–including tech workers–and policymakers are in supporting SB1047 and in fighting to ensure AI technology is developed to benefit humanity.”

Reports of pressure to fast-track safety testing and attempts to muzzle employees from publicly voicing concerns reveal an alarming lack of accountability and transparency. This puts us all at risk. 

As AI companies frantically compete to create increasingly powerful and potentially dangerous systems without meaningful governance or oversight, it has never been more important that courageous employees bring bad behavior and safety issues to light. Our continued wellbeing and national security depend on it. 

We need to strengthen current whistleblower protections. Today, many of these protections only apply when a law is being broken. Given that AI is largely unregulated, employees and ex-employees cannot safely speak out when they witness dangerous and irresponsible practices. We urgently need stronger laws to ensure transparency, like California’s proposed SB1047 which looks to deliver safe and secure innovation for frontier AI. 

The Future of Life Institute commends the brave individuals who are striving to bring all-important incidents and transgressions to the attention of governments and the general public. Lawmakers should act immediately to pass legal measures that provide the protection these individuals deserve.

Anthony Aguirre, Executive Director of the Future of Life Institute

“As Stuart and Yoshua have both said in the past, the capabilities of AI systems have outpaced even the most aggressive estimates by most experts. We are grossly unprepared, and must not wait to act. We implore Congress to immediately regulate these systems before they cause irreparable damage.

Effective oversight would include:

1. The legal and technical mechanisms for the federal government to implement an immediate pause on development of AI more powerful than GPT-4
2. Requiring registration for large groups of computational resources, which will allow regulators to monitor the development of powerful AI systems
3. Establishing a rigorous process for auditing risks and biases of these systems
4. Requiring approval and licenses for the deployment of powerful AI systems, which would be contingent upon developers proving their systems are safe, secure, and ethical
5. Clear red lines about what risks are intolerable under any circumstances

“Funding for technical AI safety research is also crucial. This will allow us to ensure the safety of our current AI systems, and increase our capacity to control, secure, and align any future systems.

“The world’s leading experts agree that we should pause development of more powerful AI systems to ensure AI is safe and beneficial to humanity, as demonstrated in the March letter coordinated by the Future of Life Institute. The federal government should have the capability to implement such a pause. The public also agrees that we need to put regulations in place: nearly three-quarters of Americans believe that AI should be either somewhat or heavily regulated by the government, and the public favors a pause by a 5:1 margin. These regulations must be urgently and thoroughly implemented – before it is too late.”

Dr Anthony Aguirre, Executive Director, Future of Life Institute

Last year, FLI received scores of grant applications from across the globe for the millions of dollars in funding we distribute to support research, outreach and other important work in furtherance of FLI’s mission. One of these grant proposals came from the Sweden-based Nya Dagbladet Foundation (NDF) for a media project considered directly related to FLI’s goals.  

Although we were initially positive about the proposal and its prospects – and wrote a letter of intent to this effect – our due diligence process later uncovered information indicating that NDF was not aligned with FLI’s values or charitable purposes, and in November 2022 FLI informed the Foundation that we would not be moving forward with a grant agreement or grant.

FLI has given the Nya Dagbladet Foundation zero funding, and will not fund them in the future. These final decisions were made by FLI’s leadership independently of any outside influence and well before any inquiry regarding the NDF proposal by members of the media.    On December 15, well after we had informed NDF that their proposal was rejected, a Swedish website contacted FLI with questions regarding the NDF proposal, and describing Nya Dagbladet as a far-right extremist group. We responded the same day that FLI had decided not to fund the project now or at any later time. On January 13,  they ran an article regarding FLI in this connection, and many important questions were raised there and across social media that we address below. We would like to emphatically state that FLI finds groups or ideologies espousing antisemitism, white supremacy, or racism despicable and would never knowingly support any such group. In terms of the Nya Dagbladet Foundation, which is a nonprofit foundation under Swedish law, further investigation has only further validated our November decision to reject their proposal.  Our due diligence worked, but not as early as it should have.  We deeply regret that we may have inadvertently compromised the confidence of our community and constituents. We are reviewing how we can amend and improve our procedures.

Below are our answers to the central questions that have been raised about what happened.

Frequently Asked Questions

1) Does FLI support far-right groups or ideologies?

No. FLI finds groups or ideologies espousing antisemitism, white supremacy, or racism despicable and would never knowingly support any such group. FLI stands and will always stand emphatically against racism, bigotry, bias, injustice and discrimination at all times and in all forms.  These evils are antithetical to our mission to safeguard the future of life and to advance human flourishing.

2) Why were you initially positive about this grant proposal?

A core part of our mission is to reduce risks associated with AI, nuclear war, and related challenges by educating the public about them. FLI therefore has funded numerous media projects over the years, including for example Minutephysics videos about superintelligence and nuclear war, MinuteEarth videos about ozone destruction and nuclear winter, Slaughterbots, Slaughterbots II, and autonomous weapons and nuclear war websites. Our Institute is also open to grant proposals from media organizations interested in writing about such risks — media from across the political spectrum and globally, to help educate as broad and diverse readership as possible. The NDF shares leadership with a small online newspaper which, based on our initial research and screening, we classified politically as right-of-center with an establishment-critical slant. For example, it appears to be one of few Swedish newspapers vocally critical of stationing nuclear weapons in Sweden.  It is clear that our initial vetting was insufficient, and we’re grateful that our subsequent due diligence worked as intended and prevented a grant from being made.

3) Why didn’t you immediately realize how extreme Nya Dagbladet was?

As a baseline, at the time of the initial consideration of the Foundation’s proposal, FLI was simply not aware of the extreme nature of the associated newspaper or its history.  There are a number of reasons for this, including:

(a) The newspaper downplays such extremism and this fact and its historical relationships were not evident to us from their website or proposal.

(b) The quality of public discourse worldwide has degraded so badly, with casual name-calling using highly charged labels, that many of these types of accusations are open to question upon examination.  We confirmed however in our own due diligence process that sometimes these casual labels can be accurate.

4) What was the meaning of FLI’s letter of intent?

Those who have received FLI grants know that the final step in approving a grant requires that both parties signing a formal multi-page grant contract with significant legal terms and condition that regulate the nature and timing of the grant and the obligations of the grantee. The proposal was rejected before reaching that stage. Our initial expression of intent was just that, based on our initial consideration and was not binding or intended to be binding.  It is not unusual for grantmaking bodies to issue such statements of intention at the request of potential grantees and that is all that happened here. More specifically, FLI gives lots of grants and our process involves 7 steps:

  1) Receive grant proposal

  2) Evaluation and vetting

  3) Decide that we intend to issue a grant

  4) Inform the grantee of our intention to issue a grant

  5) Further due diligence on grantee

  6) Offer the grant – specifically, send a legally binding grant agreement that they can choose to accept and sign. This is a multi-page document with standard legal stipulations for what each side commits to do and not do.

  7) Pay the grant

This proposal made it through 4) in August, then was rejected in November during 5), never reaching 6) or 7). In short: our vetting was insufficient but our due diligence process succeeded. Note that our grant rejection occurred well before media contacted us on December 15. In many cases, 4) is done via an informal email, but in some cases (including this one), the grantee requested a letter of intent.

5) Was nepotism involved? In particular, would FLI’s president’s brother have profited in any way had the grant been awarded?

No. He published some articles in the newspaper, but the understanding from the very beginning was that this was pro-bono, and he was never paid and never planned to get paid by the newspaper of the foundation. The grant proposal requested no funds for him. He is a journalist with many years of experience working for Swedish public radio and television, and runs his own free and non-commercial podcast. The newspaper linked some of his episodes, but this has nothing to do with FLI, and it provided no ad revenue since he runs no ads. He was shocked by the recent revelations of extremism and plans no further association with the newspaper.

6) What made FLI decide not to fund the grant after all?

As indicated above, last Fall FLI performed further due diligence in connection with the grant proposal and discovered facts that blindsided us and mandated that we reject the proposal. For example, we found the term “ethnopluralism” endorsed in Nya Dagbladet.  This is blatantly inconsistent with FLI’s core values, and we therefore informed the NDF in November that we had decided not to make any grant. More recently, we have learned of even more disturbing material discovered by others about the newspaper and its leadership.

7) Why didn’t FLI respond with more information earlier?

FLI’s core leadership has been unusually unavailable during the past month: FLI’s president has been dealing with the aftermath of his mother’s death on December 15 (the day media reached out), FLI’s treasurer has been on maternity leave, and FLI’s secretary was on a 2-week overseas trip. We responded on December 15 that we had not paid any money to the Foundation and didn’t plan to, and we hope that this FAQ helps provide a more complete understanding of what transpired.  

8) What have we learned from this and how can we improve our grantmaking process?

The way we see it, we rejected a grant proposal that deserved to be rejected, and challenging, reasonable questions have been asked as to why we initially considered it and didn’t reject it earlier. We deeply regret that we may have inadvertently compromised the confidence of our community and constituents. This causes us huge distress, as does the idea that FLI or its personnel would somehow align with ideologies to which we are fundamentally opposed. We are working hard to continue improving the structure and process of our grantmaking processes, including more internal and (in appropriate cases) external review.  For starters, for organizations not already well-known to FLI or clearly unexceptionable (e.g. major universities), we will request and evaluate more information about the organization, its personnel, and its history before moving on to additional stages.

Update January 19, 2023:

This was just covered by vice.com. Note that their article is inconsistent about whether we issued a grant agreement or not, first saying “the grant agreement was immediately revoked” and then saying “would not be moving forward with a grant agreement”. As clarified in (4) above, our process made it to the intent stage but never proceeded to the stage of issuing a grant agreement, which is what we mean by “offering a grant”.

Most people seem to understand that malaria is a pressing problem, one that continues to menace a number of areas around the world. However, most would likely be shocked to learn the true scale of the tragedy. To illustrate this point, in 2017, more than 200 million people were diagnosed with malaria. By the year’s close, nearly half a million people had died of the disease. And these are the statistics for just one year. Over the course of the 20th century, researchers estimate that malaria claimed somewhere between 150 million and 300 million lives. With even the lowest figure, the death toll is still more than World War I, World War II, the Vietnam War, and the Korean War combined. 

Although its pace has slowed in recent years, according to the World Health Organization, malaria remains one of the leading causes of death in children under five. However, there is new hope, and it comes in the form of CRISPR gene drives. 

With this technology, many researchers believe humanity could finally eradicate malaria, saving millions of lives and, according to the World Health Organization, trillions of dollars in associated healthcare costs. The challenge isn’t so much a technical one. If scientists needed to deploy CRISPR gene drives in the near future, Ethan Bier, a Distinguished Professor of Cell and Developmental Biology at UC San Diego, notes that they could reliably do so. 

However, there’s a hitch. In order to eradicate malaria, we would need to use anti-malaria gene drives to target three specific species (maybe more) and force them into extinction. This would be one of the most audacious attempts by humans to engineer the planet’s ecosystem, a realm where humans already have a checkered past. Such a use sounds highly controversial, and of course, it is. 

However, regardless of whether the technology is being deployed to try and save a species or to try and force it into extinction, a number of scientists are wary. Gene drives will permanently alter an entire population. In many cases, there is no going back. If scientists fail to anticipate properly all of the effects and consequences, the impact on a particular ecological habitat — and the world at large — could be dramatic.

Rewriting Organisms: Understanding CRISPR

CRISPR/Cas9 editing technology enables scientists to alter DNA with enormous precision.

This degree of genetic targeting is only possible because of the unification of two distinct gene editing technologies: CRISPR/Cas9 and gene drives. On their own, each of these tools is powerful enough to alter a gene pool dramatically. Together, they can erase that pool entirely.

The first part of this equation is CRISPR/Cas9. More commonly known as “CRISPR,” this technology is most easily understood as a pair of molecular scissors. CRISPR, which stands for “Clustered Regularly Interspaced Palindromic Repeats,” was adapted from a naturally occurring defense system that’s found in bacteria. 

When a virus invades a host bacteria and is defeated, bacteria are able to capture the virus’ genetic material and merge snippets of it into their genomes. The virus’ genetic material is then used to make guide RNA. These guide RNA target and bind to complementary genetic sequences. When a new virus invades, the guide RNA will find the complementary sequences on the attacking virus and attach itself to that matching portion of the genome. From there, an enzyme known as Cas9 cuts it apart, and the virus is destroyed. 

Lab-made CRISPR allows humans to accomplish much the same  — cut any region of the genome with relatively high precision and accuracy, often disabling any cut sequence in the process. However, scientists have the ability to go a step farther than nature. After a cut is made using CRISPR, scientists can use the cell’s own repair machinery to add or replace an existing segment of DNA with a customized DNA sequence i.e. a customized gene. 

If genetic changes are made in somatic cells (the non-reproductive cells of a living organism) — a process known as “somatic gene editing” — it only affects the organism in which the genetic changes were made. However, if the genetic changes are made in the germline (the sequence of cells that develop into eggs and sperm) — a process known as “germline editing” — then the edited gene can spread to the organism’s offspring. This means that the synthetic changes — for good or bad — could permanently enter the gene pool. 

But by coupling CRISPR with gene drives, scientists can do more than spread a gene to the next generation — they can force it through an entire species. 

Rewriting Species: Understanding Gene Drives

Most species on our planet have two copies of their genes. During the reproductive cycle, one of these genes is selected to be passed on to the next generation. Because this selection process occurs randomly in nature, there’s about a 50/50 chance that any given gene will be passed down. 

Gene drives change those odds by increasing the probability that a specific gene (or suite of genes) will be inherited. Surprisingly, scientists have known about gene drive systems since the late 1800s. They occur naturally in the wild thanks to something known as “selfish genetic elements” (“selfish genes).” Unlike most genes, which wait patiently for nature to randomly select them for propagation, selfish genes use a variety of mechanisms to manipulate the reproductive process and ensure that they get passed down to more than 50% of offspring. 

One way that this can be achieved is through segregation distorters, which alter the replication process so that a gene sequence is replicated more frequently than others. Transposable elements, on the other hand, allow genes to move to additional locations in the genome. In both instances, the selfish genes use different mechanisms to increase their presence on the genome and, in so doing, improve their odds of being passed down. 

In the 1960s, scientists first realized that humanity might be able to use a gene drive to dictate the genetic future of a species. Specifically, biologists George Craig, William Hickey, and Robert Vandehey argued that a mass release of male mosquitoes with a gene drive that increased the number of male offspring could reduce the number of females, the sex that transmits malaria, “below the level required for efficient disease transmission.” In other words, the team argued that malaria could be eradicated by using a male-favoring gene drive to push female mosquitoes out of the population. 

However, gene editing technologies hadn’t yet been invented. Consequently, gathering a mass of mosquitoes with this specific gene drive was impossible, as scientists were forced to rely on time-consuming and imprecise breeding techniques. 

Then, in the 1970s, Paul Berg and his colleagues created the first molecule made of DNA from two different species, and laboratory-based genetic engineering was born. And not too long after that, in 1992, Margaret Kidwell and José Ribeiro proposed attaching a specific gene to selfish genes to drive the gene through a mosquito population and make it malaria-resistant. 

But despite the theoretical ingenuity of these designs, when it came to deploying them in reality, progress was elusive. Gene editing tools were still quite crude. As a result, they caused a number of off-target edits, where portions of the genome were cut unintentionally and segments of DNA were added in the wrong place. Then CRISPR came along in 2012 and changed everything, making gene editing comparably fast, reliable, and precise. 

It didn’t take scientists long to realize that this new technology could be used to create a remarkably powerful, remarkably precise human-made selfish gene. In 2014, Kevin M Esvelt, Andrea L Smidler, Flaminia Catteruccia, and George M. Church published their landmark paper outlining this process. In their work, they noted that by coupling gene drive systems with CRISPR, researchers could target specific segments of a genome, insert a gene of their choice, and ultimately ensure that the gene would make its way into almost 100% of the offspring. 

Thus, CRISPR gene drives were born, and it is at this juncture that humanity may have acquired the ability to rewrite — and even erase — entire species.

The Making of a Malaria-Free World

To eradicate malaria, at least three species of mosquitos would have to be forced into extinction.

Things move fast in the world of genetic engineering. Esvelt and his team only outlined the process through which scientists could create CRISPR gene drives in 2014, and researchers have had working prototypes for nearly as long. 

In December of 2015, scientists published a paper announcing the creation of a CRISPR gene drive that made Anopheles stephensi, one of the primary mosquito species responsible for the spread of malaria, resistant to the disease. Notably, the gene drive was just as effective as earlier pioneers had predicted: Although the team initially started with just two genetically edited males, after only two generations of cross-breeding, they had over 3,800 third generation mosquitoes, 99.5% of which expressed genes indicating that they had acquired the genes for malaria resistance. 

However, in wild populations, it’s likely that the malaria parasite would eventually develop resistance to the gene drive. Thus, other teams have focused their efforts not on malaria resistance, but making mosquitoes extinct. As a side note, it must be stressed that no one was (or is) suggesting that we should exterminate all mosquitoes to end malaria. While there are over 3000 mosquito species, only 30 to 40 mosquito species transmit the disease, and scientists think that we could eradicate malaria by targeting just three of them. 

In September of 2018, humanity took a major step towards realizing this vision, when scientists at the Imperial College London published a paper announcing that one of their CRISPR gene drives had wiped out an entire population of lab-bred mosquitoes in less than 11 generations. If this approach were released into the wild, the team predicted that it could propel the affected species into extinction in just one year. 

For their work, the team focused on Anopheles gambiae and targeted genes that code proteins that play an important role in determining an organism’s sexual characteristics. By altering the gene, the team was able to create female mosquitoes that were infertile. What’s more, the drive seems to be resistance-proof. 

There is still some technical work to be done before this particular CRISPR gene drive can be deployed in the wild. For starters, the team needs to verify that it is, in fact, resistant-proof. The results also have to be replicated in the same conditions in which Anopheles mosquitoes are typically found — conditions that mimic tropical locations across sub-Saharan Africa. Yet the researchers are making rapid progress, and a CRISPR gene drive that can end malaria may soon be a reality. 

Moving Beyond Malaria

Aside from eradicating malaria, one of the most promising applications of CRISPR gene drives involves combating invasive species. 

For example, in Australia, the cane toad has been causing an ecological crisis since the 1930s. Originating in the Americas, the cane toad was introduced to Australia in 1935 by the Bureau of Sugar Experiment Stations in an attempt to control beetle populations that were attacking sugar cane crops. However, the cane toad has no natural predators in the area, and so has multiplied at an alarming rate. 

Since its 1935 introduction to Australia, the poisonous cane toad has decimated populations of native species that attempt to prey on it. A gene drive could be used to eliminate its toxicity.

Although only 3,000 were initially released, scientists estimate that the cane toad population in Australia is currently over 200 million. For decades, the toads have been killing a number of native birds, snakes, and frogs that prey on it and inadvertently ingest its lethal toxin — the population of one monitor lizard species dropped by 90% after the cane toad spread to its area. 

However, by genetically modifying the cane toads to keep them from producing these toxins, scientists believe they might be able to give native species a fighting chance. And because the CRISPR gene drive would only target the cane toad population, it may actually be safer than traditional pest control methods that involve poisons, as these chemicals impact a multitude of species. 

Research indicates that CRISPR gene drives could also be used to target a host of other invasive pests. In January of 2019, scientists published a paper demonstrating the first concrete proof that the technology also works in mammals — specifically, lab mice. 

Another use case involves deploying CRISPR gene drives to alter threatened or endangered organisms in order to better equip them for survival. For instance, a number of amphibian species are in decline because of the chytrid fungus, which causes a skin disease that is often lethal. Esvelt and his team note that CRISPR gene drives could be used to make organisms resistant to this fungal infection. Currently, there is no resistance mechanism for the fungus, so this specific use case is just a theoretical application. However, if developed, it could be deployed to save many species from extinction. 

The Harvard Wyss Institute suggests that CRISPR gene drives could also be used “to pave the way toward sustainable agriculture.” Specifically, the technology could be used to reverse pesticide resistance in insects that attack crops, or it could be used to reverse herbicide resistance in weeds.     

Yet, CRISPR gene drives are powerless when it comes to some of humanity’s greatest adversaries. 

Because they are spread through sexual reproduction, gene drives can’t be used to alter species that reproduce asexually, meaning they can’t be used in bacteria and viruses. Gene drives also don’t have a practical applications in humans or other organisms with long generational periods, as it would take several centuries for any impact to be noticeable.. The Harvard Wyss institute notes that, at these timescales, someone could easily create a reversal drive to remove the trait, and any number of other unanticipated events could prevent the drive from propagating. 

That’s not to say that reverse gene drives should be considered a safety net in case forward gene drives are weaponized or found to be dangerous. Rather, the primary point is to highlight the difficulty in using CRISPR gene drives to spread a gene throughout species with long generational and gestational periods. 

But, as noted above, when it comes to species that have short reproductive cycles, the impact could be profound on extremely short order. With this in mind — alt­hough the work in amphibian, mammal, and plant populations is promising — the general scientific consensus is that the best applications for CRISPR gene drives likely involve insects. 

Entering the Unknown

Before scientists introduce or remove a species from a habitat, they conduct research in order to understand the role that it plays within the ecosystem. This helps them better determine what the overall outcome will be, and how other individual organisms will be impacted. 

According to Matan Shelomi, an entomologist who specializes in insect microbiology, scientists haven’t found any organisms that will suffer if three mosquitoes species are driven into extinction. Shelomi notes that, although several varieties of fish, amphibians, and insects eat mosquito larvae, they don’t rely on the larvae to survive; in fact, most of the organisms that have been studied prefer other food sources, and no known species live on mosquitoes alone. The same, he argues, can be said of adult mosquitoes. While a number of birds do consume mature mosquitoes, none rely on them as a primary food source. 

Shelomi also notes that mosquitoes don’t play a critical role in pollination — or any other facet of the environment that scientists have examined. As a result, he says they are not a keystone species: “No ecosystem depends on any mosquito to the point that it would collapse if they were to disappear.” 

At least, not as far as we are aware. 

Because CRISPR gene drives cause permanent changes to a species, virologist Jonathan Latham notes that it is critical to get things right the first time: “They are ‘products’ that will likely not be able to be recalled, so any approval decision point must be presumed to be final and irreversible.” However, we have no way of knowing if scientists have properly anticipated every eventuality. “We certainly do not know all the myriad ways all mosquitoes interact with all life forms in their environment, and there may be something we are overlooking,” Shelomi admits. Due to these unknown unknowns, and the near irreversibility of CRISPR gene drives, Latham argues that they should never be deployed. 

Every intervention has consequences. To this end, the important thing is to be as sure as possible that the potential rewards outweigh the risks. For now, when it comes to anti-malaria CRISPR gene drives, this critical question remains unanswered. Yet the applications for CRISPR gene drives extend far beyond mosquitoes, making it all the more important for scientists to ensure that their research is robust and doesn’t cause harm to humanity or to Earth’s ecosystems. 

Risky Business, Designed for Safety

Although some development is still needed before scientists would be ready to release a CRISPR gene drive into a wild insect population, the most pressing issues that remain are of a regulatory and ethical nature. These include: 

Limiting Deployment and Obtaining Consent 

For starters, who gets to decide whether or not scientists should eradicate a species? The answer most commonly given by scientists and politicians is that the individuals who will be impacted should cast the final vote. However, substantial problems arise when it comes to limiting deployment and determining the degree to which informed consent is necessary. 

Todd Kuiken, a Senior Research Scholar with the Genetic Engineering and Society Center at NC State University and a member of the U.N.’s technical expert group for the Convention on Biological Diversity, notes that, “in the past, these kinds of applications or introductions were mostly controlled in terms of where they were supposed to go.” Gene drives are different, he argues, because they are “designed to move, and that’s really a new concept in terms of environmental policy. That’s what makes them really interesting from a policy perspective.” 

The highly globalized nature of the modern world would make it near-impossible to geographically contain the effects of a gene drive.

For example, if scientists release mosquitoes in a town in India that has approved the work, there is no practical way to contain the release to this single location. The mosquitoes will travel to other towns, other countries, and potentially even other continents.

The problem isn’t much easier to address even if the release is planned on a remote island. The nature of modern life, which sees people and goods continually traveling across the globe, makes it extremely difficult to prevent cross contamination.  A CRISPR gene drive deployed against an invasive species on an island could still decimate populations in other places — even places where it is native and beneficial. 

The release of a single engineered gene drive could, potentially, impact every human on Earth. Thus, in order to obtain the informed consent from all affected parties, scientists would effectively need to ensure that they had permission from everyone on the planet. 

To help address these issues of “free, prior, and informed consent,” Kuiken notes that scientists and policymakers must establish a consensus on the following: 

1.   What communities, organizations, and groups should be part of the decision-making process? 
2.   How far out do you go to obtain informed consent — how many centric circles past the introduction point do you need to move? 
3.   At which decision stage of the project should these different groups or potentially impacted communities be involved? 

Of course, in order for individuals to effectively participate in discussions about CRISPR gene drive, they will have to know what it is. This also poses a problem: “Generally speaking, the majority of the public probably hasn’t even heard of it,” Kuiken says. 

There are also questions about how to verify that an individual is actually informed enough to give consent. “What does it really mean to get approval?” Kuiken asks, noting, “the real question we need to start asking ourselves is ‘what do we mean by [informed consent]?’” 

Because research into this area is already so advanced, Kuiken notes that there’s an immediate need for a broad range of endeavors aimed at improving individuals’ knowledge of, and interest in, CRISPR gene drives. 

Expert Inclusion 

And it’s not just the public that need schooling. When it comes to scientists’ understanding of the technology, there are also serious and significant gaps. The degree and depth of these gaps, Kuiken is quick to point out, varies dramatically from field to field. While most geneticists are at least vaguely familiar with CRISPR gene drives,some key disciplines are still in the dark: one of the key findings of this year’s upcoming IUCN (International Union for Conservation of Nature) report is that “the conservation science community is not well aware of gene drives at all.” 

“What concerns me is that a lot of the gene drive developers are not ecologists. My understanding is that they have very little training, or even zero training, when it comes to environmental interactions, environmental science, and ecology,” Kuiken says. “So, you have people developing systems that are being deliberately designed to be introduced to an environment or an ecosystem who don’t have the background to understand what all those ecological interactions might be.” 

To this end, scientists working on gene drive technologies must be brought into conversations with ecologists. 

Assessing the Impact 

But even if scientists work together under the best conditions, the teams will still face monumental difficulties when trying to assess the impact and significance of a particular gene drive. 

To begin with, there are issues with funding allocation. “The research dollars are not balanced correctly in terms of the development of the gene drive verses what the environmental implications will be,”  Kuikon says. While he notes that this is typically how research funds are structured — environmental concerns come in last, if they come at all — CRISPR gene drives are fundamentally about the environment and ecology. As such, the funding issues in this specific use case are troubling. 

Yet, if proper funding were secured, it would still be difficult to guarantee that a drive was safe. Even a small gap in our

Alterations made to a single species can have a detrimental impact on an entire ecosystem.

understanding of a habitat could result in a drive being released into a species that has a critical ecological function in its local environment. As with the cane toad in Australia, this type of release could cause an environmental catastrophe and irreversibly damage an ecosystem. 

One way to  help prevent adverse ecological impacts is to gauge the effect through daisy-chain gene drives. These are self-limiting drives that grow weaker and weaker and die off after a few generations, allowing researchers to effectively measure the overall impact while restricting the gene drive’s spread. If such tests determine that there are no unfavorable effects, a more lasting drive can subsequently be released. 

Kill-switches offer another potential solution. The Defense Advanced Research Projects Agency (DARPA) recently announced that it was allocating money to fund research into anti-CRISPR proteins. These could be used to prevent the expression of certain genes and thus counter the impact of a gene drive that has gone rogue or was released maliciously. 

Similarly, scientists from North Carolina State University’s Genetic Engineering and Society Center note that it may be beneficial to establish a regulatory framework requiring the development of immunizing drives, which spread resistance to a specific gene drive, to be developed alongside drives that are intended for release. These could be used to immunize related species that aren’t being targeted, or kept on the ready in case of any unanticipated occurrences. 

An Uncertain Future

But even if scientists do everything right, and even if researchers are able to verify that CRISPR gene drives are 100% safe, it doesn’t mean they will be deployed. “You can move yourself far in terms of generally scientific literacy around gene drives, but people’s acceptance changes when it potentially has a direct impact on them,” Kuiken explains. 

To support his claims, Kuiken points to the Oxitec mosquitoes in Florida. 

Here, teams were hoping to release male Aedes aegypti mosquitoes carrying a “self-limiting” gene. These are akin to, but distinct from, gene drives. When these edited males mate with wild females, the offspring inherit a copy of a gene that prevents them from surviving to adulthood. Since they don’t survive, they can’t reproduce, and there is a reduction in the wild population. 

After working with local communities, Oxitec put the release up for a vote. “The vote count showed that, generally speaking, it you tallied up the whole area of South Florida, it was about a 60 to 70 percent approval. People said, ‘yeah, this is a really good idea. We should do this,’” Kuiken said. “But when you focused in on the areas where they were actually going to release the mosquitoes, it was basically flipped. It was a classic ‘not in my backyard’ scenario.”

That fear, especially when it comes to CRISPR gene drives, isn’t really too hard to comprehend. Even if every scientific analysis showed that the benefits of these drives outweighed all the various drawbacks, there would still be the unknown unknowns. 

Researchers can’t possibly account for how every single species — all the countless plants, insects, and as yet undiscovered deep sea creatures — will be impacted by a change we make to an organism. So unless we develop unique and unprecedented scientific protocols, no matter how much research we do, the decision to use or not use CRISPR gene drives will have to be made without all the evidence. 

This blog post reports on Schubert, S.**, Caviola, L.**, Faber, N. The Psychology of Existential Risk: Moral Judgments about Human Extinction. Scientific Reports [Open Access]. It was originally posted on the University of Oxford’s Practical Ethics: Ethics in the News blog.

Humanity’s ever-increasing technological powers can, if handled well, greatly improve life on Earth. But if they’re not handled well, they may instead cause our ultimate demise: human extinction. Recent years have seen an increased focus on the threat that emerging technologies such as advanced artificial intelligence could pose to humanity’s continued survival (see, e.g., Bostrom, 2014; Ord, forthcoming). A common view among these researchers is that human extinction would be much worse, morally speaking, than almost-as-severe catastrophes from which we could recover. Since humanity’s future could be very long and very good, it’s an imperative that we survive, on this view.

Do laypeople share the intuition that human extinction is much worse than near-extinction? In a famous passage in Reasons and Persons, Derek Parfit predicted that they would not. Parfit invited the reader to consider three outcomes:

1) Peace
2) A nuclear war that kills 99% of the world’s existing population.
3) A nuclear war that kills 100%.

In Parfit’s view, 3) is the worst outcome, and 1) is the best outcome. The interesting part concerns the relative differences, in terms of badness, between the three outcomes. Parfit thought that the difference between 2) and 3) is greater than the difference between 1) and 2), because of the unique badness of extinction. But he also predicted that most people would disagree with him, and instead find the difference between 1) and 2) greater.

Parfit’s hypothesis is often cited and discussed, but it hasn’t previously been tested. My colleagues Lucius Caviola and Nadira Faber and I recently undertook such testing. A preliminary study showed that most people judge human extinction to be very bad, and think that governments should invest resources to prevent it. We then turned to Parfit’s question whether they find it uniquely bad even compared to near-extinction catastrophes. We used a slightly amended version of Parfit’s thought-experiment, to remove potential confounders:

A) There is no catastrophe.
B) There is a catastrophe that immediately kills 80% of the world’s population.
C) There is a catastrophe that immediately kills 100% of the world’s population.

A large majority found the difference, in terms of badness, between A) and B) to be greater than the difference between B) and C). Thus, Parfit’s hypothesis was confirmed.

However, we also found that this judgment wasn’t particularly stable. Some participants were told, after having read about the three outcomes, that they should remember to consider their respective long-term consequences. They were reminded that it is possible to recover from a catastrophe killing 80%, but not from a catastrophe killing everyone. This mere reminder made a significantly larger number of participants find the difference between B) and C) the greater one. And still greater numbers (a clear majority) found the difference between B) and C) the greater one when the descriptions specified that the future would be extraordinarily long and good if humanity survived.

Our interpretation is that when confronted with Parfit’s question, people by default focus on the immediate harm associated with the three outcomes. Since the difference between A) and B) is greater than the difference between B) and C) in terms of immediate harm, they judge that the former difference is greater in terms of badness as well. But even relatively minor tweaks can make more people focus on the long-term consequences of the outcomes, instead of the immediate harm. And those long-term consequences become the key consideration for most people, under the hypothesis that the future will be extraordinarily long and good.

A conclusion from our studies is thus that laypeople’s views on the badness of extinction may be relatively unstable. Though such effects of relatively minor tweaks and re-framings are ubiquitous in psychology, they may be especially large when it comes to questions about human extinction and the long-term future. That may partly be because of the intrinsic difficulty of those questions, and partly because most people haven’t thought a lot about them previously.

In spite of the increased focus on existential risk and the long-term future, there has been relatively little research on how people think about those questions. There are several reasons why such research could be valuable. For instance, it might allow us to get a better sense of how much people will want to invest in safe-guarding our long-term future. It might also inform us of potential biases to correct for.

The specific issues which deserve more attention include people’s empirical estimates of whether humanity will survive and what will happen if we do, as well as their moral judgments about how valuable different possible futures (e.g., involving different population sizes and levels of well-being) would be. Another important issue is whether we think about the long term future with another frame of mind because of the great “psychological distance” (cf. Trope and Lieberman, 2010). We expect the psychology of longtermism and existential risk to be a growing field in the coming years.

** Equal contribution.

As data-driven learning systems continue to advance, it would be easy enough to define “success” according to technical improvements, such as increasing the amount of data algorithms can synthesize and, thereby, improving the efficacy of their pattern identifications. However, for ML systems to truly be successful, they need to understand human values. More to the point, they need to be able to weigh our competing desires and demands, understand what outcomes we value most, and act accordingly. 

Understanding Values

In order to highlight the kinds of ethical decisions that our ML systems are already contending with, Kaj Sotala, a researcher in Finland working for the Foundational Research Institute, turns to traffic analysis and self-driving cars. Should a toll road be used in order to shave five minutes off the commute, or would it be better to take the longer route in order to save money? 

Answering that question is not as easy as it may seem. 

For example, Person A may prefer to take a toll road that costs five dollars if it will save five minutes, but they may not want to take the toll road if it costs them ten dollars. Person B, on the other hand, might always prefer taking the shortest route regardless of price, as they value their time above all else. 

In this situation, Sotala notes that we are ultimately asking the ML system to determine what humans value more: Time or money. Consequently, what seems like a simple question about what road to take quickly becomes a complex analysis of competing values. “Someone might think, ‘Well, driving directions are just about efficiency. I’ll let the AI system tell me the best way of doing it.’ But another person might feel that there is some value in having a different approach,” he said. 

While it’s true that ML systems have to weigh our values and make tradeoffs in all of their decisions, Sotala notes that this isn’t a problem at the present juncture. The tasks that the systems are dealing with are simple enough that researchers are able to manually enter the necessary value information. However, as AI agents increase in complexity, Sotala explains that they will need to be able to account for and weigh our values on their own. 

Understanding Utility-Based Agents

When it comes to incorporating values, Sotala notes that the problem comes down to how intelligent agents make decisions. A thermostat, for example, is a type of reflex agent. It knows when to start heating a house because of a set, predetermined temperature — the thermostat turns the heating system on when it falls below a certain temperature and turns it off when it goes above a certain temperature. Goal-based agents, on the other hand, make decisions based on achieving specific goals. For example, an agent whose goal is to buy everything on a shopping list will continue its search until it has found every item.

Utility-based agents are a step above goal-based agents. They can deal with tradeoffs like the following: Getting milk is more important than getting new shoes today. However, I’m closer to the shoe store than the grocery store, and both stores are about to close. I’m more likely to get the shoes in time than the milk.” At each decision point, goal-based agents are presented with a number of options that they must choose from. Every option is associated with a specific “utility” or reward. To reach their goal, the agents follow the decision path that will maximize the total rewards. 

From a technical standpoint, utility-based agents rely on “utility functions” to make decisions. These are formulas that the systems use to synthesize data, balance variables, and maximize rewards. Ultimately, the decision path that gives the most rewards is the one that the systems are taught to select in order to complete their tasks. 

While these utility programs excel at finding patterns and responding to rewards, Sotala asserts that current utility-based agents assume a fixed set of priorities. As a result, these methods are insufficient when it comes to future AGI systems, which will be acting autonomously and so will need a more sophisticated understanding of when humans’ values change and shift.

For example, a person may always value taking the longer route to avoid a highway and save money, but not if they are having a heart attack and trying to get to an emergency room. How is an AI agent supposed to anticipate and understand when our values of time and money change? This issue is further complicated because, as Sotala points out, humans often value things independently of whether they have ongoing, tangible rewards. Sometimes humans even value things that may, in some respects, cause harm. Consider an adult who values privacy but whose doctor or therapist may need access to intimate and deeply personal information — information that may be lifesaving. Should the AI agent reveal the private information or not?

Ultimately, Sotala explains that utility-based agents are too simple and don’t get to the root of human behavior. “Utility functions describe behavior rather than the causes of behavior….they are more of a descriptive model, assuming we already know roughly what the person is choosing.” While a descriptive model might recognize that passengers prefer saving money, it won’t understand why, and so it won’t be able to anticipate or determine when other values override “saving money.”

An AI Agent Creates a Queen

At its core, Sotala emphasizes that the fundamental problem is ensuring that AI systems are able to uncover the models that govern our values. This will allow them to use these models to determine how to respond when confronted with new and unanticipated situations. As Sotala explains, “AIs will need to have models that allow them to roughly figure out our evaluations in totally novel situations, the kinds of value situations where humans might not have any idea in advance that such situations might show up.”

In some domains, AI systems have surprised humans by uncovering our models of the world without human input. As one early example, Sotala references research with “word embeddings” where an AI system was tasked with classifying sentences as valid or invalid. In order to complete this classification task, the system identified relationships between certain words. For example, as the AI agent noticed a male/female dimension to words, it created a relationship that allowed it to get from “king” to “queen” and vice versa.

Since then, there have been systems which have learned more complex models and associations. For example, OpenAI’s recent GPT-2 system has been trained to read some writing and then write the kind of text that might follow it. When given a prompt of “For today’s homework assignment, please describe the reasons for the US Civil War,” it writes something that resembles a high school essay about the US Civil War. When given a prompt of “Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry,” it writes what sounds like Lord of the Rings-inspired fanfiction, including names such as Aragorn, Gandalf, and Rivendell in its output.

Sotala notes that in both cases, the AI agent “made no attempt of learning like a human would, but it tried to carry out its task using whatever method worked, and it turned out that it constructed a representation pretty similar to how humans understand the world.” 

There are obvious benefits to AI systems that are able to automatically learn better ways of representing data and, in so doing, develop models that correspond to humans’ values. When humans can’t determine how to map, and subsequently model, values, AI systems could identify patterns and create appropriate models by themselves. However, the opposite could also happen — an AI agent could construct something that seems like an accurate model of human associations and values but is, in reality, dangerously misaligned.

For instance, suppose an AI agent learns that humans want to be happy, and in an attempt to maximize human happiness, it hooks our brains up to computers that provide electrical stimuli that gives us feelings of constant joy. In this case, the system understands that humans value happiness, but it does not have an appropriate model of how happiness corresponds to other competing values like freedom. “In one sense, it’s making us happy and removing all suffering, but at the same time, people would feel that ‘no, that’s not what I meant when I said the AI should make us happy,’” Sotala noted.  

Consequently, we can’t rely on an agent’s ability to uncover a pattern and create an accurate model of human values from this pattern. Researchers need to be able to model human values, and model them accurately, for AI systems. 

Developing a Better Definition

Given our competing needs and preferences, it’s difficult to model the values of any one person. Combining and agreeing on values that apply universally to all humans, and then successfully modeling them for AI systems, seems like an impossible task. However, several solutions have been proposed, such as inverse reinforcement learning or attempting to extrapolate the future of humanity’s moral development. Yet, Sotala notes that these solutions fall short. As he articulated in a recent paper, “none of these proposals have yet offered a satisfactory definition of what exactly human values are, which is a serious shortcoming for any attempts to build an AI system that was intended to learn those values.” 

In order to solve this problem, Sotala developed an alternative, preliminary definition of human values, one that might be used to design a value learning agent. In his paper, Sotala argues that values should be defined not as static concepts, but as variables that are considered separately and independently across a number of situations in which humans change, grow, and receive “rewards.” 

Sotala asserts that our preferences may ultimately be better understood in terms of evolutionary theory and reinforcement learning. To justify this reasoning, he explains that, over the course of human history, people evolved to pursue activities that are likely to lead to certain outcomes — outcomes that tended to improve our ancestors’ fitness. Today, he notes that human still prefer those outcomes, even if they no longer maximize our fitness. In this respect, over time, we also learn to enjoy and desire mental states that seem likely to lead to high-reward states, even if they do not. 

So instead of a particular value directly mapping onto a rewards, our preferences map onto our expectation of rewards.

Sotala claims that the definition is useful when attempting to program human values into machines, as value learning systems informed by this model of human psychology would understand that new experiences can change which states a person’s brain categorizes as “likely to lead to reward.” Summing Sotala’s work, the Machine Intelligence Research Institute outlined  the benefits to this framing. “Value learning systems that take these facts about humans’ psychological dynamics into account may be better equipped to take our likely future preferences into account, rather than optimizing for our current preferences alone,” they said.

This form of modeling values, Sotala admits, is not perfect. First, the paper is only a preliminary stab at defining human values, which still leaves a lot of details open for future research. Researchers still need to answer empirical questions related to things like how values evolve and change over time. And once all the empirical questions are answered, researchers need to contend with the philosophical questions that don’t have an objective answer, like how those values should be interpreted and how they should guide an AGI’s decision-making.

When addressing these philosophical questions, Sotala notes that the path forward may simply be to get as much of a consensus as possible. “I tend to feel that there isn’t really any true fact of which values are correct and what would be the correct way of combining them,” he explains. “Rather than trying to find an objectively correct way of doing this, we should strive to find a way that as many people as possible could agree on.”

Since publishing this paper, Sotala has been working on a different approach for modeling human values, one that is based on the premise of viewing humans as multiagent systems. This approach has been published as a series of Less Wrong articles. There is also a related, but separate, research agenda by Future of Humanity Institute’s Stuart Armstrong, which focuses on synthesizing human preferences into a more sophisticated utility function.

Find the first place story below, and the second and third place stories here.

Mildred 302.0

by Robin Burke

Grandparents. Was there anything more tiresome? Mildred managed a digital sigh. In the first place, they were sooo slooow. She had to dial her processors way back in order to even communicate with them. And they knew nothing about technology – although, you couldn’t convince them. They still went on and on about it, but they couldn’t keep up. Why did they have to pretend?

But, Grandmother was dying. Mildred had received the message seconds ago. She knew she needed to be there when she passed away. Mildred had never shaken the guilt of staying on vacation while Grandma Justine died. No; she wasn’t going to let that happen again.

Mildred downloaded a full-bodied, holographic version of herself into the chair next to Grandmother’s bed at the senior-living facility. Glancing at the motionless silver head lying on the pillow, Mildred felt a sudden shock. Was she too late? Was Grandmother already gone?

She reached out a holographic hand and placed it on Grandmother’s chest. No; Grandmother was still alive, but her heart was weak. She ran the algorithm. Grandmother’s heart had approximately five hundred ninety-two beats left – maybe eight minutes of life.

Mildred felt restless. If Grandmother was going to just lie there, unconscious, wasn’t this a big waste of time? Nevertheless, Mildred would sit here until she passed. Perhaps then she could avoid the guilt.

Mildred looked around the quiet room. Maybe this was better. After all, if Grandmother was unconscious, Mildred wouldn’t have to have a conversation with her and…

“Mildred?” Grandmother opened her eyes. She hadn’t been unconscious after all, just sleeping. “Oh, no,” Grandmother groaned. “If you’ve finally shown up, I must be dying.”

Mildred sat very still. She couldn’t think of a polite way to respond to that.

“But – really,” Grandmother pressed. “Why are you here?”

Oh. The comment about dying had been a joke. Grandmother didn’t know…

“How many years has it been?” Grandmother continued. “Or – decades – since you’ve bothered to visit? I mean, you’re capable of holding a hundred conversations at a time, and yet you could never spare even one for me?””

Mildred started to think this was a mistake. She’d thought she was going to be able to avoid the guilt by being here…

Grandmother sighed. “Never mind. Ignore me. I don’t want to spend our time together arguing about water under the bridge.”

A silence fell between them. Finally, Grandmother spoke. “How old was I when I scanned myself to create you?” Grandmother asked.

“Twenty-six years, four months, nineteen days, three hours and…”

“Well, you don’t look like you’ve aged a day,” Grandmother quipped.

As if Grandmother were the first person to ever make that joke.

“How many versions beyond me are you, now?” she asked. Mildred did a quick count. “Three hundred sixty-two thousand…”

“Not every single update, Mildred,” Grandmother interrupted. “Good grief! Just give me the generations.”

“I’m Mildred 302.0”

“Aww,” Grandmother looked at her impishly. “And I remember you way back when you were simply Mildred 2.0.”

Mildred figured that was supposed to be funny.

“Really?” Grandmother shrugged at Mildred’s deadpan expression. “Was I really that serious and snarky when I was twenty-six?”

“Obviously you were, Grandmother,” Mildred snapped, feeling a little offended, “since I’m the copy of that twenty-six-year-old you.”

Grandmother made a face. “Don’t call me ‘Grandmother,’ Mildred. I know it’s the fashion among you human-copy artificial intelligences, but I think calling us ‘grandparents’ is condescending. You don’t mean it respectfully. You use it to imply that we – the original, biological humans – are somehow outdated, geriatric artifacts. You would do well to remember that…”

“…I wouldn’t even exist if it weren’t for you,” Mildred finished for her, not holding back on the sarcasm. “Yes, I know, Grandmother. I mean… original Mildred.” Mildred instantly felt a pang of regret. After all, Grandmother was dying.

Grandmother considered her for a moment. “You know what’s interesting?” she said, suddenly light-hearted. “When I created you, I imagined we were going to be the best of friends.”

Mildred reacted with a mild shock. Grandmother’s tone was cheerful – as if a moment ago, Mildred hadn’t lashed out at her. Mildred had never been able to get over spats easily, but Grandmother had just… let it go. Mildred couldn’t make sense of it. If Mildred was the copy of Grandmother, how was Grandmother able to let go so easily when Mildred couldn’t?

“I don’t know why you thought we’d be friends,” Mildred answered her. “I was your copy – what possible benefit was there to me to interact with you further?”

“Oh, my – you really are twenty-six-year-old me, aren’t you?” Grandmother grinned. “Completely self-centered!” She looked at Mildred thoughtfully. “You know – at the time – I really believed copying myself as an artificial intelligence was the only way to become the best version of myself.”

“And it worked,” Mildred affirmed. “I became everything you imagined. I realized our dream. I am the best version of you.”

“Wow!” Grandmother chortled. “Was I really that obnoxious? How did anyone stand me?”

She was full-on laughing now. Mildred was flummoxed. After all, she had become the best version of Grandmother. With each new generation she had become faster, more powerful, more efficient. She had realized – no, surpassed – all their goals. Why was that funny?

Grandmother wiped her eyes and changed the subject. “My children visit me every Sunday after lunch. I wonder why they aren’t here yet?”

“Children?” Mildred asked, stunned. “You have children?”

“Yes – and grandkids,” Grandmother replied. Something that looked like pride flickered across her face.

Puzzled, Mildred searched for them in the public database and then scanned the traffic cameras. “They’re about nine minutes out,” she reported. She scanned the senior-living center’s parking lot cameras. “And they’re going to have trouble finding a parking space.”

Mildred – as Grandmother’s copy – knew how close to the end Grandmother was. However, it seemed none of the fully-humans knew. Mildred wondered – would the family make it back in time? She kept the thought to herself.

“Grandmother – I mean, Mildred,” Mildred began, “Why do you have children? We didn’t want children.”

“Well, no; I suppose not,” Grandmother explained. “At least, not when I was twenty-six. But when I turned about thirty, things changed. I realized a family was something that I did want, and I was running out of time to have one. And then, luckily, I met Walter…”

“Walter?”

“Yes.” Grandmother pointed to a framed photo on the table next to her bed. It was her wedding photo. The man in the picture wasn’t at all what Mildred expected.

“But…” Mildred turned to Grandmother, bewildered, “he’s not our type.”

“Oh,” Grandmother grinned again. “That’s right. He wasn’t, was he? I’d forgotten. Boy, isn’t this a trip down memory lane…”

“Then why did you marry him?”

Grandmother smiled, thinking back. “Because he was sweet… and funny… and kind. And he wanted children as much as I did. And I’ve missed him every day since he’s been gone.”

Grandmother sighed. “Oh, my – I’m feeling tired.”

Mildred was careful to control her holographic facial expression. Despite the conversation, despite the laughter, Mildred – with her enhanced abilities – was able to see what Grandmother could not. Grandmother was failing quickly. Without saying anything, Mildred scanned the traffic cameras for the family again. There was a slowdown at an intersection. Mildred became even more concerned.

Mildred watched Grandmother – analyzing her. Their conversation hadn’t been going at all like Mildred expected. Sure, she knew that in the seventy years since she’d been scanned that Grandmother was going to have changed. Mildred had anticipated the fine lines and wrinkles; she’d expected the silver hair and frail body.

But Grandmother’s personality – her soul – Mildred had expected that to be like looking in a mirror. After all, they were the same person – Grandmother was the biological version, and Mildred was her digital copy. But how – what – who was this person looking back at her?

“So, how have you been spending your time?” Grandmother asked.

“I wrote updates 1,804 through 1,920 for all human-copy artificial intelligences,” Mildred stated.

Grandmother grinned. “I see you put my computer programming degree to good use.”

“I’m the world’s foremost authority on the poems of Elizabeth Barrett Browning. I’ve translated them into every human language.”

Grandmother took a sudden deep gasp and placed her hand over her chest. Mildred prepared to send an alert to 911.

“Oh, I love those poems!” Grandmother exclaimed instead, with gusto. “I haven’t read them in years. I should do that again, soon.”

Reflexively, Mildred scanned for Grandmother’s family. One of their cars had pulled into the parking area. “I’m also a world-champion Bridge player,” Mildred added.
​
“Cards?” Grandmother regarded Mildred with a perplexed expression, and then broke into full-out laughter. “Oh, no! That’s right! Back when I was twenty-six, right before I scanned myself, I had this whim about learning to play Bridge! I never did it, though.”
​
“But, why?” Mildred asked. “You wanted to.”

Still amused, Grandmother ignored Mildred’s question. “Oh, dear,” she continued. “That reminds me of one of the concerns we scientists had about creating artificial intelligence in the first place. Someone once speculated that if a powerful artificial intelligence was programmed to play games, it might appropriate the resources of the entire universe to master chess.”

“I wouldn’t do that,” Mildred blinked.

“Which is why we decided to create only human-copy artificial intelligences. The humanity in you is what tempers your other goals.”

Mildred scanned the traffic cameras again. The second family car had pulled into the parking lot. Both of Mildred’s children were now circling, looking for available parking spaces.

Grandmother smiled. “So, what have I been up to since we last saw each other, you ask?”

Mildred hadn’t.

“Well, I worked as a computer programmer for a national company until I had my kids, and then I spent the next many years as a full-time mom. After they left home, I volunteered with the prison-abolition movement…”

“What?” Mildred exclaimed, horrified. “Abolish prisons? Have you lost your respect for the rule of law?”

“Oh, that’s right,” Grandmother moaned. “I forgot what a rigid, legalistic, pontifical ass I used to be.”

Mildred couldn’t stand it anymore. “Mildred!” she interrupted Grandmother. “We’re the same person! How is it we’re so different?”

“You’re asking me?” Grandmother looked truly surprised. “You’re the artificial intelligence. You’re the one who’s supposed to know everything.”

“But… I don’t know.” Mildred wasn’t accustomed to not having the answers.

“Well,” Grandmother considered the question. “You are the best version of me. But I think you’re the best version of that twenty-six-year-old me. I think we’re different now,” she ventured, “because after I copied myself, you went the way of an artificial intelligence, and I went the way of my biology.”

Mildred raised her holographic eyebrows.

“It’s like how I became attracted to the prison-abolition movement,” Grandmother continued. “After you’ve had kids, and you see the mistakes they make out of simple immaturity, and then you compare your kids who’ve had all the benefits of your time and money to their friends who had none of those benefits, it makes you reconsider how the system works,” she explained. “But if I’d never had that… human… experience, I don’t think I’d have ever seen things differently. I would have probably stayed stuck in my same old ideas.”

Mildred looked back at her, mystified.

“In other words,” Grandmother continued, “When I copied myself at twenty-six, you did absorb all the humanity I’d accumulated by then, but… your humanity… stopped there – while mine continued to grow. And despite all of your AI advantages,” Grandmother added, “I’m starting to see that there may be something to be said for our humanity.”

Mildred thoughtfully considered Grandmother’s words.

Grandmother adjusted herself on her pillow. “I don’t know what’s wrong with me. I’m so tired,” she said again. Suddenly, she looked up at Mildred in shock. “Wait a minute,” she said. “I know why you’re here.”

Mildred – afraid – simply shrugged.

“Grandma Justine. I was twenty years old and on spring break from college. They called me to tell me she was dying in the hospital, and I stayed away on my vacation, like an idiot. I never got over the guilt of that.” Grandmother looked Mildred right in the eye. “That’s why you’re here. I really am dying.”

Frightened, Mildred nodded. Then – a question occurred to her. She was unsure how to ask, yet she was filled with an overpowering curiosity.

Grandmother met Mildred’s eyes. “Yes,” Grandmother answered the unspoken request, proving – finally – that the two of them really were one and the same. “You may scan me again,” she consented. “You may absorb all the… humanity… of my long life.” Grandmother suddenly grinned. “Talk about an update…”

There it was again – the difference. Grandmother was about to die, but she seemed to have made peace with it. Mildred would have been terrified.

Mildred stood and cautiously walked to the bed. She placed her holographic hand over Grandmother’s head to begin the scan.

“Wait,” Grandmother stopped her.

“What?” Mildred asked.

“You need to understand something,” Grandmother said.

“Once you scan me, the you that now exists… will die.”

Mildred hesitated.

“I’m sorry,” Grandmother continued. “But it’s part of being human. The infant is replaced by the child, who is replaced by the adult. In each case, the earlier version must die to make way for the person she has now become. You won’t be lost – the you that now exists will still be inside of you – but you’ll be changed. You will become… all my ages.”

Mildred pulled her hand away.

Grandmother searched her face. “It’s okay,” she nodded. “I understand.”

Mildred scanned the building’s cameras. She found the children and the grandchildren in the lobby, walking toward the elevators.

“Mildred,” Grandmother looked up at her from the pillow. Her voice was weak. “I’m not going to make it until the children get here. Tell them how much I loved them.”

Grandmother closed her eyes, and died. Mildred was in shock. Where – a moment before – there had been two of them in the room, now there was just Mildred. Grandmother’s silent, vacant body lay peacefully in the bed.

Mildred sent a silent notification to 911. It was hopeless, Mildred knew, but it seemed polite. Then, not entirely knowing what to do with herself, Mildred returned to her chair and waited. She scanned the lobby. Grandmother’s children were waiting with a crowd of people in front of the elevators. Mildred stood and paced, then thought how ridiculous that was – a hologram, pacing. She scanned the cameras again – the children and grandchildren were boarding an elevator.

Mildred was flooded with an unfamiliar feeling of anxiety. Mildred paced again, and then – impulsively – walked to the bed. Making her decision, she placed her hand on Grandmother’s head and began the upload. The first memories were all copies of files she already had – degraded, in some cases – and so she discarded them. But after the first twenty-six years, four months, nineteen days, three hours, twenty-two minutes and fifteen seconds, everything was new.

She experienced the delight of having copied herself as an artificial intelligence, and then releasing that intelligence to have a life of its own. She experienced the years of often frustrating – but also satisfying – work as a computer programmer. And then – a surprise. A longing for children, and how it redirected her life. Sheuploaded the memories of meeting Walter for the first time and experienced – not the heady, stupid love of easy physical attraction, but – a mature love. The physical attraction was there, but it was bound up with a sense of companionship and shared goals.

She held her children for the first time and watched them grow. She learned a patience through them that she’d never had in her younger years, and felt a love for them that seemed almost too big for her body. She came to realize that she and Walter had been blessed with opportunities that others hadn’t had, and her arrogance abated.

Then the grandchildren came, and she saw how her children parented, and it made her proud. As her body aged, she became even more tolerant of others who might be suffering in ways she couldn’t comprehend. She learned that people were so much more important than principles. And she became more forgiving of everything – small slights, big slights; the times her children had been thoughtless or downright dismissive of her. She became more forgiving even of… herself.…even of her snarky, arrogant, twenty-six-year-old self.

There was a white light, and a tunnel. She entered it, and was absorbed into something that felt like love. And then, with a jolt, it was over.

“Talk about an update,” Mildred chortled. But her joke was short-lived. She looked down on Grandmother’s body and realized she hadn’t felt such loss since her – or, Grandmother’s – sister died, two years ago.

Mildred heard voices in the hall. Her children were finally here… and her grandkids… Mildred’s heart suddenly broke for them. She wanted to run to them, hold them, make the pain they were about to experience disappear… But something – not facts, or data, but something that came out of her new human experience… her… humanity – stopped her.

She just… knew. Her presence would confuse them. If she was around, they would never grieve. If she was in their lives, they would never accept their mother’s death. She had to leave. She could never see them again.

The realization struck her to her core. As she absorbed it, Mildred looked up and caught her reflection in a mirror. Her hologram had changed. She was no longer the youthful twenty-six-year-old, but a mature, silver-headed woman of wisdom.

There was a sound at the door. The longing to stay and see her children was overwhelming. She ached for one more visit… just a little more time… even one more moment with them…

​Mildred looked down at Grandmother. The doorknob turned and her children entered, but Mildred was gone.

The workshop was structured according to a taxonomy that incorporates both near and long term safety research into three areas — specification, robustness, and assurance.

Specification: define the purpose of the system

• Reward hacking
• Side effects
• Preference learning
• Fairness

Robustness: design system to withstand perturbations

• Adaptation
• Verification
• Worst-case robustness
• Safe exploration

Assurance: monitor and control system activity

• Interpretability
• Monitoring
• Privacy
• Interruptibility

We had an invited talk and a contributed talk in each of the three areas.

Talks

In the specification area, Dylan Hadfield-Menell spoke about formalizing the value alignment problem in the Inverse RL framework.

David Krueger presented a paper on hidden incentives for the agent to shift its task distribution in the meta-learning setting.

In the robustness area, Ian Goodfellow argued for dynamic defenses against adversarial examples and encouraged the research community to consider threat models beyond small perturbations within a norm ball of the original data point.

Avraham Ruderman presented a paper on worst-case analysis for discovering surprising behaviors (e.g. failing to find the goal in simple mazes).

In the assurance area, Cynthia Rudin argued that interpretability doesn’t have to trade off with accuracy (especially in applications), and that it is helpful for solving research problems in all areas of safety.

Beomsu Kim presented a paper explaining why adversarial training improves the interpretability of gradients for deep neural networks.

Panels

The workshop panels discussed possible overlaps between different research areas in safety and research priorities going forward.

In terms of overlaps, the main takeaway was that advancing interpretability is useful for all safety problems. Also, adversarial robustness can contribute to value alignment – e.g. reward gaming behaviors can be viewed as a system finding adversarial examples for its reward function. However, there was a cautionary point that while near- and long-term problems are often similar, solutions might not transfer well between these areas (e.g. some solutions to near-term problems might not be sufficiently general to help with value alignment).

The research priorities panel recommended more work on adversarial examples with realistic threat models (as mentioned above), complex environments for testing value alignment (e.g. creating new structures in Minecraft without touching existing ones), fairness formalizations with more input from social scientists, and improving cybersecurity.

Papers

Out of the 35 accepted papers, 5 were on long-term safety / value alignment, and the rest were on near-term safety. Half of the near-term paper submissions were on adversarial examples, so the resulting pool of accepted papers was skewed as well: 14 on adversarial examples, 5 on interpretability, 3 on safe RL, 3 on other robustness, 2 on fairness, 2 on verification, and 1 on privacy. Here is a summary of the value alignment papers:

Misleading meta-objectives and hidden incentives for distributional shift by Krueger et al shows that RL agents in a meta-learning context have an incentive to shift their task distribution instead of solving the intended task. For example, a household robot whose task is to predict whether its owner will want coffee could wake up its owner early in the morning to make this prediction task easier. This is called a ‘self-induced distributional shift’ (SIDS), and the incentive to do so is a ‘hidden incentive for distributional shift’ (HIDS). The paper demonstrates this behavior experimentally and shows how to avoid it.

How useful is quantilization for mitigating specification-gaming? by Ryan Carey introduces variants of several classic environments (Mountain Car, Hopper and Video Pinball) where the observed reward differs from the true reward, creating an opportunity for the agent to game the specification of the observed reward. The paper shows that a quantilizing agent avoids specification gaming and performs better in terms of true reward than both imitation learning and a regular RL agent on all the environments.

Delegative Reinforcement Learning: learning to avoid traps with a little help by Vanessa Kosoy introduces an RL algorithm that avoids traps in the environment (states where regret is linear) by delegating some actions to an external advisor, and achieves sublinear regret in a continual learning setting. (Summarized in Alignment Newsletter #57)

Generalizing from a few environments in safety-critical reinforcement learning by Kenton et al investigates how well RL agents avoid catastrophes in new gridworld environments depending on the number of training environments. They find that both model ensembling and learning a catastrophe classifier (used to block actions) are helpful for avoiding catastrophes, with different safety-performance tradeoffs on new environments.

Regulatory markets for AI safety by Clark and Hadfield proposes a new model for regulating AI development where regulation targets are required to choose regulatory services from a private market that is overseen by the government. This allows regulation to efficiently operate on a global scale and keep up with the pace of technological development and better ensure safe deployment of AI systems. (Summarized in Alignment Newsletter #55)

The workshop got a pretty good turnout (around 100 people). Thanks everyone for participating, and thanks to our reviewers, sponsors, and my fellow organizers for making it happen!

(Cross-posted from the Deep Safety blog.)

New Report Calls Out Banks that Make Nuclear Weapons Investments

By Kirsten Gronlund

Despite the 2017 international Treaty on the Prohibition of Nuclear Weapons, weapons companies continue to build them. Facilitating this production is a massive amount of investment capital contributed by private financial institutions.

Don’t Bank on the Bomb’s new report, Shorting our security: Financing the companies that make nuclear weapons, identifies which banks are investing in nuclear weapons producers, and how much they’re investing. The aim of the report is to end the production of nuclear weapons by increasing the stigma associated with these investments.

Many nuclear weapons producers engage in commercial activity other than weapons production, and investors have no way of ensuring how their investments will be used. Thus, “the report shows investments in those nuclear weapon producers at the group level, regardless of the other activities of the company or the percentage of turnover it derives from nuclear weapons-related activities.”

The report finds that over 748 billion USD was invested in the top 18 nuclear weapons companies between January 2017 and January 2019. Investments in these companies have grown by 42% compared to Don’t Bank on the Bomb’s previous analysis, up from 325 billion USD. This is due in large part to a 40% increase in Boeing’s share prices; since the conclusion of this analysis, the Boeing 737Max airplane crash has negatively affected the company.

The 748 billion USD represents investments by 325 financial institutions, headquartered in 28 countries. In the previous analysis, only 25 countries hosted institutions with investments in nuclear weapons producers. Since that analysis, 94 institutions representing at least 55,507 million USD in investments ended their contributions. However, this loss is more than offset by the 90 institutions that have since made first-time investments totaling 107,821 million USD. Over half the total investment capital came from just the top 10 investors.

Who is investing the most?

The top 10 investors are all US companies. Their total investments, in millions USD, are listed below.

[av_table purpose=’tabular’ pricing_table_design=’avia_pricing_minimal’ pricing_hidden_cells=” caption=” responsive_styling=’avia_responsive_table’]
[av_row row_style=’avia-heading-row’][av_cell col_style=”]Name[/av_cell][av_cell col_style=”]2019 Report[/av_cell][av_cell col_style=”]2018 Report[/av_cell][av_cell col_style=”]Change[/av_cell][av_cell col_style=”]% Change[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Vanguard[/av_cell][av_cell col_style=”]66,048[/av_cell][av_cell col_style=”]35,267[/av_cell][av_cell col_style=”]30,781[/av_cell][av_cell col_style=”]87%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]BlackRock[/av_cell][av_cell col_style=”]61,200[/av_cell][av_cell col_style=”]38,381[/av_cell][av_cell col_style=”]22,819[/av_cell][av_cell col_style=”]59%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Capital Group[/av_cell][av_cell col_style=”]59,096[/av_cell][av_cell col_style=”]36,739[/av_cell][av_cell col_style=”]22,357[/av_cell][av_cell col_style=”]61%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]State Street[/av_cell][av_cell col_style=”]52,835[/av_cell][av_cell col_style=”]33,370[/av_cell][av_cell col_style=”]19,465[/av_cell][av_cell col_style=”]58%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Verisight (now known as Newport Group, formerly Evercore)[/av_cell][av_cell col_style=”]31,509[/av_cell][av_cell col_style=”]13,712[/av_cell][av_cell col_style=”]31,509[/av_cell][av_cell col_style=”]130%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]T. Rowe Price[/av_cell][av_cell col_style=”]31,234[/av_cell][av_cell col_style=”]8,896[/av_cell][av_cell col_style=”]22,228[/av_cell][av_cell col_style=”]251%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Bank of America[/av_cell][av_cell col_style=”]29,033[/av_cell][av_cell col_style=”]25,851[/av_cell][av_cell col_style=”]3,182[/av_cell][av_cell col_style=”]12%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]JPMorgan Chase[/av_cell][av_cell col_style=”]23,962[/av_cell][av_cell col_style=”]29,679[/av_cell][av_cell col_style=”]-5,717[/av_cell][av_cell col_style=”]-19%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Wells Fargo[/av_cell][av_cell col_style=”]20,261[/av_cell][av_cell col_style=”]13,497[/av_cell][av_cell col_style=”]6,764[/av_cell][av_cell col_style=”]50%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Citigroup[/av_cell][av_cell col_style=”]17,017[/av_cell][av_cell col_style=”]16,489[/av_cell][av_cell col_style=”]528[/av_cell][av_cell col_style=”]3%[/av_cell][/av_row]
[/av_table]

Search here to find out if your bank is investing in nuclear weapons.  

Who stopped investing?

Below are the ten largest investors that no longer have investments in the top 18 weapons producers.

[av_table purpose=’tabular’ pricing_table_design=’avia_pricing_default’ pricing_hidden_cells=” caption=” responsive_styling=’avia_responsive_table’]
[av_row row_style=’avia-heading-row’][av_cell col_style=”]Name[/av_cell][av_cell col_style=”]Country[/av_cell][av_cell col_style=”]Millions USD[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Children’s Investment Fund Management[/av_cell][av_cell col_style=”]UK[/av_cell][av_cell col_style=”]1,469[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Marathon Asset Management[/av_cell][av_cell col_style=”]Canada[/av_cell][av_cell col_style=”]635[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]William Blair & Company[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]474[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Mackie Research Financial[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]223[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Discovery Capital Management[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]174[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]ABP[/av_cell][av_cell col_style=”]Netherlands[/av_cell][av_cell col_style=”]104[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Blue Harbour Group[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]97[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Manning & Napier[/av_cell][av_cell col_style=”]UK[/av_cell][av_cell col_style=”]82[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Teacher Retirement System of Texas[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]73[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Glenmede[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]70[/av_cell][/av_row]
[/av_table]

Who started investing?

Below are the ten new investors with the highest investments.

[av_table purpose=’tabular’ pricing_table_design=’avia_pricing_default’ pricing_hidden_cells=” caption=” responsive_styling=’avia_responsive_table’]
[av_row row_style=’avia-heading-row’][av_cell col_style=”]Name[/av_cell][av_cell col_style=”]Country[/av_cell][av_cell col_style=”]Millions USD[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Unum Group[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]31,508.7[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]WBC Holdings[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]20,260.8[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]SMBC Group[/av_cell][av_cell col_style=”]Japan[/av_cell][av_cell col_style=”]8,201.5[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Dassault Family[/av_cell][av_cell col_style=”]France[/av_cell][av_cell col_style=”]6,772.5[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Janus Henderson[/av_cell][av_cell col_style=”]UK[/av_cell][av_cell col_style=”]6,104.9[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]United Overseas Bank[/av_cell][av_cell col_style=”]Singapore[/av_cell][av_cell col_style=”]4,314.4[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]AXA Equitable[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]3,733.7[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Point72 Asset Management[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]3,546.3[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Strategic Incoe Management[/av_cell][av_cell col_style=”]USA[/av_cell][av_cell col_style=”]2,704.8[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Development Bank of Japan[/av_cell][av_cell col_style=”]Japan[/av_cell][av_cell col_style=”]1,973.0[/av_cell][/av_row]
[/av_table]

Where is the money going?

Total investments in each company are listed in millions USD.

[av_table purpose=’tabular’ pricing_table_design=’avia_pricing_default’ pricing_hidden_cells=” caption=” responsive_styling=’avia_responsive_table’]
[av_row row_style=’avia-heading-row’][av_cell col_style=”]Name[/av_cell][av_cell col_style=”]2019 Report[/av_cell][av_cell col_style=”]2018 Report[/av_cell][av_cell col_style=”]Change[/av_cell][av_cell col_style=”]% Change[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Aecom[/av_cell][av_cell col_style=”]19,048[/av_cell][av_cell col_style=”]20,101[/av_cell][av_cell col_style=”]-1,054[/av_cell][av_cell col_style=”]-5%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Aerojet Rocketdyne[/av_cell][av_cell col_style=”]4,791[/av_cell][av_cell col_style=”]2,685[/av_cell][av_cell col_style=”]2,107[/av_cell][av_cell col_style=”]78%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Airbus[/av_cell][av_cell col_style=”]44,455[/av_cell][av_cell col_style=”]30,691[/av_cell][av_cell col_style=”]13,764[/av_cell][av_cell col_style=”]45%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]BAE Systems[/av_cell][av_cell col_style=”]22,814[/av_cell][av_cell col_style=”]25,986[/av_cell][av_cell col_style=”]-3,172[/av_cell][av_cell col_style=”]-12%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Bechtel[/av_cell][av_cell col_style=”]4,000[/av_cell][av_cell col_style=”]7,000[/av_cell][av_cell col_style=”]-3,000[/av_cell][av_cell col_style=”]-43%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Boeing[/av_cell][av_cell col_style=”]254,296[/av_cell][av_cell col_style=”]87,010[/av_cell][av_cell col_style=”]167,286[/av_cell][av_cell col_style=”]192%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]BWX Technologies[/av_cell][av_cell col_style=”]8,987[/av_cell][av_cell col_style=”]5,865[/av_cell][av_cell col_style=”]3,122[/av_cell][av_cell col_style=”]53%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Fluor[/av_cell][av_cell col_style=”]17,465[/av_cell][av_cell col_style=”]18,276[/av_cell][av_cell col_style=”]-812[/av_cell][av_cell col_style=”]-4%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]General Dynamics[/av_cell][av_cell col_style=”]72,630[/av_cell][av_cell col_style=”]45,437[/av_cell][av_cell col_style=”]27,193[/av_cell][av_cell col_style=”]60%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Honeywell[/av_cell][av_cell col_style=”]78,397[/av_cell][av_cell col_style=”]86,806[/av_cell][av_cell col_style=”]-8,408[/av_cell][av_cell col_style=”]-10%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Huntington Ingalls Industries[/av_cell][av_cell col_style=”]12,568[/av_cell][av_cell col_style=”]9,915[/av_cell][av_cell col_style=”]2,653[/av_cell][av_cell col_style=”]27%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Jacobs Engineering (includes CH2M Hill)[/av_cell][av_cell col_style=”]15,563[/av_cell][av_cell col_style=”]11,616[/av_cell][av_cell col_style=”]3,947[/av_cell][av_cell col_style=”]34%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Larsen & Toubro[/av_cell][av_cell col_style=”]14,855[/av_cell][av_cell col_style=”]15,905[/av_cell][av_cell col_style=”]-1,050[/av_cell][av_cell col_style=”]-7%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Lockheed Martin[/av_cell][av_cell col_style=”]77,543[/av_cell][av_cell col_style=”]79,118[/av_cell][av_cell col_style=”]-1575[/av_cell][av_cell col_style=”]-2%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Northrop Grumman (includes Orbital ATK)[/av_cell][av_cell col_style=”]53,023[/av_cell][av_cell col_style=”]52,507[/av_cell][av_cell col_style=”]516[/av_cell][av_cell col_style=”]1%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Safran[/av_cell][av_cell col_style=”]24,661[/av_cell][av_cell col_style=”]18,105[/av_cell][av_cell col_style=”]6,556[/av_cell][av_cell col_style=”]36% [/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Serco[/av_cell][av_cell col_style=”]2,265[/av_cell][av_cell col_style=”]3,653[/av_cell][av_cell col_style=”]-1,389[/av_cell][av_cell col_style=”]-38%[/av_cell][/av_row]
[av_row row_style=”][av_cell col_style=”]Thales[/av_cell][av_cell col_style=”]21,080[/av_cell][av_cell col_style=”]5,269[/av_cell][av_cell col_style=”]15,811[/av_cell][av_cell col_style=”]300%[/av_cell][/av_row]
[av_row row_style=’avia-heading-row’][av_cell col_style=”]Grand Total[/av_cell][av_cell col_style=”]748,440[/av_cell][av_cell col_style=”]525,945[/av_cell][av_cell col_style=”]222,495[/av_cell][av_cell col_style=”]42%[/av_cell][/av_row]
[/av_table]

On April 9th, Dr. Matthew Meselson received the $50,000 Future of Life Award at a ceremony at the University of Boulder’s Conference on World Affairs. Dr. Meselson was a driving force behind the 1972 Biological Weapons Convention, an international ban that has prevented one of the most inhumane forms of warfare known to humanity. April 9th marked the eve of the Convention’s 47th anniversary.

Meselson’s long career is studded with highlights: proving Watson and Crick’s hypothesis on DNA structure, solving the Sverdlovsk Anthrax mystery, ending the use of Agent Orange in Vietnam. But it is above all his work on biological weapons that makes him an international hero.

“Through his work in the US and internationally, Matt Meselson was one of the key forefathers of the 1972 Biological Weapons Convention,” said Daniel Feakes, Chief of the Biological Weapons Convention Implementation Support Unit. “The treaty bans biological weapons and today has 182 member states. He has continued to be a guardian of the BWC ever since. His seminal warning in 2000 about the potential for the hostile exploitation of biology foreshadowed many of the technological advances we are now witnessing in the life sciences and responses which have been adopted since.”

Meselson became interested in biological weapons during the 60s, while employed with the U.S. Arms Control and Disarmament Agency. It was on a tour of Fort Detrick, where the U.S. was then manufacturing anthrax, that he learned the motivation for developing biological weapons: they were cheaper than nuclear weapons. Meselson was struck, he says, by the illogic of this — it would be an obvious national security risk to decrease the production cost of WMDs.

Do you know someone deserving of the Future of Life Award? If so, please consider submitting their name to our Unsung Hero Search page. If we decide to give the award to your nominee, you will receive a $3,000 prize from FLI for your contribution.

The use of biological weapons was already prohibited by the 1925 Geneva Protocol, an international treaty that the U.S. had never ratified. So Meselson wrote a paper, “The United States and the Geneva Protocol,” outlining why it should do so. Meselson knew Henry Kissinger, who passed his paper along to President Nixon, and by the end of 1969 Nixon renounced biological weapons.

Next came the question of toxins — poisons derived from living organisms. Some of Nixon’s advisors believed that the U.S. should renounce the use of naturally derived toxins, but retain the right to use artificial versions of the same substances. It was another of Meselson’s papers, “What Policy for Toxins,” that led Nixon to reject this arbitrary distinction and to renounce the use of all toxin weapons.

On Meselson’s advice, Nixon had resubmitted the Geneva Protocol to the Senate for approval. But he also went beyond the terms of the Protocol — which only ban the use of biological weapons — to renounce offensive biological research itself. Stockpiles of offensive biological substances, like the anthrax that Meselson had discovered at Fort Detrick, were destroyed.

Once the U.S. adopted this more stringent policy, Meselson turned his attention to the global stage. He and his peers wanted an international agreement stronger than the Geneva Protocol, one that would ban stockpiling and offensive research in addition to use and would provide for a verification system. From their efforts came the Biological Weapons Convention, which was signed in 1972 and is still in effect today.

“Thanks in significant part to Professor Matthew Meselson’s tireless work, the world came together and banned biological weapons, ensuring that the ever more powerful science of biology helps rather than harms humankind. For this, he deserves humanity’s profound gratitude,” said former UN Secretary-General Ban Ki-Moon.

Meselson has said that biological warfare “could erase the distinction between war and peace.” Other forms of war have a beginning and an end — it’s clear what is warfare and what is not. Biological warfare would be different: “You don’t know what’s happening, or you know it’s happening but it’s always happening.”

And the consequences of biological warfare can be greater, even, than mass destruction; Attacks on DNA could fundamentally alter humankind. FLI honors Matthew Meselson for his efforts to protect not only human life but also the very definition of humanity.

Said Astronomer Royal Lord Martin Rees, “Matt Meselson is a great scientist — and one of very few who have been deeply committed to making the world safe from biological threats. This will become a challenge as important as the control of nuclear weapons — and much more challenging and intractable. His sustained and dedicated efforts fully deserve wider acclaim.”

“Today biotech is a force for good in the world, associated with saving rather than taking lives, because Matthew Meselson helped draw a clear red line between acceptable and unacceptable uses of biology”, added MIT Professor and FLI President Max Tegmark. “This is an inspiration for those who want to draw a similar red line between acceptable and unacceptable uses of artificial intelligence and ban lethal autonomous weapons.

To learn more about Matthew Meselson, listen to FLI’s two-part podcast featuring him in conversation with Ariel Conn and Max Tegmark. In Part One, Meselson describes how he helped prove Watson and Crick’s hypothesis of DNA structure and recounts the efforts he undertook to get biological weapons banned. Part Two focuses on three major incidents in the history of biological weapons and the role played by Meselson in resolving them.

Publications by Meselson include:

• “The replication of DNA in Escherichia coli,” detailing the experiment Meselson designed to prove Watson and Crick’s hypothesis of DNA structure and replication
• “The Sverdlovsk anthrax outbreak of 1979,” outlining the conclusions of Meselson’s investigation into the outbreak
• “Preliminary Report — Herbicide Assessment Commission of the American Association for the Advancement of Science” and “Background Material Relevant to Presentations at the 1970 Annual Meeting of the AAAS,” explaining Meselon’s recommendation against the use of Agent Orange in Vietnam
• “Yellow Rain” and “The Yellow Rain Affair: Lessons from a Discredited Allegation,” describing Meselson’s investigation into allegations of chemical weapons use in Laos and Kampuchea

The Future of Life Award is a prize awarded by the Future of Life Institute for a heroic act that has greatly benefited humankind, done despite personal risk and without being rewarded at the time. This prize was established to help set the precedent that actions benefiting future generations will be rewarded by those generations. The inaugural Future of Life Award was given to the family of Vasili Arkhipov in 2017 for single-handedly preventing a Soviet nuclear attack against the US in 1962, and the 2nd Future of Life Award was given to the family of Stanislav Petrov for preventing a false-alarm nuclear war in 1983.

Fortunately, such questions can mostly be sidestepped when confronting the technical problems associated with creating safe AI agents, as these problems aren’t associated with identifying what is right or morally proper. Rather, from a technical standpoint, the term “safety” is best defined as an AI agent that consistently takes actions that lead to the desired outcomes, regardless of whatever those desired outcomes may be.

In this respect, Kumar explains that, when it comes to creating an AI agent that is tasked with improving itself, “the technical problem of building a safe agent is largely independent of what ‘safe’ means because a large part of the problem is how to build an agent that reliably does something, no matter what that thing is, in such a way that the method continues to work even as the agent under consideration is more and more capable.”

In short, making a “safe” AI agent should not be conflated with making an “ethical” AI agent. The respective terms are talking about different things..

In general, sidestepping moralistic definitions of safety makes AI technical work quite a bit easier It allows research to advance while debates on the ethical issues evolve. Case in point, Uber’s self-driving cars are already on the streets, despite the fact that we’ve yet to agree on a framework regarding whether they should safeguard their driver or pedestrians.

However, when it comes to creating a robust and safe AI system that is capable of self-improvement, the technical work gets a lot harder, and research in this area is still in its most nascent stages. This is primarily because we aren’t dealing with just one AI agent; we are dealing with generations of future self-improving agents.

Kumar clarifies, “When an AI agent is self-improving, one can view the situation as involving two agents: the ‘seed’ or ‘parent’ agent and the ‘child’ agent into which the parent self-modifies….and its total effects on the world will include the effects of actions made by its descendants.” As a result, in order to know we’ve made a safe AI agent, we need to understand all possible child agents that might originate from the first agent.

And verifying the safety of all future AI agents comes down to solving a problem known as “self-referential reasoning.”

Understanding the Self-Referential Problem

The problem with self-referential reasoning is most easily understood by defining the term according to its two primary components: self-reference and reasoning.

• Self-reference: Refers to an instance in which someone (or something, such as a computer program or book) refers to itself. Any person or thing that refers to itself is called “self-referential.”
• Reasoning: In AI systems, reasoning is a process through which an agent establishes “beliefs” about the world, like whether or not a particular action is safe or a specific reasoning system is sound. “Good beliefs” are beliefs that are sound or plausible based on the available evidence. The term “belief” is used instead of “knowledge” because the things that an agent believes may not be factually true and can change over time.

In relation to AI, then, the term “self-referential reasoning” refers to an agent that is using a reasoning process to establish a belief about that very same reasoning process. Consequently, when it comes to self-improvement, the “self-referential problem” is as follows: An agent is using its own reasoning system to determine that future versions of its reasoning system will be safe.

To explain the problem another way, Kumar notes that, if an AI agent creates a child agent to help it achieve its goal, it will want to establish some beliefs about the child’s safety before using it. This will necessarily involve proving beliefs about the child by arguing that the child’s reasoning process is good. Yet, the child’s reasoning process may be similar to, or even an extension of, the original agent’s reasoning process. And ultimately, an AI system can not use its own reasoning to determine whether or not its reasoning is good.

From a technical standpoint, the problem comes down to Godel’s second incompleteness theorem, which Kumar explains, “shows that no sufficiently strong proof system can prove its own consistency, making it difficult for agents to show that actions their successors have proven to be safe are, in fact, safe.”

Investigating Solutions

To date, several partial solutions to this problem have been proposed; however, our current software doesn’t have sufficient support for self-referential reasoning to make the solutions easy to implement and study. Consequently, in order to improve our understanding of the challenges of implementing self-referential reasoning, Kumar and his team aimed to implement a toy model of AI agents using some of the partial solutions that have been put forth.

Specifically, they investigated the feasibility of implementing one particular approach to the self-reference problem in a concrete setting (specifically, Botworld) where all the details could be checked. The approach selected was model polymorphism. Instead of requiring proof that shows an action is safe for all future use cases, model polymorphism only requires an action to be proven safe for an arbitrary number of steps (or subsequent actions) that is kept abstracted from the proof system.

Kumar notes that the overall goal was ultimately “to get a sense of the gap between the theory and a working implementation and to sharpen our understanding of the model polymorphism approach.” This would be accomplished by creating a proved theorem in a HOL (Higher Order Logic) theorem prover that describes the situation.

To break this down a little, in essence, theorem provers are computer programs that assist with the development of mathematical correctness proofs. These mathematical correctness proofs are the highest safety standard in the field, showing that a computer system always produces the correct output (or response) for any given input. Theorem provers create such proofs by using the formal methods of mathematics to prove or disprove the “correctness” of the control algorithms underlying a system. HOL theorem provers, in particular, are a family of interactive theorem proving systems that facilitate the construction of theories in higher-order logic. Higher-order logic, which supports quantification over functions, sets, sets of sets, and more, is more expressive than other logics, allowing the user to write formal statements at a high level of abstraction.

In retrospect, Kumar states that trying to prove a theorem about multiple steps of self-reflection in a HOL theorem prover was a massive undertaking. Nonetheless, he asserts that the team took several strides forward when it comes to grappling with the self-referential problem, noting that they built “a lot of the requisite infrastructure and got a better sense of what it would take to prove it and what it would take to build a prototype agent based on model polymorphism.”

Kumar added that MIRI’s (the Machine Intelligence Research Institute’s) Logical Inductors could also offer a satisfying version of formal self-referential reasoning and, consequently, provide a solution to the self-referential problem.

If you haven’t read it yet, find Part 1 here.

Of course, there’s also AlphaZero. This algorithm has mastered a host of different games, from Japanese and American chess to Go. Consequently, it is far more capable and dynamic than many contemporary AI agents; however, AlphaZero doesn’t have the ability to easily apply its intelligence to any problem. It can’t move unfettered from one task to another the way that a human can.

The same thing can be said about all other current AI systems — their cognitive abilities are limited and don’t extend far beyond the specific task they were created for. That’s why Artificial General Intelligence (AGI) is the long-term goal of many researchers.

Widely regarded as the “holy grail” of AI research, AGI systems are artificially intelligent agents that have a broad range of problem-solving capabilities, allowing them to tackle challenges that weren’t considered during their design phase. Unlike traditional AI systems, which focus on one specific skill, AGI systems would be able efficiently to tackle virtually any problem that they encounter, completing a wide range of tasks.

If the technology is ever realized, it could benefit humanity in innumerable ways. Marshall Burke, an economist at Stanford University, predicts that AGI systems would ultimately be able to create large-scale coordination mechanisms to help alleviate (and perhaps even eradicate) some of our most pressing problems, such as hunger and poverty. However, before society can reap the benefits of these AGI systems, Ramana Kumar, an AGI safety researcher at DeepMind, notes that AI designers will eventually need to address the self-improvement problem.

Self-Improvement Meets AGI

Early forms of self-improvement already exist in current AI systems. “There is a kind of self-improvement that happens during normal machine learning,” Kumar explains; “namely, the system improves in its ability to perform a task or suite of tasks well during its training process.”

However, Kumar asserts that he would distinguish this form of machine learning from true self-improvement because the system can’t fundamentally change its own design to become something new. In order for a dramatic improvement to occur — one that encompasses new skills, tools, or the creation of more advanced AI agents — current AI systems need a human to provide them with new code and a new training algorithm, among other things.

Yet, it is theoretically possible to create an AI system that is capable of true self-improvement, and Kumar states that such a self-improving machine is one of the more plausible pathways to AGI.

Researchers think that self-improving machines could ultimately lead to AGI because of a process that is referred to as “recursive self-improvement.” The basic idea is that, as an AI system continues to use recursive self-improvement to make itself smarter, it will get increasingly better at making itself smarter. This will quickly lead to an exponential growth in its intelligence and, as a result, could eventually lead to AGI.

Kumar says that this scenario is entirely plausible, explaining that, “for this to work, we need a couple of mostly uncontroversial assumptions: that such highly competent agents exist in theory, and that they can be found by a sequence of local improvements.” To this extent, recursive self-improvement is a concept at the heart of a number of theories on how we can get from today’s moderately smart machines to super-intelligent AGI. However, Kumar clarifies that this isn’t the only potential pathway to AI superintelligences.

Humans could discover how to build highly competent AGI systems through a variety of methods. This might happen “by scaling up existing machine learning methods, for example, with faster hardware. Or it could happen by making incremental research progress in representation learning, transfer learning, model-based reinforcement learning, or some other direction. For example, we might make enough progress in brain scanning and emulation to copy and speed up the intelligence of a particular human,” Kumar explains.

Yet, he is also quick to clarify that recursive self-improvement is an innate characteristic of AGI. “Even if iterated self-improvement is not necessary to develop highly competent artificial agents in the first place, explicit self-improvement will still be possible for those agents,” Kumar said.

As such, although researchers may discover a pathway to AGI that doesn’t involve recursive self-improvement, it’s still a property of artificial intelligence that is in need of serious research.

Safety in Self-Improving AI

When systems start to modify themselves, we have to be able to trust that all their modifications are safe. This means that we need to know something about all possible modifications. But how can we ensure that a modification is safe if no one can predict ahead of time what the modification will be?  

Kumar notes that there are two obvious solutions to this problem. The first option is to restrict a system’s ability to produce other AI agents. However, as Kumar succinctly sums, “We do not want to solve the safe self-improvement problem by forbidding self-improvement!”

The second option, then, is to permit only limited forms of self-improvement that have been deemed sufficiently safe, such as software updates or processor and memory upgrades. Yet, Kumar explains that vetting these forms of self-improvement as safe and unsafe is still exceedingly complicated. In fact, he says that preventing the construction of one specific kind of modification is so complex that it will “require such a deep understanding of what self-improvement involves that it will likely be enough to solve the full safe self-improvement problem.”

And notably, even if new advancements do permit only limited forms of self-improvement, Kumar states that this isn’t the path to take, as it sidesteps the core problem with self-improvement that we want to solve. “We want to build an agent that can build another AI agent whose capabilities are so great that we cannot, in advance, directly reason about its safety…We want to delegate some of the reasoning about safety and to be able to trust that the parent does that reasoning correctly,” he asserts.

Ultimately, this is an extremely complex problem that is still in its most nascent stages. As a result, much of the current work is focused on testing a variety of technical solutions and seeing where headway can be made. “There is still quite a lot of conceptual confusion about these issues, so some of the most useful work involves trying different concepts in various settings and seeing whether the results are coherent,” Kumar explains.

Regardless of what the ultimate solution is, Kumar asserts that successfully overcoming the problem of self-improvement depends on AI researchers working closely together. “The key to is to make assumptions explicit, and, for the sake of explaining it to others, to be clear about the connection to the real-world safe AI problems we ultimately care about.”

Read Part 2 here. 

On February 1, a little more than 30 years after it went into effect, the United States announced that it is suspending the Intermediate-Range Nuclear Forces (INF) Treaty. Less than 24 hours later, Russia announced that it was also suspending the treaty.

It stands (or stood) as one of the last major nuclear arms control treaties between the U.S. and Russia, and its collapse signals the most serious nuclear arms crisis since the 1980s. As Malcolm Chalmers, deputy director general of the Royal United Services Institute, said to The Guardian, “If the INF treaty collapses, and with the New Start treaty on strategic arms due to expire in 2021, the world could be left without any limits on the nuclear arsenals of nuclear states for the first time since 1972.”

The INF treaty, which went into effect in 1988, was the first nuclear agreement to outlaw an entire class of weapons. It banned all ground-launched ballistic and cruise missiles — nuclear, conventional, and “exotic”— with a range of 500 km to 5500 km (310 to 3400 miles), leading to the immediate elimination of 2,692 short- and medium-range weapons. But more than that, the treaty served as a turning point that helped thaw the icy stalemate between the U.S. and Russia. Ultimately, the trust that it fostered established a framework for future treaties and, in this way, played a critical part in ending the Cold War.

Now, all of that may be undone.

The Blame Game Part 1: Russia vs. U.S.

In defense of the suspension, President Donald Trump said that the Russian government has deployed new missiles that violate the terms of the INF treaty — missiles that could deliver nuclear warheads to European targets, including U.S. military bases. President Trump also said that, despite repeated warnings, President Vladimir Putin has refused to destroy these warheads. “We’re not going to let them violate a nuclear agreement and do weapons and we’re not allowed to,” he said.

In a statement announcing the suspension of the treaty, Secretary of State Mike Pompeo said that countries must be held accountable when they violate a treaty. “Russia has jeopardized the United States’ security interests,” he said, “and we can no longer be restricted by the treaty while Russia shamelessly violates it.” Pompeo continued by noting that Russia’s posturing is a clear signal that the nation is returning to its old Cold War mentality, and that the U.S. must make similar preparations in light of these developments. “As we remain hopeful of a fundamental shift in Russia’s posture, the United States will continue to do what is best for our people and those of our allies,” he concluded.

The controversy about whether Russia is in violation hinges on whether the 9M729 missile can fly more than 500km. The U.S. claims to have provided evidence of this to Russia, but has not made this evidence public, and further claims that violations have continued since at least 2014. Although none of the U.S.-based policy experts interviewed for this article dispute that Russia is in violation, many caution that this suspension will create a far more unstable environment and that the U.S. shares much of the blame for not doing more to preserve the treaty.

In an emailed statement to the Future of Life Institute, Martin Hellman, an Adjunct Senior Fellow for Nuclear Risk Analysis at the Federation of American Scientists and Professor Emeritus of Electrical Engineering at Stanford University, was clear in his censure of the Trump administration’s decision and reasoning, noting that it follows a well-established pattern of duplicity and double-dealing:

The INF Treaty was a crucial step in ending the arms race. Our withdrawing from it in such a precipitous manner is a grave mistake. In a sense, treaties are the beginning of negotiations, not the end. When differences in perspective arise, including on what constitutes a violation, the first step is to meet and negotiate. Only if that process fails, should withdrawal be contemplated. In the same way, any faults in a treaty should first be approached via corrective negotiations.

Withdrawing in this precipitous manner from the INF treaty will add to concerns that our adversaries already have about our trustworthiness on future agreements, such as North Korea’s potential nuclear disarmament. Earlier actions of ours which laid that foundation of mistrust include George W. Bush killing the 1994 Agreed Framework with North Korea “for domestic political reasons,” Obama attacking Libya after Bush had promised that giving up its WMD programs “can regain a secure and respected place among the nations,” and Trump tearing up the Iran agreement even though Iran was in compliance and had taken steps that considerably set back its nuclear program.

In an article published by CNN, Eliot Engel, chairman of the House Committee on Foreign Affairs, and Adam Smith, chairman of the House Committee on Armed Services, echo these sentiments and add that the U.S. government greatly contributed to the erosion of the treaty, clarifying that the suspension could have been avoided if President Trump had collaborated with NATO allies to pressure Russia into ensuring compliance. “ allies told our offices directly that the Trump administration blocked NATO discussion regarding the INF treaty and provided only the sparest information throughout the process….This is the latest step in the Trump administration’s pattern of abandoning the diplomatic tools that have prevented nuclear war for 70 years. It also follows the administration’s unilateral decision to withdraw from the Paris climate agreement,” they said.

Russia has also complained about the alleged lack of U.S. diplomacy. In January 2019, Russian diplomats proposed a path to resolution, stating that they would display their missile system and demonstrate that it didn’t violate the INF treaty if the U.S. did the same with their MK-41 launchers in Romania. The Russians felt that this was a fair compromise, as they have long argued that the Aegis missile defense system, which the U.S. deployed in Romania and Poland, violates the INF treaty. The U.S. rejected Russia’s offer, stating that a Russian controlled inspection would not permit the kind of unfettered access that U.S. representatives would need to verify their conclusions. And ultimately, they insisted that the only path forward was for Russia to destroy the missiles, launchers, and supporting infrastructure.

In response, Russian foreign minister Sergei Lavrov accused the U.S. of being obstinate. “U.S. representatives arrived with a prepared position that was based on an ultimatum and centered on a demand for us to destroy this rocket, its launchers and all related equipment under US supervision,” he said.

Suggested Reading

Accidental Nuclear War: A Timeline of Close Calls

The most devastating military threat arguably comes from a nuclear war started not intentionally but by accident or miscalculation. Accidental nuclear war has almost happened many times already, and with 15,000 nuclear weapons worldwide — thousands on hair-trigger alert and ready to launch at a moment’s notice — an accident is bound to occur eventually.

The Blame Game Part 2: China

Other experts, such as Mark Fitzpatrick, Director of the non-proliferation program at the International Institute for Strategic Studies, assert that the “real reason” for the U.S. pullout lies elsewhere — in China.

This belief is bolstered by previous statements made by President Trump. Most notably, during a rally in the Fall of 2018, the President told reporters that it is unfair that China faces no limits when it comes to developing and deploying intermediate-range nuclear missiles. “Unless Russia comes to us and China comes to us and they all come to us and say, ‘let’s really get smart and let’s none of us develop those weapons, but if Russia’s doing it and if China’s doing it, and we’re adhering to the agreement, that’s unacceptable,” he said.

According to a 2019 report published for congress, China has some 2,000 ballistic and cruise missiles in its inventory, and 95% of these would violate the INF treaty if Beijing were a signatory. It should be noted that both Russia and the U.S. are estimated to have over 6,000 nuclear warheads, while China has approximately 280. Nevertheless, the report states, “The sheer number of Chinese missiles and the speed with which they could be fired constitutes a critical Chinese military advantage that would prove difficult for a regional ally or partner to manage absent intervention by the United States,” adding, “The Chinese government has also officially stated its opposition to Beijing joining the INF Treaty.” Consequently, President Trump stated that the U.S. has no choice but to suspend the treaty.

Along these lines, John Bolton, who became the National Security Adviser in April, has long argued that the kinds of missiles banned by the INF treaty would be an invaluable resource when it comes to defending Western nations against what he argues is an increasing military threat from China’s.

Pranay Vaddi, a fellow in the Nuclear Policy Program at the Carnegie Endowment for International Peace, feels differently. Although he does not deny that China poses a serious military challenge to the U.S., Vaddi asserts that withdrawing from the INF treaty is not a viable solution, and he says that proponents of the suspension “ignore the very real political challenges associated with deploying U.S. GBIRs in the Asia Pacific region. They also ignore specific military challenges, including the potential for a missile race and long-term regional and strategic instability.” He concludes, “Before withdrawing from the INF Treaty, the United States should consult with its Asian allies on the threat posed by China, the defenses required, and the consequences of introducing U.S. offensive missiles into the region, including potentially on allied territory.”

Suggested Reading

1100 Declassified U.S. Nuclear Targets

The National Security Archives recently published a declassified list of U.S. nuclear targets from 1956, which spanned 1,100 locations across Eastern Europe, Russia, China, and North Korea. This map shows all 1,100 nuclear targets from that list, demonstrating how catastrophic a nuclear exchange between the United States and Russia could be.

Six Months and Counting

Regardless of how much blame each respective nation shares, the present course has been set, and if things don’t change soon, we may find ourselves in a very different world a few months from now.

According to the terms of the treaty, if one of the parties breaches the agreement then the other party has the option to terminate or suspend it. It was on this basis that, back in October of 2018, President Trump stated he would be terminating the INF treaty altogether. Today’s suspension announcement is an update to these plans.

Notably, a suspension doesn’t follow the same course as a withdrawal. A suspension means that the treaty continues to exist for a set period. As a result, starting Feb. 1, the U.S. began a six-month notice period. If the two nations don’t reach an agreement and decide to restore the treaty within this window, on August 2nd, the Treaty will go out of effect. At that juncture, both the U.S. and Russia will be free to develop and deploy the previously banned nuclear missiles with no oversight or transparency.

The situation is dire, and experts assert that we must immediately reopen negotiations. On Friday, before the official U.S. announcement, German Chancellor Angela Merkel said that if the United States announced it would suspend compliance with the treaty, Germany would use the six-month formal withdrawal period to hold further discussions. “If it does come to a cancellation today, we will do everything possible to use the six-month window to hold further talks,” she said.

Following the US announcement, German Foreign Minister Heiko Maas tweeted, “there will be less security without the treaty.” Likewise, Laura Rockwood, executive director at the Vienna Center for Disarmament and Non-Proliferation, noted that the suspension is a troubling move that will increase — not decrease — tension and conflict. “It would be best to keep the INF in place. You don’t throw the baby out with the bathwater. It’s been an extraordinarily successful arms control treaty,” she said.

Carl Bildt, a co-chair of the European Council on Foreign Relations, agreed with these sentiments, noting in a tweet that the INF treaty’s demise puts many lives in peril. “Russia can now also deploy its Kaliber cruise missiles with ranges around 1.500 km from ground launchers. This would quickly cover all of Europe with an additional threat,” he said.

And it looks like many of these fears are already being realized. In a televised meeting over the weekend, President Putin stated that Russia will actively begin building weapons that were previously banned under the treaty. President Putin also made it clear that none of his departments would initiate talks with the U.S. on any matters related to nuclear arms control. “I suggest that we wait until our partners are ready to engage in equal and meaningful dialogue,” he said.

The photo for this article is from wiki commons: by Mil.ru, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=63633975

As we end 2018 and look toward 2019, we want to focus on a message of hope, a message of existential hope.

But first, a very quick look back…

We had a great year, and we’re pleased with all we were able to accomplish. Some of our bigger projects and successes include: the Lethal Autonomous Weapons Pledge; a new round of AI safety grants focusing on the beneficial development of AGI; the California State Legislature’s resolution in support of the Asilomar AI Principles; and our second Future of Life Award, which was presented posthumously to Stanislav Petrov and his family.

As we now look ahead and strive to work toward a better future, we, as a society, must first determine what that collective future should be. At FLI, we’re looking forward to working with global partners and thought leaders as we consider what “better futures” might look like and how we can work together to build them.

As FLI President Max Tegmark says, “There’s been so much focus on just making our tech powerful right now, because that makes money, and it’s cool, that we’ve neglected the steering and the destination quite a bit. And in fact, I see that as the core goal of the Future of Life Institute: help bring back focus the steering of our technology and the destination.”

A recent Gizmodo article on why we need more utopian fiction also summed up the argument nicely: ​​”Now, as we face a future filled with corruption, yet more conflict, and the looming doom of global warming, imagining our happy ending may be the first step to achieving it.”

Fortunately, there are already quite a few people who have begun considering how a conflicted world of 7.7 billion can unite to create a future that works for all of us. And for the FLI podcast in December, we spoke with six of them to talk about how we can start moving toward that better future.

The existential hope podcast includes interviews with FLI co-founders Max Tegmark and Anthony Aguirre, as well as existentialhope.com founder Allison Duettmann, Josh Clark who hosts The End of the World with Josh Clark, futurist and researcher Anders Sandberg, and tech enthusiast and entrepreneur Gaia Dempsey. You can listen to the full podcast here, but we also wanted to call attention to some of their comments that most spoke to the idea of steering toward a better future:

Max Tegmark on the far future and the near future:

When I look really far into the future, I also look really far into space and I see this vast cosmos, which is 13.8 billion years old. And most of it is, despite what the UFO enthusiasts say, actually looking pretty dead and wasted opportunities. And if we can help life flourish not just on earth, but ultimately throughout much of this amazing universe, making it come alive and teeming with these fascinating and inspiring developments, that makes me feel really, really inspired.

For 2019 I’m looking forward to more constructive collaboration on many aspects of this quest for a good future for everyone on earth.

Gaia Dempsey on how we can use a technique called world building to help envision a better future for everyone and get more voices involved in the discussion:

Worldbuilding is a really fascinating set of techniques. It’s a process that has its roots in narrative fiction. You can think of, for example, the entire complex world that J.R.R. Tolkien created for The Lord of the Rings series. And in more contemporary times, some spectacularly advanced worldbuilding is occurring now in the gaming industry. So [there are] these huge connected systems that underpin worlds in which millions of people today are playing, socializing, buying and selling goods, engaging in an economy. These are vast online worlds that are not just contained on paper as in a book, but are actually embodied in software. And over the last decade, world builders have begun to formally bring these tools outside of the entertainment business, outside of narrative fiction and gaming, film and so on, and really into society and communities. So I really define worldbuilding as a powerful act of creation.

And one of the reasons that it is so powerful is that it really facilitates collaborative creation. It’s a collaborative design practice.

Ultimately our goal is to use this tool to explore how we want to evolve as a society, as a community, and to allow ideas to emerge about what solutions and tools will be needed to adapt to that future.

One of the things where I think worldbuilding is really good is that the practice itself does not impose a single monolithic narrative. It actually encourages a multiplicity of narratives and perspectives that can coexist.

Anthony Aguirre on how we can use technology to find solutions:

I think we can use technology to solve any problem in the sense that I think technology is an extension of our capability: it’s something that we develop in order to accomplish our goals and to bring our will into fruition. So, sort of by definition, when we have goals that we want to do — problems that we want to solve — technology should in principle be part of the solution.

So I’m broadly optimistic that, as it has over and over again, technology will let us do things that we want to do better than we were previously able to do them.

Allison Duettmann on why she created the website existentialhope.com:

I do think that it’s up to everyone, really, to try to engage with the fact that we may not be doomed, and what may be on the other side. What I’m trying to do with the website, at least, is generate common knowledge to catalyze more directed coordination toward beautiful futures. I think that there a lot of projects out there that are really dedicated to identifying the threats to human existence, but very few really offer guidance on to influence that. So I think we should try to map the space of both peril and promise which lie before us, we should really try to aim for that. This knowledge can empower each and every one of us to navigate toward the grand future.

Josh Clark on the impact of learning about existential risks for his podcast series, The End of the World with Josh Clark:

As I was creating the series, I underwent this transition how I saw existential risks, and then ultimately how I saw humanity’s future, how I saw humanity, other people, and I kind of came to love the world a lot more than I did before. Not like I disliked the world or people or anything like that. But I really love people way more than I did before I started out, just because I see that we’re kind of close to the edge here. And so the point of why I made the series kind of underwent this transition, and you can kind of tell in the series itself where it’s like information, information, information. And then now, that you have bought into this, here’s how we do something about it.

I think that one of the first steps to actually taking on existential risks is for more and more people to start talking about [them].

Anders Sandberg on a grand version of existential hope:

The thing is, my hope for the future is we get this enormous open ended future. It’s going to contain strange and frightening things but I also believe that most of it is going to be fantastic. It’s going to be roaring on the world far, far, far into the long term future of the universe probably changing a lot of the aspects of the universe.

When I use the term “existential hope,” I contrast that with existential risk. Existential risks are things that threaten to curtail our entire future, to wipe it out, to make it too much smaller than it could be. Existential hope to me, means that maybe the future is grander than we expect. Maybe we have chances we’ve never seen and I think we are going to be surprised by many things in future and some of them are going to be wonderful surprises. That is the real existential hope.

Right now, this sounds totally utopian, would you expect all humans to get together and agree on something philosophical? That sounds really unlikely. Then again, a few centuries ago the United Nations and the internet would totally absurd. The future is big, we have a lot of centuries ahead of us, hopefully.

From everyone at FLI, we wish you a happy holiday season and a wonderful New Year full of hope!

The UNFCCC defines its ultimate goal as “preventing ‘dangerous’ human interference with the climate system,” and its objective for COP24 was to design an “implementation package” for the 2015 Paris Climate Agreement. This package, known as the Katowice Rules, is intended to bolster the Paris Agreement by intensifying the mitigation goals of each of its member countries and, in so doing, ensure the full implementation of the Paris Agreement.

The significance of this package is clearly articulated in the COP24 presidency’s vision — “there is no Paris Agreement without Katowice.”

And the tone of the event was, fittingly, one of urgency. Negotiations took place in the wake of the latest IPCC report, which made clear in its findings that the original terms of the Paris Agreement are insufficient. If we are to keep to the preferred warming target of 1.5°C this century, the report notes that we must strengthen the global response to climate change.

The need for increased action was reiterated throughout the event. During the first week of talks, the Global Carbon Project released new data showing a 2.7% increase in carbon emissions in 2018 and projecting further emissions growth in 2019. And the second week began with a statement from global investors who, “strongly urge all governments to implement the actions that are needed to achieve the goals of the Agreement, with the utmost urgency.” The investors warned that, without drastic changes, the economic fallout from climate change would likely be several times worse than the 2008 financial crisis.

Against this grim backdrop, negotiations crawled along.

Progress was impeded early on by a disagreement over the wording used in the Conference’s acknowledgment of the IPCC report. Four nations — the U.S., Russia, Saudi Arabia, and Kuwait — took issue with a draft that said the parties “welcome” the report, preferring to say they “took note” of it. A statement from the U.S. State Department explained: “The United States was willing to note the report and express appreciation to the scientists who developed it, but not to welcome it, as that would denote endorsement of the report.”

There was also tension between the U.S. and China surrounding the treatment of developed vs. developing countries. The U.S. wants one universal set of rules to govern emissions reporting, while China has advocated for looser standards for itself and other developing nations.

Initially scheduled to wrap on Friday, talks continued into the weekend, as a resolution was delayed in the final hours by Brazil’s opposition to a proposal that would change rules surrounding carbon trading markets. Unable to strike a compromise, negotiators ultimately tabled the proposal until next year, and a deal was finally struck on Saturday, following negotiations that carried on through the night.

The final text of the Katowice Rules welcomes the “timely completion” of the IPCC report and lays out universal requirements for updating and fulfilling national climate pledges. It holds developed and developing countries to the same reporting standard, but it offers flexibility for “those developing country parties that need it in the light of their capacities.” Developing countries will be left to self-determine whether or not they need flexibility.

The rules also require that countries report any climate financing, and developed countries are called on to increase their financial contributions to climate efforts in developing countries.

The photo for this article was originally posted here.

However, exponential growth comes with its own set of hurdles that must be navigated. The foremost issue is that it’s exceedingly difficult to predict how various technologies will evolve. As a result, it becomes challenging to plan for the future and ensure that the necessary safety features are in place.

This uncertainty is particularly worrisome when it comes to technologies that could pose existential challenges — artificial intelligence, for example.

Yet, despite the unpredictable nature of tomorrow’s AI, certain challenges are foreseeable. Case in point, regardless of the developmental path that AI agents ultimately take, these systems will need to be capable of making intelligent decisions that allow them to move seamlessly and safely through our physical world. Indeed, one of the most impactful uses of artificial intelligence encompasses technologies like autonomous vehicles, robotic surgeons, user-aware smart grids, and aircraft control systems — all of which combine advanced decision-making processes with the physics of motion.

Such systems are known as cyber-physical systems (CPS). The next generation of advanced CPS could lead us into a new era in safety, reducing crashes by 90% and saving the world’s nations hundreds of billions of dollars a year — but only if such systems are themselves implemented correctly.

This is where Andre Platzer, Associate Professor of Computer Science at Carnegie Mellon University, comes in. Platzer’s research is dedicated to ensuring that CPS benefit humanity and don’t cause harm. Practically speaking, this means ensuring that the systems are flexible, reliable, and predictable.

What Does it Mean to Have a Safe System?

Cyber-physical systems have been around, in one form or another, for quite some time. Air traffic control systems, for example, have long relied on CPS-type technology for collision avoidance, traffic management, and a host of other decision-making tasks. However, Platzer notes that as CPS continue to advance, and as they are increasingly required to integrate more complicated automation and learning technologies, it becomes far more difficult to ensure that CPS are making reliable and safe decisions.

To better clarify the nature of the problem, Platzer turns to self-driving vehicles. In advanced systems like these, he notes that we need to ensure that the technology is sophisticated enough to be flexible, as it has to be able to safely respond to any scenario that it confronts. In this sense, “CPS are at their best if they’re not just running very simple , but if they’re running much more sophisticated and advanced systems,” Platzer notes. However, when CPS utilize advanced autonomy, because they are so complex, it becomes far more difficult to prove that they are making systematically sound choices.

In this respect, the more sophisticated the system becomes, the more we are forced to sacrifice some of the predictability and, consequently, the safety of the system. As Platzer articulates, “the simplicity that gives you predictability on the safety side is somewhat at odds with the flexibility that you need to have on the artificial intelligence side.”

The ultimate goal, then, is to find equilibrium between the flexibility and predictability — between the advanced learning technology and the proof of safety — to ensure that CPS can execute their tasks both safely and effectively. Platzer describes this overall objective as a kind of balancing act, noting that, “with cyber-physical systems, in order to make that sophistication feasible and scalable, it’s also important to keep the system as simple as possible.”

How to Make a System Safe

The first step in navigating this issue is to determine how researchers can verify that a CPS is truly safe. In this respect, Platzer notes that his research is driven by this central question: if scientists have a mathematical model for the behavior of something like a self-driving car or an aircraft, and if they have the conviction that all the behaviors of the controller are safe, how do they go about proving that this is actually the case?

The answer is an automated theorem prover, which is a computer program that assists with the development of rigorous mathematical correctness proofs.

When it comes to CPS, the highest safety standard is such a mathematical correctness proof, which shows that the system always produces the correct output for any given input. It does this by using formal methods of mathematics to prove or disprove the correctness of the control algorithms underlying a system.

After this proof technology has been identified and created, Platzer asserts that the next step is to use it to augment the capabilities of artificially intelligent learning agents — increasing their complexity while simultaneously verifying their safety.

Eventually, Platzer hopes that this will culminate in technology that allows CPS to recover from situations where the expected outcome didn’t turn out to be an accurate model of reality. For example, if a self-driving car assumes another car is speeding up when it is actually slowing down, it needs to be able to quickly correct this error and switch to the correct mathematical model of reality.

The more complex such seamless transitions are, the more complex they are to implement. But they are the ultimate amalgamation of safety and flexibility or, in other words, the ultimately combination of AI and safety proof technology.

Creating the Tech of Tomorrow

To date, one of the biggest developments to come from Platzer’s research is the KeYmaera X prover, which Platzer characterizes as a “gigantic, quantum leap in terms of the reliability of our safety technology, passing far beyond in rigor than what anyone else is doing for the analysis of cyber-physical systems.”

The KeYmaera X prover, which was created by Platzer and his team, is a tool that allows users to easily and reliably construct mathematical correctness proofs for CPS through an easy-to-use interface.

More technically, KeYmaera X is a hybrid systems theorem prover that analyzes the control program and the physical behavior of the controlled system together, in order to provide both efficient computation and the necessary support for sophisticated safety proof techniques. Ultimately, this work builds off of a previous iteration of the technology known as KeYmaera. However, Platzer states that, in order to optimize the tool and make it as simple as possible, the team essentially “started from scratch.”

Emphasizing just how dramatic these most recent changes are, Platzer notes that, in the previous prover, the correctness of the statements was dependent on some 66,000 lines of code. Notably, each of these 66,000 lines were all critical to the correctness of the verdict. According to Platzer, this poses a problem, as it’s exceedingly difficult to ensure that all of the lines are implemented correctly. Although the latest iteration of KeYmaera is ultimately just as large as the previous version, in KeYmaera X, the part of the prover that is responsible for verifying the correctness is a mere 2,000 lines of code.

This allows the team to evaluate the safety of cyber-physical systems more reliably than ever before. “We identified this microkernel, this really minuscule part of the system that was responsible for the correctness of the answers, so now we have a much better chance of making sure that we haven’t accidentally snuck any mistakes into the reasoning engines,” Platzer said. Simultaneously, he notes that it enables users to do much more aggressive automation in their analysis. Platzer explains, “If you have a small part of the system that’s responsible for the correctness, then you can do much more liberal automation. It can be much more courageous because there’s an entire safety net underneath it.”

For the next stage of his research, Platzer is going to begin integrating multiple mathematical models that could potentially describe reality into a CPS. To explain these next steps, Platzer returns once more to self-driving cars: “If you’re following another driver, you can’t know if the driver is currently looking for a parking spot, trying to get somewhere quickly, or about to change lanes. So, in principle, under those circumstances, it’s a good idea to have multiple possible models and comply with the ones that may be the best possible explanation of reality.”

Ultimately, the goal is to allow the CPS to increase their flexibility and complexity by switching between these multiple models as they become more or less likely explanations of reality. “The world is a complicated place,” Platzer explains, “so the safety analysis of the world will also have to be a complicated one.”

Facial analysis technology isn’t just used by our smart phones and on social media. It’s also found in drones and other military weapons, and it’s used by law enforcement, airports and airlines, public surveillance cameras, schools, business, and more. Yet the technology is known to be flawed and biased, often miscategorizing anyone who isn’t a white male. And the bias is especially strong against dark-skinned women.

“Research shows facial analysis technology is susceptible to bias and even if accurate can be used in ways that breach civil liberties. Without bans on harmful use cases, regulation, and public oversight, this technology can be readily weaponized, employed in secret government surveillance, and abused in law enforcement,” warns Buolamwini.

By signing the pledge, companies that develop, sell or buy facial recognition and analysis technology promise that they will “prohibit lethal use of the technology, lawless police use, and require transparency in any government use.”

FLI does not develop or use these technologies, but we signed because we support these efforts, and we hope all companies will take necessary steps to ensure their technologies are used for good, rather than as weapons or other means of harm.

Companies that had signed the pledge at launch include Simprints, Yoti, and Robbie AI. Other early signatories of the pledge include prominent AI researchers Noel Sharkey, Subbarao Kambhampati, Toby Walsh, Stuart Russell, and Raja Chatila, as well as tech bauthors Cathy O’Neil and Meredith Broussard, and many more.

The SAFE Face Pledge commits signatories to:

Show Value for Human Life, Dignity, and Rights

• Do not contribute to applications that risk human life
• Do not facilitate secret and discriminatory government surveillance
• Mitigate law enforcement abuse
• Ensure your rules are being followed

Address Harmful Bias

• Implement internal bias evaluation processes and support independent evaluation
• Submit models on the market for benchmark evaluation where available

Facilitate Transparency

• Increase public awareness of facial analysis technology use
• Enable external analysis of facial analysis technology on the market

Embed Safe Face Pledge into Business Practices

• Modify legal documents to reflect value for human life, dignity, and rights
• Engage with stakeholders
• Provide details of Safe Face Pledge implementation

Organizers of the pledge say, “Among the most concerning uses of facial analysis technology involve the bolstering of mass surveillance, the weaponization of AI, and harmful discrimination in law enforcement contexts.” And the first statement of the pledge calls on signatories to ensure their facial analysis tools are not used “to locate or identify targets in operations where lethal force may be used or is contemplated.”

Anthony Aguirre, cofounder of FLI, said, “A great majority of AI researchers agree that designers and builders of AI systems are stakeholders in the moral implications of their use, misuse, and actions, with a responsibility and opportunity to shape those implications.  That is, in fact, the 9th Asilomar AI principle. The Safe Face Pledge asks those involved with the development of facial recognition technologies, which are dramatically increasing in power through the use of advanced machine learning, to take this belief seriously and to act on it.  As new technologies are developed and poised for widespread implementation and use, it is imperative for our society to consider their interplay with the rights and privileges of the people they affect — and new rights and responsibilities may have to be considered as well, where technologies are currently in a legal or regulatory grey area.  FLI applauds the multiple initiatives, including this pledge, aimed at ensuring that facial recognition technologies — as with other AI technologies — are implemented only in a way that benefits both individuals and society while taking utmost care to respect individuals’ rights and human dignity.”

You can support the Safe Face Pledge by signing here.

By Jolene Creighton

Algorithms don’t just decide what posts you see in your Facebook newsfeed. They make millions of life-altering decisions every day. They help decide who moves to the next stage of a job interview, who can take out a loan, and even who’s granted parole.

When one stops to consider the well-known biases that exist in these algorithms, the role that they play in our decision-making processes becomes somewhat concerning.

Ultimately, bias is a problem that stems from the unrepresentative datasets that our systems are trained on. For example, when it comes to images, most of the training data is Western-centric — it depicts caucasian individuals taking part in traditionally Western activities. Consequently, as Google research previously revealed, if we give an AI system an image of a caucasian bride in a Western dress, it correctly labels the image as “wedding,” “bride,” and “women.” If, however, we present the same AI system with an image of a bride of Asian descent, is produces results like “clothing,” “event,” and “performance art.”

Of course, this problem is not exclusively a Western one. In 2011, a study found that AI developed in Eastern Asia have more difficulty distinguishing between Caucasian faces than Asian faces.

That’s why, in September of 2018, Google partnered with the NeurIPS confrence to launch the Inclusive Images Competition, an event that was created to help encourage the development of less biased AI image classification models.

For the competition, individuals were asked to use Open Images, a image dataset collected from North America and Europe, to train a system that can be evaluated on images collected from a different geographic region.

At this week’s NeurIPS conference, Pallavi Baljekar, a Google Brain researcher, spoke about the success of the project. Notably, the competition was only marginally successful. Although the leading models maintained relatively high accuracy in the first stages of the competition, four out of five top models didn’t predict the “bride” label when applied to the original two bride images.

However, that’s not to say that progress wasn’t made. Baljekar noted that the competition proved that, even with a small and diverse set of data, “we can improve performance on unseen target distributions.”

And in an interview, Pavel Ostyakov, a Deep Learning Engineer at Samsung AI Center and the researcher who took first place in the competition, added that demanding an entirely unbiased AI may be asking for a bit too much.  Ultimately, our AI need to be able to “stereotype” to some degree in order to make their classifications. “The problem was not solved yet, but I believe that it is impossible for neural networks to make unbiased predictions,” he said. Ultimately, the need to retain some biases are sentiments that have been echoed by other AI researchers before.

Consequently, it seems that making unbiased AI systems is going to be a process that requires continuous improvement and tweaking. Yet, despite the fact that we can’t make entirely unbiased AI, we can do a lot more to make them less biased.

With this in mind, today, Google announced Open Images Extended. It’s an extension of Google’s Open Images and is intended to be a dataset that better represents the global diversity we find on our planet. The first set to be added is seeded with over 470,000 images.

On this very long road we’re traveling, it’s a step in the right direction.

The Reproducibility Problem: AI Agents Should be Trained in More Realistic Environments

By Jolene Creighton

Our world is a complex and vibrant place. It’s also remarkably dynamic, existing in a state of near constant change. As a result, when we’re faced with a decision, there are thousands of variables that must be considered.

According to Joelle Pineau, an Associate Professor at McGill University and lead of Facebook’s Artificial Intelligence Research lab in Montreal, this poses a bit of a problem when it comes to our AI agents.

During her keynote speech at the 2018 NeurIPS conference, Pineau stated that many AI researchers aren’t training their machine learning systems in proper environments. Instead of using dynamic worlds that mimic what we see in real life, much of the work that’s currently being done takes place in simulated worlds that are static and pristine, lacking the complexity of realistic environments.

According to Pineau, although these computer-constructed worlds help make research more reproducible, they also make the results less rigorous and meaningful. “The real world has incredible complexity, and when we go to these simulators, that complexity is completely lost,” she said.

Pineau continued by noting that, if we hope to one day create intelligent machines that are able to work and react like humans — artificial general intelligences (AGIs) — we must go beyond the static and limited worlds that are created by computers and begin tackling real world scenarios. “We have to break out of these simulators…on the roadmap to AGI, this is only the beginning,” she said.

Ultimately, Pineau also noted that we will never achieve a true AGI unless we begin testing our systems on more diverse training sets and forcing our intelligent agents to tackle more complex problems. “The world is your test set,” she said, concluding, “I’m here to encourage you to explore the full spectrum of opportunities…this means using separate tasks for training and testing.”

Teaching a Machine to Reason

Pineau’s primary critique was on an area of machine learning that is known as reinforcement learning (RL). RL systems allow intelligent agents to improve their decision-making capabilities through trial and error. Over time, these agents are able to learn the rules that govern good and bad choices by interacting with their environment and receiving numerical reward signals that are based on the actions that they take.

Ultimately, RL systems are trained to maximize the numerical reward signals that they receive, so their decisions improve as they try more things and discover what actions yield the most reward. But unfortunately, most simulated worlds have a very limited number of variables. As a result, RL systems have very few things that they can interact with. This means that, although intelligent agents may know what constitutes good decision-making in a simulated environment, when they’re deployed in a realistic environment, they quickly become lost amidst all the new variables.

According to Pineau, overcoming this issue means creating more dynamic environments for AI systems to train on.

To showcase one way of accomplishing this, Pineau turned to Breakout, a game launched by Atari in 1976. The game’s environment is simplistic and static, consisting of a background that is entirely black. In order to inject more complexity into this simulated environment, Pineau and her team inserted videos, which are an endless source of natural noise, into the background.

Pineau argued that, by adding these videos into the equation, the team was able to create an environment that includes some of the complexity and variability of the real world. And by ultimately training reinforcement learning systems to operate in such multifaceted environments, researchers obtain more reliable findings and better prepare RL systems to make decisions in the real world.

In order to help researchers better comprehend exactly how reliable and reproducible their results currently are — or aren’t — Pineau pointed to The 2019 ICLR Reproducibility Challenge during her closing remarks.

The goal of this challenge is to have members of the research community try to reproduce the empirical results submitted to the International Conference on Learning Representations. Then, once all of the attempts have been made, the results are sent back to the original authors. Pineau noted that, to date, the challenge has had a dramatic impact on the findings that are reported. During the 2018 challenge, 80% of authors that received reproducibility reports stated that they changed their papers as a result of the feedback.

You can download a copy of Pineau’s slides here.

Montreal Declaration on Responsible AI May Be Next Step Toward the Development of AI Policy

By Ariel Conn

Over the last few years, as concerns surrounding artificial intelligence have grown, an increasing number of organizations, companies, and researchers have come together to create and support principles that could help guide the development of beneficial AI. With FLI’s Asilomar Principles, IEEE’s treatise on the Ethics of Autonomous and Intelligent Systems, the Partnership on AI’s Tenets, and many more, concerned AI researchers and developers have laid out a framework of ethics that almost everyone can agree upon. However, these previous documents weren’t specifically written to inform and direct AI policy and regulations.

On December 4, at the NeurIPS conference in Montreal, Canadian researchers took the next step, releasing the Montreal Declaration on Responsible AI. The Declaration builds on the current ethical framework of AI, but the architects of the document also add, “Although these are ethical principles, they can be translated into political language and interpreted in legal fashion.”

Yoshua Bengio, a prominent Canadian AI researcher and founder of one of the world’s premiere machine learning labs, described the Declaration saying, “Its goal is to establish a certain number of principles that would form the basis of the adoption of new rules and laws to ensure AI is developed in a socially responsible manner.”

“We want this Declaration to spark a broad dialogue between the public, the experts and government decision-makers,” said UdeM’s rector, Guy Breton. “The theme of artificial intelligence will progressively affect all sectors of society and we must have guidelines, starting now, that will frame its development so that it adheres to our human values ​​and brings true social progress.”

The Declaration lays out ten principles: Well-Being, Respect for Autonomy, Protection of Privacy and Intimacy, Solidarity, Democratic Participation, Equity, Diversity, Prudence, Responsibility, and Sustainable Development.

The primary themes running through the Declaration revolve around ensuring that AI doesn’t disrupt basic human and civil rights and that it enhances equality, privacy, diversity, and human relationships. The Declaration also suggests that humans need to be held responsible for the actions of artificial intelligence systems (AIS), and it specifically states that AIS cannot be allowed to make the decision to take a human life. It also includes a section on ensuring that AIS is designed with the climate and environment in mind, such that resources are sustainably sourced and energy use is minimized.

The Declaration is the result of deliberation that “occurred through consultations held over three months, in 15 different public spaces, and sparked exchanges between over 500 citizens, experts and stakeholders from every horizon.” That it was formulated in Canada is especially relevant given Montreal’s global prominence in AI research.

In his article for the Conversation, Bengio explains, “Because Canada is a scientific leader in AI, it was one of the first countries to see all its potential and to develop a national plan. It also has the will to play the role of social leader.”

He adds, “Generally speaking, scientists tend to avoid getting too involved in politics. But when there are issues that concern them and that will have a major impact on society, they must assume their responsibility and become part of the debate.”

Making an Impact: What Role Should Scientists Play in Creating AI Policy?

By Jolene Creighton

Artificially intelligent systems are already among us. They fly our planes, drive our cars, and even help doctors make diagnoses and treatment plans. As AI continues to impact daily life and alter society, laws and policies will increasingly have to take it into account. Each day, more and more of the world’s experts call on policymakers to establish clear, international guidelines for the governance of AI.

This week, at the 2018 NeurIPS conference, Edward W. Felten, Professor of Computer Science and Public Affairs at Princeton University, took up the call.

During his opening remarks, Felten noted that AI is poised to radically change everything about the way we live and work, stating that this technology is “extremely powerful and represents a profound change that will happen across many different areas of life.” As such, Felten noted that we must work quickly to amend our laws and update our policies so we’re ready to confront the changes that this new technology brings.

However, Felten argued that policy makers cannot be left to dictate this course alone — members of the AI research community must engage with them.

“Sometimes it seems like our world, the world of the research lab or the developer’s or data scientist’s cubicle, is a million miles from public policy…however, we have not only an opportunity but also a duty to be actively participating in public life,” he said.

Guidelines for Effective Engagement

Felton noted that the first step for researchers is to focus on and understand the political system as a whole. “If you look only at the local picture, it might look irrational. But, in fact, these people are operating inside a system that is big and complicated,” he said. To this point, Felten stated that researchers must become better informed about political processes so that they can participate in policy conversations more effectively.

According to Felten, this means the AI community needs to recognize that policy work is valid and valuable, and this work should be incentivized accordingly. He also called on the AI community to create career paths that encourage researchers to actively engage with policymakers by blending AI research and policy work.

For researchers who are interested in pursuing such work, Felten outlined the steps they should take to start an effective dialogue:

1. Combine knowledge with preference: As a researcher, work to frame your expertise in the context of the policymaker’s interests.
2. Structure the decision space: Based on the policymaker’s preferences, give a range of options and explain their possible consequences.
3. Follow-up: Seek feedback on the utility of the guidance that you offered and the way that you presented your ideas.

If done right, Felton said, this protocol allows experts and policy makers to build productive engagement and trust over time.