As frontier artificial intelligence (AI) systems become more capable, it becomes more important that developers can explain why their systems are sufficiently safe. One way to do so is via safety cases: reports that...

This white paper investigates practical remedies to increase voice in and access to AI governance and capabilities for the Global AI Majority, while addressing the security and commercial concerns of frontier AI states.

This paper examines the profound challenges that transformative advances in AI towards Artificial General Intelligence (AGI) will pose for economists and economic policymakers.

Over the past year, artificial intelligence (AI) companies have been increasingly adopting AI safety frameworks. These frameworks outline how companies intend to keep the potential risks associated with developing and deploying...

Regulators in the US and EU are using thresholds based on training compute--the number of computational operations used in training--to identify general-purpose artificial intelligence (GPAI) models that may pose risks of...

AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical...

Several jurisdictions are starting to regulate frontier artificial intelligence (AI) systems, i.e. general-purpose AI systems that match or exceed the capabilities present in the most advanced systems. To reduce risks...

Frontier artificial intelligence (AI) systems could pose increasing risks to public safety and security. But what level of risk is acceptable? One increasingly popular approach is to...

At the core of the financial system is the processing and aggregation of vast amounts of information into price signals that coordinate participants in the economy. Throughout history, advances in information processing...

We propose a framework for evaluating the potential impacts of large-language models (LLMs) and associated technologies on work by considering their relevance to the tasks workers perform in their jobs.

Matthew van der Merwe, Ketan Ramakrishnan, and Markus Anderljung recently published this piece on Lawfare.

Existing strategies for managing risks from advanced AI systems often focus on affecting what AI systems are developed and how they diffuse. This approach becomes less feasible as the number of developers of advanced AI grows.

We analyze how output and wages behave under different scenarios for technological progress that may culminate in Artificial General Intelligence (AGI), defined as the ability of AI systems to perform all tasks that humans can...

Compute providers can play an essential role in a regulatory ecosystem via four key capacities: as securers, safeguarding AI systems and critical infrastructure; as record keepers, enhancing visibility for policymakers...

Recent AI progress has largely been driven by increases in the amount of computing power used to train new models. Governing compute could be an effective way to achieve AI policy goals, but could also introduce new risks.

We welcome the opportunity to respond to the Request for Information (RFI) Related to NIST’s Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning AI. We offer the following submission for your...

Training advanced AI models requires large investments in computational resources, or compute. Yet, as hardware innovation reduces the price of compute and algorithmic advances make its use more efficient, the cost of training...

To deal with the risks of these high-compute frontier AI systems, we must govern not only how they can be used but also how they are developed and made available to people in the first place.

With the increasing integration of frontier large language models (LLMs) into society and the economy, decisions related to their training, deployment, and use have far-reaching implications...

In this written testimony to the U.S. Senate's bipartisan AI Insights Forum, GovAI Economics of AI Lead Anton Korinek encourages lawmakers to engage in scenario planning to prepare the workforce for advances in AI.

Recent releases of frontier artificial intelligence (AI) models have largely been gated, providing the benefit of limiting the proliferation of increasingly powerful dual-use capabilities. However, such release strategies...

Organizations that develop and deploy artificial intelligence (AI) systems need to manage the associated risks—for economic, legal, and ethical reasons. However, it is not always clear who is responsible for AI risk management.

To address security and safety risks stemming from highly capable artificial intelligence (AI) models, we propose that the US government should ensure compute providers implement Know-Your-Customer (KYC) schemes.

This paper proposes an evaluation-based coordination scheme for situations in which frontier AI developers discover that their models have certain dangerous capabilities.

We evaluate the risks and benefits of open-sourcing, as well as alternative methods for pursuing open-source objectives.

We analyse the structure of the market for foundation models, i.e., large AI models such as those that power ChatGPT and that are adaptable to downstream uses, and we examine the implications for competition policy and regulation.

This report describes trade-offs in the design of international governance arrangements for civilian artificial intelligence (AI) and presents one approach in detail.

Markus Anderljung and Paul Scharre recently published this piece in Foreign Affairs

We welcome the opportunity to respond to the OSTP Request for Information on National Priorities for Artificial Intelligence and look forward to future opportunities to provide additional input. We offer the following...

There are increasing concerns that AGI would pose catastrophic risks. In light of this, AGI companies need to drastically improve their risk management practices. This paper reviews popular risk assessment techniques...

Advanced AI models hold the promise of tremendous benefits for humanity, but society needs to proactively manage the accompanying risks. In this paper, we focus on what we term “frontier AI” models — highly capable...

We welcome the opportunity to respond to the NTIA’s AI Accountability Policy Request for Comment and look forward to future opportunities to provide additional input. We offer the following...

Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing...

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as...

A number of leading AI companies, including OpenAI, Google DeepMind, andAnthropic, have the stated goal of building artificial general intelligence (AGI)—AI systems that achieve or exceed human performance...

This study investigates how LLMs can be used for spear phishing, a form of cybercrime that involves manipulating targets into divulging sensitive information. It first explores LLMs’ ability to assist with the reconnaissance...

As large-scale pre-trained AI models gain popularity in the West, many Chinese AI labs have developed their own models capable of generating coherent text and realistic images and videos. These models represent the frontier...

Organizations that develop and deploy artificial intelligence (AI) systems need to take measures to reduce the associated risks. In this paper, we examine how AI companies could design an AI ethics board in a way that reduces...

We are pleased to see the publication of the UK’s Future of Compute Review. However, we also believe there is a significant missed opportunity: the review does not address how to ensure that compute is used responsibly or how...

This paper outlines four different notions of “AI democratisation”, three of which are used almost synonymously with “increasing accessibility”. The democratisation of AI use and the democratisation of AI development are about...

The development of privacy-enhancing technologies has made immense progress in reducing trade-offs between privacy and performance in data exchange and analysis. Similar tools could be useful for AI governance...

Artificial intelligence (AI) systems will increasingly be used to cause harm as they grow more capable. In fact, AI systems are already starting to be used to automate fraudulent activities, violate human rights, create...

The proposed EU AI Act is the first comprehensive attempt to regulate AI in a major jurisdiction. This article analyses Article 9, the key risk management provision in the AI Act. It gives an overview of the regulatory...

Stuart Russell has had a seminal influence on many people with his writings on aligning artificial intelligence with human values and regulating autonomous weapons systems. His work has highlighted the difficulties...

This web app is a tool for exploring the dynamics of risky AI competition: the safety-performance tradeoff. Will AI safety breakthroughs always lead to safer AI systems? Before long, we may be capable of creating AI systems...

This report summarises the most important aspects of the development of atomic weapons and draws out a number of important insights for the development of similarly important technologies.

The great powers appear to be entering an era of heightened competition to master security-relevant technologies in areas such as AI. This is concerning because deploying new technologies can create substantial...

In this report, we ask whether the EU’s upcoming regulation for AI will diffuse globally, producing a so-called “Brussels Effect”.

This Brookings working paper presents an analysis of how institutions could be set up to prepare for a scenario of increasingly smart autonomous machines which could replace human labor and...

We welcome the opportunity to respond to the Future of Compute Review’s call for evidence...Our response focuses on the future of compute used for Artificial Intelligence (AI). In particular, we emphasise the risks posed by...

In recent decades, governments have been ineffectual at regulating dangerous emerging technologies like lethal autonomous weapons and synthetic biology. In today’s era of great power competition...

The world appears to be entering an era of heightened great power technological competition in areas such as artificial intelligence. This is concerning because deploying new technologies often involves private benefits and...

This report gives comments on the interim report of the National AI Research Resource (NAIRR) Task Force. The key recommendations are: Provides researchers with access to pre-trained models by providing infrastructure...

Advances in artificial intelligence (AI) are shaping modern life, from transportation, health care, science, finance, to the military. Forecasts of AI development could help improve policy- and decision making. We report...

We study how the information environment affects races to implement a powerful new technology such as advanced artificial intelligence. In particular, we analyse a model in which a potentially unsafe technology may cause...

The recent news of Elon Musk attempting to take over Twitter has put a spotlight on the implications of free speech in the age of social media. The decisions made by these companies as to what content is...

Ethics statements have been proposed as a mechanism to increase transparency and promote reflection on the societal impacts of published research. In 2020, the machine learning (ML) conference NeurIPS broke new ground by...

The development of lethal autonomous weapons systems (LAWS) is no longer just a concept of science fiction. With the potential of these weapons to cause destruction, there is an urgent need for an international approach to...

In this Brookings working paper, Korinek and Balwit discuss the AI alignment problem – how to ensure that AI systems pursue the goals that we want them to pursue. This article distinguishes two types of alignment problems...

This report gives comments on the Initial Draft of the NIST AI Risk Management Framework (AI RMF). The key recommendations are to put more emphasis on low-probability, high-impact risks, especially catastrophic risks to...

This DPhil dissertation is about attempts to govern how artificial intelligence (AI) researchers share their work. There is growing concern that the software artefacts built by AI researchers will have adverse impacts on society..

The study of international races has focused almost exclusively on security motivations for competitive arming. But international races may also be motivated by prestige. This article defines a “prestige race” and outlines the...

The range of application of artificial intelligence (AI) is vast, as is the potential for harm. Growing awareness of potential risks from AI systems has spurred action to address those risks while eroding confidence in AI...

With the development of ever more advanced artificial intelligence (AI) systems, some of the world’s leading scientists, AI engineers and businesspeople have expressed concerns that humanity may lose control over its creations...

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for...

What makes coercion succeed? For most international relations scholars, the answer is credible threats. Yet scholars have neglected a second key component of successful coercion: credible assurances. This article makes two...

Out of the wreckage of the Second World War, the UK transformed itself. It rebuilt its shattered economy. It founded the NHS. It created national insurance. And it helped establish international institutions like the United...

Advances in artificial intelligence and automation have the potential to be labor-saving and to increase inequality and poverty around the globe. They also give rise to winner-takes-all dynamics th...

Much AI governance work involves preparation for a constitutional moment: an opportunity to create long-lasting, decision-shaping, institutions. Doing this well is a formidable task. It requires a fine balance. Institutions...

Major theories of military innovation focus on relatively narrow technological developments, such as nuclear weapons or aircraft carriers. Arguably the most profound military implications of technological change, however, come...

What resources and technologies are strategic? Policy and theoretical debates often focus on this question, since the “strategic” designation yields valuable resources and elevated attention. The ambiguity of the very concept...

A large literature has argued that offensive advantage makes states worse off because it can induce a security dilemma, preemption, costly conflict, and arms races. We argue instead that state welfare is u-shaped under...

Historically, progress in the field of cryptography has been enormously consequential. Over the past century, for instance, cryptographic discoveries have played a key role in a world war and made it possible to use the internet..

Future advances in AI that automate away human labor may have stark implications for labor markets and inequality. This paper proposes a framework to analyze the effects of specific types of AI systems on the labor market...

Machine learning (ML) and artificial intelligence (AI) researchers play an important role in the ethics and governance of AI, including taking action against what they perceive to be unethical uses of AI...

Countries, companies, and universities are increasingly competing over top-tier artificial intelligence (AI) researchers. Where are these researchers likely to immigrate and what affects their immigration decisions? We...

To help humanity solve fundamental problems of cooperation, scientists need to reconceive artificial intelligence as deeply social. Artificial-intelligence assistants and recommendation algorithms interact with billions of peop...

This essay examines the growing role of China in international standards-setting organizations as a window into its efforts to influence global digital governance institutions and highlights areas where the U.S. can preserve...

The invention of atomic energy posed a novel global challenge: could the technology be controlled to avoid destructive uses and an existentially dangerous arms race while permitting the broad shari...

The U.S. government (USG) has taken increasing interest in the national security implications of artificial intelligence (AI). In this report, we ask: Given its national security concerns, how migh...

The covid-19 pandemic has necessitated interventions that reduce physical contact among people, with dire effects on our economy. By some estimates, a quarter of all jobs in the economy require physical interaction and are thus...

Reputations for resolve are said to be one of the few things worth fighting for, yet they remain inadequately understood. Discussions of reputation focus almost exclusively on first-order belief change—A stands firm, B updates...

Threats and force, by increasing expected costs, should reduce the target’s resolve. However, they often seem to increase resolve. We label this phenomenon provocation. We review instances of apparent provocation in interstate...

Turning principles into practice is one of the most pressing challenges of artificial intelligence (AI) governance. In this Perspective, we reflect on a governance initiative by one of the world’s ...

At least since Herbert Simon’s (1960) prediction that artificial intelligence would soon replace all human labor, many economists have understood that there is a possibility that sooner or later ar...

Progress in artificial intelligence and related forms of automation technologies threatens to reverse the gains that developing countries and emerging markets have experienced from integrating into the world economy over the...

Artificial intelligence (AI) talent is global. AI researchers and engineers come from, and are in high demand, all over theworld. Countries and companies trying to recruit and retain AI talent thus...

Arguably, the success of the human species is rooted in our ability to cooperate. Since machines powered by artificial intelligence are playing an ever greater role in our lives, it will be importa...

Many socially valuable activities depend on sensitive information, such as medical research, public health policies, political coordination, and personalized digital services. This is often posed a...

Biology can be misused, and the risk of this causing widespread harm increases in step with the rapid march of technological progress. A key security challenge involves attribution: determining, in...

Preferences for conflict and cooperation are systematically different for men and women: across a variety of contexts, women generally prefer more peaceful options and are less supportive of making threats and initiating...

Advances in AI are likely to be among the most impactful global developments in the coming decades, and that AI governance will become among the most important global issue areas. AI governance is ...

Artificial Intelligence (AI)—like past general purpose technologies such as railways, the internet, and electricity—is likely to have significant effects on both national security and market struct...

In this response, we focus our analysis and recommendations on the proposed “ecosystem of trust” and associated international efforts. We believe these measures can mitigate the risks that this tec...

Over time, the exercise of assessing impact could enhance the ML community’s expertise in technology governance, and otherwise help build bridges to other researchers and policymakers. To help maxi...

Governments around the world are busy talking about the critical next steps — how to keep people safe from the coronavirus as economies start to reopen. In the United States and in Europe, this involves officials looking at how...

This report suggests various steps that different stakeholders in AI development can take to make it easier to verify claims about AI development, with a focus on providing evidence about the safet...

This working paper is a preliminary analysis of the legal rules, norms, and strategies governing artificial intelligence (AI)-related intellectual property (IP). We analyze the existing AI-related ...

One way of carving up the broad ‘AI ethics and society ’ research space that has emerged in recent years is to distinguish between ‘near-term’ and ‘long-term’ research. While such ways of breaking ...

Can effective international governance for artificial intelligence remain fragmented, or is there a need for a centralised international organisation for AI? We draw on the history of other internat...