How to ensure AI safety incidents are identified, investigated, and used to improve safety across the industry.

Developers rely on model-level, deployment-level, and governance-level interventions to address extreme misuse risks.

Policymakers should complement biosecurity measures with stronger biosafety systems.

As AI agents make government services easier to access, governments need to balance measures against surges in demand against the risk of reducing equitable access.

The US military faces organizational, technical, and operational barriers to adopting AI at scale.

A cyberattack of such magnitude would require a blackout affecting around 100 million people for roughly a week – four orders of magnitude more disruption than any grid cyberattack recorded to date.

Scalable insurance for AI agents is will require coordinated industry infrastructure for pricing and managing rapidly evolving risks.

A risk model for AI helping threat actors to develop data-damaging worms.

In recent years, frontier AI companies have sometimes delayed the release of new models to the EU and UK – or not released them at all. To what extent has...

Frontier AI developers increasingly rely on safeguards to limit the biological misuse risk of their models. However, there is no agreed way to...

Frontier AI companies use offline monitoring to address risks from internally deployed AI agents. AI developers increasingly rely on...

Frontier AI governance often centres on the model-level governance paradigm, which assumes...

Generative AI is set to transform the legal profession, but its full impact remains uncertain. While AI models like GPT-4 improve...

The EU GPAI Code of Practice commits Signatories to providing a description of intended model behavior...

Future AI systems could impact the risk of terrorist bomb plots. This technical report analyses two channels by which...

AI companies and governments have raised concerns about frontier AI systems enabling cyber-attacks and cybercrime. Some have expressed interest in...

The automation of AI R&D (AIRDA) could have significant implications, but its extent and ultimate effects remain uncertain. We need...

Large language models (LLMs) perform strongly on biological benchmarks, raising concerns that they may help novice actors acquire.

We outline a vision for frontier AI auditing, which we define as rigorous third-party verification of frontier AI developers' safety and security claims, and evaluation of...

We construct an occupation-level adaptive capacity index that measures a set of worker characteristics relevant for navigating...

American hyperscalers are increasingly exploring building data centers in the UAE, but it is unclear whether doing so is...

Several frontier AI companies test their AI systems for dual-use biological capabilities that might be misused by...

There is deep uncertainty about how significant AI’s workforce effects will be, how quickly they will emerge, and which groups they will affect most. Some economists expect...

The rise of Large Language Model (LLM)-based web agents represents a significant shift in automated interactions with...

The online activities of AI agents could distort human beliefs and behaviors. For example, humans could mistake...

When frontier AI companies assess the risks from their models, they increasingly focus on marginal risk. This aims to measure how much their models increase risk compared to some baseline...

The shift from scaling up the compute used to pre-train AI systems (pre-training compute) to scaling up the amount used to run them (inference compute) may have...

Evaluations of dangerous AI capabilities are important for managing catastrophic risks. Public transparency into these evaluations - including...

Governments around the world have recognised the need to manage risks from...

As AI agents become more widely deployed, we are likely to see an increasing number of incidents: events involving AI agent use that...

The UK has pursued a distinctive path in AI regulation: less cautious than the EU but more willing to address risks than...

Capabilities of large language models (LLMs) on several biological benchmarks have prompted excitement about their usefulness for...

Washington has made it clear that retaining AI dominance over China is both an economic and national security imperative. In November 2024, the U.S.-China Economic and Security Review Commission recommended...

Safety frameworks have emerged as a best practice for managing risks from frontier artificial intelligence (AI) systems. However, it may be difficult...

We extend Eloundou et al. (2024) to build firm-level measures of exposure to large language models (LLMs) with data from two sources...

Governments are starting to impose requirements on AI models based on how much compute was used to train them. For example, the EU AI Act imposes...

The widespread deployment of general-purpose AI (GPAI) systems introduces significant new risks. Yet the infrastructure, practices, and norms for reporting flaws in GPAI systems remain seriously underdeveloped...

Downstream developers - actors who fine-tune or otherwise modify foundational models - can create or amplify risks from foundation models by improving their capabilities or compromising safety features...

Artificial Intelligence (AI) systems have the potential to cause, directly or indirectly, immense harm. They also may immensely improve human welfare. The regulatory challenge is...

AI agents are AI systems that can achieve complex goals autonomously. Assessing the level of agent autonomy is crucial for...

Advanced AI systems could generate substantial economic and other societal benefits, but these benefits may not be widely shared by default. For a range of reasons, a number of prominent actors and institutions have called for...

Drawing from academic studies and public polling data, this report synthesises public attitudes towards AI with a focus on the United Kingdom and United States. It discusses public views on issues such as concern about job loss...

Increasingly many AI systems can plan and execute interactions in open-ended environments, such as making phone calls or buying online goods. As developers grow the space of tasks that such AI agents can accomplish...

This paper presents a survey of local US policymakers' views on the future impact and regulation of AI. Our survey provides insight into...

As artificial intelligence (AI) advances, states increasingly recognise the need for international governance to address shared benefits and challenges. However, international cooperation is complex and...

Frontier artificial intelligence (AI) systems pose increasing risks to society, making it essential for developers to provide assurances about their safety. One approach to offering such assurances is through a safety case...

AI systems are increasingly pervasive, yet information needed to decide whether and how to engage with them may not exist or be accessible. A user may not be able to verify whether a system has certain safety certifications...

As frontier artificial intelligence (AI) systems become more capable, it becomes more important that developers can explain why their systems are sufficiently safe. One way to do so is via safety cases: reports that...

This white paper investigates practical remedies to increase voice in and access to AI governance and capabilities for the Global AI Majority, while addressing the security and commercial concerns of frontier AI states.

This paper examines the profound challenges that transformative advances in AI towards Artificial General Intelligence (AGI) will pose for economists and economic policymakers.

Over the past year, artificial intelligence (AI) companies have been increasingly adopting AI safety frameworks. These frameworks outline how companies intend to keep the potential risks associated with developing and deploying...

Several jurisdictions are starting to regulate frontier artificial intelligence (AI) systems, i.e. general-purpose AI systems that match or exceed the capabilities present in the most advanced systems. To reduce risks...

Regulators in the US and EU are using thresholds based on training compute--the number of computational operations used in training--to identify general-purpose artificial intelligence (GPAI) models that may pose risks of...

AI progress is creating a growing range of risks and opportunities, but it is often unclear how they should be navigated. In many cases, the barriers and uncertainties faced are at least partly technical...

According to one survey, about half of Americans think that the increased use of AI will lead to greater income inequality and a more polarized society. Roughly two thirds...

We propose a framework for evaluating the potential impacts of large-language models (LLMs) and associated technologies on work by considering their relevance to the tasks workers perform in their jobs.

At the core of the financial system is the processing and aggregation of vast amounts of information into price signals that coordinate participants in the economy. Throughout history, advances in information processing...

Frontier artificial intelligence (AI) systems could pose increasing risks to public safety and security. But what level of risk is acceptable? One increasingly popular approach is to...

Increased delegation of commercial, scientific, governmental, and personal activities to AI agents—systems capable of pursuing complex goals with limited supervision—may...

Matthew van der Merwe, Ketan Ramakrishnan, and Markus Anderljung recently published this piece on Lawfare.

Existing strategies for managing risks from advanced AI systems often focus on affecting what AI systems are developed and how they diffuse. This approach becomes less feasible as the number of developers of advanced AI grows.

We analyze how output and wages behave under different scenarios for technological progress that may culminate in Artificial General Intelligence (AGI), defined as the ability of AI systems to perform all tasks that humans can...

Compute providers can play an essential role in a regulatory ecosystem via four key capacities: as securers, safeguarding AI systems and critical infrastructure; as record keepers, enhancing visibility for policymakers...

We welcome the opportunity to respond to the Request for Information (RFI) Related to NIST’s Assignments Under Sections 4.1, 4.5 and 11 of the Executive Order Concerning AI. We offer the following submission for your...

Recent AI progress has largely been driven by increases in the amount of computing power used to train new models. Governing compute could be an effective way to achieve AI policy goals, but could also introduce new risks.

Training advanced AI models requires large investments in computational resources, or compute. Yet, as hardware innovation reduces the price of compute and algorithmic advances make its use more efficient, the cost of training...

To deal with the risks of these high-compute frontier AI systems, we must govern not only how they can be used but also how they are developed and made available to people in the first place.

With the increasing integration of frontier large language models (LLMs) into society and the economy, decisions related to their training, deployment, and use have far-reaching implications...

In this written testimony to the U.S. Senate's bipartisan AI Insights Forum, Anton Korinek encourages lawmakers to engage in scenario planning to prepare the workforce for advances in AI.

Organizations that develop and deploy artificial intelligence (AI) systems need to manage the associated risks—for economic, legal, and ethical reasons. However, it is not always clear who is responsible for AI risk management.

Recent releases of frontier artificial intelligence (AI) models have largely been gated, providing the benefit of limiting the proliferation of increasingly powerful dual-use capabilities. However, such release strategies...

To address security and safety risks stemming from highly capable artificial intelligence (AI) models, we propose that the US government should ensure compute providers implement Know-Your-Customer (KYC) schemes.

This paper proposes an evaluation-based coordination scheme for situations in which frontier AI developers discover that their models have certain dangerous capabilities.

We evaluate the risks and benefits of open-sourcing, as well as alternative methods for pursuing open-source objectives.

This Brookings Working Paper analyses the structure of the market for foundation models, and examines the implications for competition policy and regulation.

This report describes trade-offs in the design of international governance arrangements for civilian artificial intelligence (AI) and presents one approach in detail.

Markus Anderljung and Paul Scharre recently published this piece in Foreign Affairs

We welcome the opportunity to respond to the OSTP Request for Information on National Priorities for Artificial Intelligence and look forward to future opportunities to provide additional input. We offer the following...

There are increasing concerns that AGI would pose catastrophic risks. In light of this, AGI companies need to drastically improve their risk management practices. This paper reviews popular risk assessment techniques...

Advanced AI models hold the promise of tremendous benefits for humanity, but society needs to proactively manage the accompanying risks. In this paper, we focus on what we term “frontier AI” models — highly capable...

We welcome the opportunity to respond to the NTIA’s AI Accountability Policy Request for Comment and look forward to future opportunities to provide additional input. We offer the following...

Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing...

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as...

A number of leading AI companies, including OpenAI, Google DeepMind, andAnthropic, have the stated goal of building artificial general intelligence (AGI)—AI systems that achieve or exceed human performance...

This study investigates how LLMs can be used for spear phishing, a form of cybercrime that involves manipulating targets into divulging sensitive information. It first explores LLMs’ ability to assist with the reconnaissance...

As large-scale pre-trained AI models gain popularity in the West, many Chinese AI labs have developed their own models capable of generating coherent text and realistic images and videos. These models represent the frontier...

Organizations that develop and deploy artificial intelligence (AI) systems need to take measures to reduce the associated risks. In this paper, we examine how AI companies could design an AI ethics board in a way that reduces...

We are pleased to see the publication of the UK’s Future of Compute Review. However, we also believe there is a significant missed opportunity: the review does not address how to ensure that compute is used responsibly or how...

This paper outlines four different notions of “AI democratisation”, three of which are used almost synonymously with “increasing accessibility”. The democratisation of AI use and the democratisation of AI development are about...

The development of privacy-enhancing technologies has made immense progress in reducing trade-offs between privacy and performance in data exchange and analysis. Similar tools could be useful for AI governance...

Artificial intelligence (AI) systems will increasingly be used to cause harm as they grow more capable. In fact, AI systems are already starting to be used to automate fraudulent activities, violate human rights, create...

The proposed EU AI Act is the first comprehensive attempt to regulate AI in a major jurisdiction. This article analyses Article 9, the key risk management provision in the AI Act. It gives an overview of the regulatory...

This web app is a tool for exploring the dynamics of risky AI competition: the safety-performance tradeoff. Will AI safety breakthroughs always lead to safer AI systems? Before long, we may be capable of creating AI systems...

Stuart Russell has had a seminal influence on many people with his writings on aligning artificial intelligence with human values and regulating autonomous weapons systems. His work has highlighted the difficulties...

This report summarises the most important aspects of the development of atomic weapons and draws out a number of important insights for the development of similarly important technologies.

The great powers appear to be entering an era of heightened competition to master security-relevant technologies in areas such as AI. This is concerning because deploying new technologies can create substantial...

In this report, we ask whether the EU’s upcoming regulation for AI will diffuse globally, producing a so-called “Brussels Effect”.

This Brookings working paper presents an analysis of how institutions could be set up to prepare for a scenario of increasingly smart autonomous machines which could replace human labor and...

We welcome the opportunity to respond to the Future of Compute Review’s call for evidence...Our response focuses on the future of compute used for Artificial Intelligence (AI). In particular, we emphasise the risks posed by...