New technologies can drastically shift the balance of power in society. Great Britain’s early dominance in the Industrial Revolution, for example, helped empower its global empire.¹

With AI technology rapidly advancing, there’s a serious risk that it might enable an even more extreme global power grab.

Advanced AI is particularly concerning because it could be controlled by a small number of people, or even just one. An AI could be copied indefinitely, and with enough computing infrastructure and a powerful enough system, a single person could control a virtual or literal army of AI agents.

And since advanced AI could potentially trigger explosive growth in the economy, technology, and intelligence, anyone with unilateral control over the most powerful systems might be able to dominate the rest of humanity.

One factor that enhances this threat is the possibility of secret loyalties. It may be possible to create AI systems that appear to have society’s best interests in mind but are actually loyal to just one person or small group.² As these systems are deployed throughout the economy, government, and military, they could constantly seek opportunities to advance the interests of their true masters.

Here are three possible pathways through which AI could enable an unprecedented power grab:

1. AI developers seize control — in this scenario, actors within a company or organisation developing frontier AI systems use their technology to seize control. This could happen if they deploy their systems to be used widely in the economy, military, and government while it retains secret loyalty to them. Or they could potentially create powerful enough systems internally that can gather enough wealth and resources to launch a hostile takeover of other centres of power.
2. Military coups — as militaries incorporate AI for competitive advantage, they introduce new vulnerabilities. AI-controlled weapons systems and autonomous military equipment could be designed to follow orders unscrupulously, without the formal and informal checks on power that militaries traditionally provide — such as the potential for mutiny in the face of unlawful orders. A military leader or other actor (including potentially hostile foreign governments) could find a way to ensure the military AI is loyal to them, and use it to assert far-reaching control.
3. Autocratisation — political leaders could use advanced AI systems to entrench their power. They may be elected or unelected to start, but either way, they could use advanced AI systems to undermine any potential political challenger. For example, they could use enhanced surveillance and law enforcement to subdue the opposition.

Extreme power concentrated in the hands of a small number of people would pose a major threat to the interests of the rest of the world. It could even undermine the potential of a prosperous future, since the course of events may depend on the whims of those who happened to have dictatorial aspirations.

There are also ways AI could likely be used to broadly improve governance, but we’d expect scenarios in which AI enables hostile or illegitimate power grabs would be bad for the future of humanity.

What can be done to mitigate these risks?

We’d like to see much more work done to figure out the best methods for reducing the risk of an AI-enabled power grab. Several approaches that could help include:

• Safeguards on internal use: Implement sophisticated monitoring of how AI systems are used within frontier companies, with restrictions on access to “helpful-only” models that will follow any instructions without limitations.
• Transparency about model specifications: Publish detailed information about how AI systems are designed to behave, including safeguards and limitations on their actions, allowing for external scrutiny and identification of potential vulnerabilities.
• Sharing capabilities broadly: Ensure that powerful AI capabilities are distributed among multiple stakeholders rather than concentrated in the hands of a few individuals or organizations. This creates checks and balances that make power grabs more difficult. Note though that there are also risks to having powerful AI capabilities distributed widely, so the competing considerations need to be carefully weighed.
• Inspections for secret loyalties: Develop robust technical methods to detect whether AI systems have been programmed with hidden agendas or backdoors that would allow them to serve interests contrary to their stated purpose.
• Military AI safeguards: Require that AI systems deployed in military contexts have robust safeguards against participating in coups, including principles against attacking civilians and multiple independent authorisation requirements for extreme actions.

For much more detail on this problem, listen to our interview with Tom Davidson.

Learn more

• AI-Enabled Coups: How a small group could use AI to seize power by Tom Davidson, Lukas Finnveden, and Rose Hadshar
• Podcast: Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared
• Our career review on AI governance and policy
• Our problem profile on stable totalitarianism
• Sleeper agents: training deceptive LLMs that persist through safety training by Evan Hubinger et al. of Anthropic

The post AI-enabled power grabs appeared first on 80,000 Hours.

Advancing technology has historically benefited humanity. The invention of fire, air conditioning, and antibiotics have all come with some downsides, but overall they’ve helped humans live healthier, happier, and more comfortable lives.

But this trend isn’t guaranteed to continue.

We’ve written about how the development of advanced AI technology poses existential risks. One prominent and particularly concerning threat model is that as AI systems get more powerful, they’ll develop interests that are not aligned with humanity. They may, unbeknownst to their creators, become power-seeking. They may intentionally deceive us about their intentions and use their superior intelligence and advanced planning capabilities to disempower humanity or drive us to extinction.

It’s possible, though, that the development of AI systems could lead to human disempowerment and extinction even if we succeed in preventing AI systems from becoming power-seeking and scheming against us.

In a recent paper, Jan Kulveit and his co-authors call this threat model gradual disempowerment. They argue for the following six claims:

1. Large societal systems, such as economies and governments, tend to be roughly aligned to human interests.¹
2. This rough alignment of the societal systems is maintained by multiple factors, including voting systems, consumer demand signals, and the reliance on human labour and thinking.
3. Societal systems that rely less on human labour and thinking — and rely more on increasingly advanced and powerful AI systems — will be less aligned with human interests.
4. AI systems may indeed outcompete human labour for key roles in societal systems in part because they can more ruthlessly pursue the directions they’re given. And this may cause the systems to be even less aligned with human interests.
5. If one societal system becomes misaligned with human interests, like a national economy, it may increase the chance that other systems become misaligned. Powerful economic actors have historically wielded influence over national governments, for example.
6. Humans could gradually become disempowered, perhaps permanently, as AIs increasingly control societal systems and these systems become increasingly misaligned from human interests. In the extreme case, it could lead to human extinction.

Kulveit et al. discuss how AI systems could come to dominate the economy, national governments, and even culture in ways that act against humanity’s interests.

It may be hard to imagine how humans would let this happen, because in this scenario, the AI systems aren’t being actively deceptive. Instead, they follow human directions.

The trouble is that due to competitive pressures, we may find ourselves narrowly incentivised to hand over more and more control to the AI systems themselves. Some human actors — corporations, governments, or other institutions — will initially gain significant power through AI deployment, using these systems to advance their interests and missions.

Here’s how it might happen:

• First, economic and political leaders adopt AI systems that enhance their existing advantages. A financial firm deploys AI trading systems that outcompete human traders. Politicians use AI advisers to win elections and keep voters happy. These initial adopters don’t experience disempowerment — they experience success, which encourages their competitors to also adopt AI.
• As time moves on, humans have less control. Corporate boards might try to change direction against the advice of their AIs, only to find share prices plummeting because the AIs had a far better business strategy. Government officials may realise they don’t understand the AI systems running key services enough to change what they’re doing successfully.
• Only later, as AI systems become increasingly powerful, might there be signs that the systems are drifting out of alignment with human interests — not because they are trying to, but because they are advancing proxies of success that don’t quite line up with what’s actually good for people.
• In the cultural sphere, for example, media companies might deploy AI to create increasingly addictive content, reshaping human preferences. What begins as entertainment evolves into persuasion technology that can shape political outcomes, diminishing democratic control.

Once humans start losing power in these ways, they may irreversibly have less and less ability to influence the future course of events. Eventually, their needs may not be addressed at all by the most powerful global actors. In the most extreme case, the species as we know it may not survive.

Many other scenarios are possible.

There are some versions of apparent “disempowerment” that could look like a utopia: humans flourishing and happy in a society expertly managed and fundamentally controlled by benevolent AI systems. Or maybe one day, humanity will decide it’s happy to cede the future to AI systems that we consider worthy descendants.

But this risk is that humanity could “hand over” control unintentionally and in a way that few of us would endorse. We might be gradually replaced by AI systems with no conscious experiences, or the future may eventually be dominated by fierce Darwinian competition between various digital agents. That could mean the future is sapped of most value — a catastrophic loss.

We want to better understand these dynamics and risks to increase the prospects that the future goes well.

How pressing is this issue?

We feel very uncertain about how likely various gradual disempowerment scenarios are. It is difficult to disentangle the possibilities from related risks of power-seeking AI systems and questions about the moral status of digital minds, which are also hard to be certain about.

Because the area is steeped in uncertainty, it’s unclear what the best interventions are. We think more work should be done to understand this problem and its potential solutions at least — and it’s likely some people should be focusing on it.

What are the arguments against this being a pressing problem?

There are several reasons you might not think this problem is very pressing:

• You might think it will be solved by default, because if we avoid other risks from AI, advanced AI systems will help us navigate these problems.
• You might think it’s very unlikely that AI systems, if not actively scheming against us, will end up contributing to an existential catastrophe for humanity — even if there are some problems of disempowerment. This might make you think this is an issue, but not nearly as big an issue as other, more existential risks from AI.
• You might think there just aren’t good solutions to this problem.
• You might think the gradual disempowerment of humanity wouldn’t constitute an existential catastrophe. For example, perhaps it’d be good or nearly as good as other futures.

What can you do to help?

Given the relatively limited state of our knowledge on this topic, we’d guess the best way to help with this problem is likely carrying out more research to understand it better. (Read more about research skills.)

Backgrounds in philosophy, history, economics, sociology, and political science — in addition to machine learning and AI — may be particularly relevant.

You might want to work in academia, think tanks, or at nonprofit research institutions.

At some point, if we have a better understanding of threat models and potential solutions, it will likely be important to have people working in AI governance and policy who are focused on reducing these risks. So pursuing a career in AI governance, while building an understanding of this emerging area of research as well as the other major AI risks, may be a promising strategy for eventually helping to reduce the risk of gradual disempowerment.

Kulveit et al. suggest some approaches to mitigating the risk of gradual disempowerment, including:

• Measuring and monitoring
  • Develop metrics to track human and AI influence in economic, cultural, and political systems
  • Make plans to identify warning signs of potential disempowerment
• Preventing excessive AI influence
  • Implement regulatory frameworks requiring human oversight
  • Apply progressive taxation on AI-generated revenues
  • Establish cultural norms supporting human agency
• Strengthening human control:
  • Create more robust democratic processes
  • Ensure that AI systems remain understandable to humans
  • Develop AI delegates that represent human interests while remaining competitive
• System-wide alignment
  • Research “ecosystem alignment” that maintains human values within complex socio-technical systems
  • Develop frameworks for aligning civilisation-wide interactions between humans and AI

Key organisations in this space

Some organisations where you might be able to do relevant research include:

• Alignment of Complex Systems Research Group (ACS) — which led the work on the “Gradual Disempowerment” paper discussed above
• Forethought Research
• Open Philanthropy²
• Rethink Priorities

You can also explore roles at other organisations that work on AI safety and policy.

Our job board features opportunities in AI safety and policy:

View all opportunities

Learn more

• Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development by Jan Kulveit et al.
• Two Types of AI Existential Risk: Decisive and Accumulative by Atoosa Kasirzadeh
• What failure looks like by Paul Christiano
• Will Humanity Choose Its Future? by Guive Assadi
• Natural Selection Favors AIs over Humans by Dan Hendrycks
• TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI by Andrew Critch and Stuart Russell
• Our interview with Carl Shulman on the economy and national security after AGI, which talks about why humanity seems likely to hand over more control to AI systems
• Our interview with Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared
• Out interview with Tom Davidson on how AI-enabled coups could allow a tiny group to seize power
• Our article on longtermism

Read next:  Explore other pressing world problems

Want to learn more about global issues we think are especially pressing? See our list of issues that are large in scale, solvable, and neglected, according to our research.

Continue →

The post Gradual disempowerment appeared first on 80,000 Hours.

• Growth in the number of postings in the field
• The types of organisations that are hiring
• The most in-demand skills
• The experience level required for these roles

We’ve ranked catastrophic risks from AI as the world’s most pressing problem since 2016, but it’s only in the last few years that the topic has really hit the mainstream.

As AI has advanced rapidly and the risks have become more salient, we’ve seen many more jobs available to help mitigate the dangers.

The number of AI-related jobs we posted on our job board rose throughout 2023 and then plateaued in 2024. But in January 2025, we posted the most AI-relevant jobs yet!

In 2023, we posted an average of 63 AI-related roles per month. In 2024, the average rose to 105 — a 67% increase.

Over this time, nonprofit jobs have been the most common, though they were briefly overtaken by both company and government jobs in early 2024.

This trend could reflect our vantage point. As a nonprofit that works closely with other nonprofits, we may be best positioned to find and assess high-impact roles in this sector while potentially missing other great roles in sectors more opaque to us.

That said, one reason we’ve prioritised AI risk reduction is the potential failure of market and political mechanisms to produce a proportionate response to the challenge. So it’s not that surprising that nonprofits might be more likely to offer great opportunities for this work.

If you are looking for work in this field, you may want to know what skills employers look for.

Here are the trends in the AI-related roles we’ve posted in the last two years, broken down by the popular skills we tagged them with.

This shows that:

• Research continues to be the most in-demand skill, as AI safety is fundamentally an area of research.
• Demand for policy-relevant skills increased after mid-2023 and remained high in 2024, coinciding with our updated career path rankings, which placed AI governance and policy in the top slot.
• Roles requiring outreach skills, which could also be categorised as communications skills, trail the pack. That’s not too surprising, but we think there may be more opportunities here in the future. Many people working on AI risk think that people with strong communication skills and AI knowledge could be highly valuable. If that’s you, consider applying for our 1-1 advising so we can guide you to potentially high-impact opportunities.

If you want to build skills to work on AI but aren’t sure how, check out our guides to building key skills, which includes:

• Research
• Programming
• Policy
• Information security
• Outreach / communication
• Operations and organisation-building

Next, let’s look at the breakdown of AI-related roles by experience level.

Opportunities for junior (1–4 years) and mid-career (5–9 years) professionals dominate the pack and have grown the most as the total number of jobs increased. We find fewer jobs aimed at senior and entry-level folks, though there have been somewhat more senior-level jobs posted in 2024 than in the first half of 2023.

However, these numbers may understate the level of demand for senior-level hires. We frequently hear from employers in our network that they would love to hire people with many years or decades of professional experience. But these types of roles may not often be publicly advertised and so wouldn’t appear on our job board.

If you have many years of experience and are looking for these kinds of roles, we’d encourage you to apply for our 1-1 advising.

The lack of entry-level positions may feel discouraging to some, but it just means that you’ll likely need to build career capital before aiming for most of the roles that appear on our job board. Our career guide is the best place to start if you’re at that stage.

Here are some other interesting trends:

• We’ve seen fewer safety-relevant job postings at OpenAI over the last year, and more safety-relevant jobs at Google DeepMind.
• In February, as the Trump administration began to look for roles to cut in the federal government, we saw an increase in people searching for roles on our job board in D.C.

Some caveats on this data:

• Our goal is to only list roles that can contribute to a career aimed at reducing the largest risks from AI, so this certainly isn’t a comprehensive overview of all AI jobs.
  The judgements about which roles to include and how to categorise them are difficult and complex, and we’re surely mistaken about them in some ways — and that likely affects the trends above!
• We’ve also gotten better at adding more jobs over the years, which may influence some of the observed trends.

Learn more:

• Our career review of AI governance and policy
• Our career review of AI safety technical research
• Our job board FAQ

The post Understanding trends in our AI job postings appeared first on 80,000 Hours.

In fact, we face the possibility of biological disasters that are worse than ever before due to developments in technology.

The chances of such catastrophic pandemics — bad enough to potentially derail civilisation and threaten humanity’s future — seem uncomfortably high. We believe this risk is one of the world’s most pressing problems.

And there are a number of practical options for reducing global catastrophic biological risks (GCBRs). So we think working to reduce GCBRs is one of the most promising ways to safeguard the future of humanity right now.

Why focus your career on preventing severe pandemics?

COVID-19 highlighted our vulnerability to worldwide pandemics and revealed weaknesses in our ability to respond. Despite advances in medicine and public health, around seven million deaths worldwide from the disease have been recorded, and many estimates put the figure far higher.

Historical events like the Black Death and the 1918 flu show that pandemics can be some of the most damaging disasters for humanity, killing tens of millions and significant portions of the global population.

It is sobering to imagine the potential impact of a pandemic pathogen that is much more contagious and deadly than any we’ve seen so far.

Unfortunately, such a pathogen is possible in principle, particularly in light of advancing biotechnology. Researchers can design and create biological agents much more easily and precisely than before. (More on this below.) As the field advances, it may become increasingly feasible to engineer a pathogen that poses a major threat to all of humanity.

States or malicious actors with access to these pathogens could use them as offensive weapons or wield them as threats to obtain leverage over others.

Dangerous pathogens engineered for research purposes could also be released accidentally through a failure of lab safety.

Either scenario could result in a catastrophic ‘engineered pandemic,’ which we believe could pose an even greater threat to humanity than pandemics that arise naturally, as we argue below.

Thankfully, few people seek to use disease as a weapon, and even those willing to conduct such attacks may not aim to produce the most harmful pathogen possible. But the combined possibilities of accident, recklessness, desperation, and unusual malice suggest a disturbingly high chance of a pandemic pathogen being released that could kill a very large percentage of the population. The world might be especially at risk during great power conflicts.

But could an engineered pandemic pose an extinction threat to humanity?

There is reasonable debate here. In the past, societies have recovered from pandemics that killed as much as 50% of the population, and perhaps more.¹

But we believe future pandemics may be one of the largest contributors to existential risk this century, because it now seems within the reach of near-term biological advances to create pandemics that would kill greater than 50% of the population — not just in a particular area, but globally. It’s possible they could be bad enough to drive humanity to extinction, or at least be so damaging that civilisation never recovers.

Reducing the risk of biological catastrophes by constructing safeguards against potential outbreaks and preparing to mitigate their worst effects therefore seems extremely important.

It seems relatively uncommon for people in the broader field of biosecurity and pandemic preparedness to work specifically on reducing catastrophic risks and engineered pandemics. Projects that reduce the risk of biological catastrophe also seem to receive a relatively small proportion of health security funding.²

In our view, the costs of biological disasters grow nonlinearly with severity because of the increasing potential for the event to contribute to existential risk. This suggests that projects to prevent the gravest outcomes in particular should receive more funding and attention than they currently do.

In the rest of this section, we’ll discuss how artificial pandemics compare to natural pandemic risks. Later on, we’ll discuss what kind of work can and should be done in this area to reduce the risks.

We also have a career review of biorisk research, strategy, and policy paths, which gives more specific and concrete advice about impactful roles to aim for and how to enter the field.

Natural pandemics show how destructive biological threats can be

Four of the worst pandemics in recorded history were:³

1. The Plague of Justinian (541-542 CE) is thought to have arisen in Asia before spreading into the Byzantine Empire around the Mediterranean. The initial outbreak is thought to have killed around 6 million (about ~3% of world population)⁴ and contributed to reversing the territorial gains of the Byzantine empire.
2. The Black Death (1335-1355 CE) is estimated to have killed 20–75 million people (about 10% of world population) and believed to have had profound impacts on the course of European history.
3. The Columbian Exchange (1500-1600 CE) was a succession of pandemics, likely including smallpox and paratyphoid, brought by the European colonists that devastated Native American populations. It likely played a major role in the loss of around 80% of Mexico’s native population during the 16th century. Other groups in the Americas appear to have lost even greater proportions of their communities. Some groups may have lost as much as 98% of their people to these diseases.⁵
4. The 1918 Influenza Pandemic (1918 CE) spread across almost the whole globe and killed 50–100 million people (2.5%–5% of the world population). It may have been deadlier than either world war.

These historical pandemics show the potential for mass destruction from biological threats, and they are a threat worth mitigating all on their own. They also show that the key features of a global catastrophe, such as high proportional mortality and civilisational collapse, can be driven by highly destructive pandemics.

But despite the horror of these past events, it seems unlikely that a natural pandemic could be bad enough on its own to drive humanity to total extinction in the foreseeable future, given what we know of events in natural history.⁶

As philosopher Toby Ord argues in the section on natural risks in his book The Precipice, history suggests humanity faces a very low baseline extinction risk — the chance of being wiped out in ordinary circumstances — from natural causes over the course of, say, 100 years.

That’s because if the baseline risk were around 10% per century, we’d have to conclude we’ve gotten very lucky for the 200,000 years or so of humanity’s existence. The fact of our existence is much less surprising if the risk has been about 0.001% per century.

None of the worst plagues we know about in history was enough to destabilise civilization worldwide or clearly imperil our species’ future. And more broadly, pathogen-driven extinction events in nature appear to be relatively rare for animals.⁷

Is the risk from natural pandemics increasing or decreasing?

Are we safer from pandemics now than we used to be? Or do developments in human society actually put us at greater risk from natural pandemics?

Good data on these questions is hard to find. The burden of infectious disease generally in human society is on a downward trend, but this doesn’t tell us much about whether infrequent outbreaks of mass pandemics could be getting worse.

In the abstract, we can think of many reasons that the risk from naturally arising pandemics might be falling. They include:

• We have better hygiene and sanitation than past eras, and these will likely continue to improve.
• We can produce effective vaccinations and therapeutics.
• We better understand disease transmission, infection, and effects on the body.
• The human population is healthier overall.

On the other hand:

• Trade and air travel allow much faster and wider transmission of disease.⁸ For example, air travel seems to have played a large role in the spread of COVID-19 from country to country.⁹ In previous eras, the difficulty of travelling over long distances likely kept disease outbreaks more geographically confined.
• Climate change may increase the likelihood of new zoonotic diseases.
• Greater human population density may increase the likelihood that diseases will spread rapidly.
• Much larger populations of domestic animals can potentially pass diseases on to humans.

There are likely many other relevant considerations. Our guess is that the frequency of natural pandemics is increasing, but that they’ll be less bad on average.¹⁰ A further guess is that the second factor is more important than the first factor, netting out to reduced overall danger. There remain many open questions.

Engineered pathogens could be even more dangerous

But even if natural pandemic risks are declining, the risks from engineered pathogens are almost certainly growing.

This is because advancing technology makes it increasingly feasible to create threatening viruses and infectious agents.¹¹ Accidental and deliberate misuse of this technology is a credible global catastrophic risk and could potentially threaten humanity’s future.

One way this could play out is if some dangerous actor wanted to bring back catastrophic outbreaks of the past.

Polio, the 1918 pandemic influenza strain, and most recently horsepox (a close relative of smallpox) have all been recreated from scratch. The genetic sequence of all these pathogens and others are publicly available, and the progress and proliferation of biotechnology opens up terrifying opportunities.¹²

Beyond the resurrection of past plagues, advanced biotechnology could let someone engineer a pathogen more dangerous than those that have occurred in natural history.

When viruses evolve, they aren’t naturally selected to be as deadly or destructive as possible. But someone who is deliberately trying to cause harm could intentionally combine the worst features of possible viruses in a way that is very unlikely to happen naturally.

Gene sequencing, editing, and synthesis are now possible and becoming easier. We’re getting closer to being able to produce biological agents the way we design and produce computers or other products (though how long it takes remains unclear). This may allow people to design and create pathogens that are deadlier or more transmissible, or perhaps have wholly new features. (Read more.)

Scientists are also investigating what makes pathogens more or less lethal and contagious, which may help us better prevent and mitigate outbreaks.

But it also means that the information required to design more dangerous pathogens is increasingly available.

All the technologies involved have potential medical uses in addition to hazards. For example, viral engineering has been employed in gene therapy and vaccines (including some used to combat COVID-19).

Yet knowledge of how to engineer viruses to be better as vaccines or therapeutics could be misused to develop ‘better’ biological weapons. Properly handling these advances involves a delicate balancing act.

Hints of the dangers can be seen in the scientific literature. Gain-of-function experiments with influenza suggested that artificial selection could lead to pathogens with properties that enhance their danger.¹³

And the scientific community has yet to establish strong enough norms to discourage and prevent the unrestricted sharing of dangerous findings, such as methods for making a virus deadlier. That’s why we warn people going to work in this field that biosecurity involves information hazards. It’s essential for people handling these risks to have good judgement.

Scientists can make dangerous discoveries unintentionally in lab work. For example, vaccine research can uncover virus mutations that make a disease more infectious. And other areas of biology, such as enzyme research, show how our advancing technology can unlock new and potentially threatening capabilities that haven’t appeared before in nature.¹⁴

In a world of many ‘unknown unknowns,’ we may find many novel dangers.

So while the march of science brings great progress, it also brings the potential for bad actors to intentionally produce new or modified pathogens. Even with the vast majority of scientific expertise focused on benefiting humanity, a much smaller group can use the community’s advances to do great harm.

If someone or some group has enough motivation, resources, and sufficient technical skill, it’s difficult to place an upper limit on how catastrophic an engineered pandemic they might one day create. As technology progresses, the tools for creating a biological disaster will become increasingly accessible; the barriers to achieving terrifying results may get lower and lower — raising the risk of a major attack. The advancement of AI, in particular, may catalyse the risk. (See more about this below.)

Both accidental and deliberate misuse are threats

We can divide the risks of artificially created pandemics into accidental and deliberate misuse — roughly speaking, imagine a science experiment gone wrong compared to a bioterrorist attack.

The history of accidents and lab leaks which exposed people to dangerous pathogens is chilling:

• In 1977, an unusual flu strain emerged that disproportionately sickened young people and was found to be genetically frozen in time from a 1950 strain, suggesting a lab origin from a faulty vaccine trial.
• In 1978, a lab leak at a UK facility resulted in the last smallpox death.
• In 1979, an apparent bioweapons lab in the USSR accidentally released anthrax spores that drifted over a town, sickening residents and animals, and killing about 60 people. Though initially covered up, Russian President Boris Yeltsin later revealed it was an airborne release from a military lab accident.
• In 2014, dozens of CDC workers were potentially exposed to live anthrax after samples meant to be inactivated were improperly killed and shipped to lower-level labs that didn’t always use proper protective equipment.
• We don’t really know how often this kind of thing happens because lab leaks are not consistently tracked. And there have been many more close calls.

And history has seen many terrorist attacks and state development of mass-casualty weapons. Incidents of bioterrorism and biological warfare include:

• In 1763, British forces at Fort Pitt gave blankets from a smallpox ward to Native American tribes, aiming to spread the disease and weaken these communities. It’s unclear if this effort achieved its aims, though smallpox devastated many of these groups.
• During World War II, the Japanese military’s Unit 731 conducted horrific human experiments and biological warfare in China. They used anthrax, cholera, and plague, killing thousands and potentially many more. The details of these events were only uncovered later.
• In the 1960s and 1970s, the South African government developed a covert chemical and biological warfare program known as Project Coast. The program aimed to develop biological and chemical agents targeted at specific ethnic groups and political opponents, including efforts to develop sterilisation and infertility drugs.
• In 1984, followers of the Rajneesh movement contaminated salad bars in Oregon with Salmonella, causing more than 750 infections. It was an attempt to influence an upcoming election.
• In 2001, shortly after the September 11 attacks, anthrax spores were mailed to several news outlets and two U.S. Senators, causing 22 infections and five deaths.

So should we be more concerned about accidents or bioterrorism? We’re not sure. There’s not a lot of data to go on, and considerations pull in both directions.

It may seem releasing a deadly pathogen on purpose is more concerning. As discussed, the worst pandemics would most likely be intentionally created rather than emerge by chance, as discussed above. Plus, there are ways to make a pathogen’s release more or less harmful, and an accidental release probably wouldn’t be optimised for maximum damage.

On the other hand, many more people are well-intentioned and want to use biotechnology to help the world rather than harm it. And efforts to eliminate state bioweapons programs likely reduce the number of potential attackers. (But see more about the limits on these efforts below.) So it seems most plausible that there are more opportunities for a disastrous accident to occur than for a malicious actor to pull off a mass biological attack.

We guess that, all things considered, the former considerations are the more significant factors.¹⁷ So we suspect that deliberate misuse is more dangerous than accidental releases, though both are certainly worth guarding against.

This image is borrowed from Claire Zabel’s talk on biosecurity.¹⁸

Overall, the risk seems substantial

We’ve seen a variety of estimates regarding the chances of an existential biological catastrophe, including the possibility of engineered pandemics.¹⁹ Perhaps the best estimates come from the Existential Risk Persuasion Tournament (XPT).

This project involved getting groups of both subject matter experts and experienced forecasters to estimate the likelihood of extreme events. For biological risks, the range of median estimates between forecasters and domain experts were as follows:

• Catastrophic event (meaning an event in which 10% or more of the human population dies) by 2100: ~1–3%
• Human extinction event: 1 in 50,000 to 1 in 100
• Genetically engineered pathogen killing more than 1% of the population by 2100: 4–10%²⁰
• Note: the forecasters tended to have lower estimates of the risk than domain experts.

Although they are the best available figures we’ve seen, these numbers have plenty of caveats. The main three are:

1. There is little evidence that anyone can achieve long-term forecasting accuracy. Previous forecasting work has assessed performance for questions that would resolve in months or years, not decades.
2. There was a lot of variation in estimates within and between groups — some individuals gave numbers many times, or even many orders of magnitude, higher or lower than one another.²¹
3. The domain experts were selected for those already working on catastrophic risks — the typical expert in some areas of public health, for example, might generally rate extreme risks lower.

It’s hard to be confident about how to weigh up these different kinds of estimates and considerations, and we think reasonable people will come to different conclusions.

Our view is that given how bad a catastrophic pandemic would be, the fact that there seems to be few limits on how destructive an engineered pandemic could be, and how broadly beneficial mitigation measures are, many more people should be working on this problem than current are.

Reducing catastrophic biological risks is highly valuable according to a range of worldviews

Because we prioritise world problems that could have a significant impact on future generations, we care most about work that will reduce the biggest biological threats — especially those that could cause human extinction or derail civilisation.

But biosecurity and catastrophic risk reduction could be highly impactful for people with a range of worldviews, because:

1. Catastrophic biological threats would harm near-term interests too. As COVID-19 showed, large pandemics can bring extraordinary costs to people today, and even more virulent or deadly diseases would cause even greater death and suffering.
2. Interventions that reduce the largest biological risks are also often beneficial for preventing more common illnesses. Disease surveillance can detect both large and small outbreaks; counter-proliferation efforts can stop both higher- and lower-consequence acts of deliberate misuse; better PPE could prevent all kinds of infections; and so on.

There is also substantial overlap between biosecurity and other world problems, such as global health (e.g. the Global Health Security Agenda), factory farming (e.g. ‘One Health‘ initiatives), and AI.

There are clear actions we can take to reduce these risks

Biosecurity and pandemic preparedness are multidisciplinary fields. To address these threats effectively, we need a range of approaches, including:

• Technical and biological researchers to investigate and develop tools for controlling outbreaks
• Entrepreneurs and industry professionals to develop and implement these
• Strategic researchers and forecasters to develop plans
• People in government to pass and implement policies aimed at reducing biological threats

Specifically, you could:

• Work with government, academia, industry, and international organisations to improve the governance of gain-of-function research involving potential pandemic pathogens, commercial DNA synthesis, and other research and industries that may enable the creation of (or expand access to) particularly dangerous engineered pathogens
• Strengthen international commitments to not develop or deploy biological weapons, e.g. the Biological Weapons Convention (see below)
• Develop new technologies that can mitigate or detect pandemics, or the use of biological weapons,²² including:
  • Broad-spectrum testing, therapeutics, and vaccines — and ways to develop, manufacture, and distribute all of these quickly in an emergency²³
  • Detection methods, such as wastewater surveillance, that can find novel and dangerous outbreaks
  • Non-pharmaceutical interventions, such as better personal protective equipment
  • Other mechanisms for impeding high-risk disease transmission, such as anti-microbial far UVC light
• Deploying and otherwise promoting the above technologies to protect society against pandemics and to lower the incentives for trying to create one
• Improving information security to protect biological research that could be dangerous in the wrong hands
• Investigating whether advances in AI will exacerbate biorisks and potential solutions to this challenge
• For more discussion of biosecurity priorities, you can read our article on advice from biosecurity experts about the best way to fight the next pandemic.

The broader field of biosecurity and pandemic preparedness has made major contributions to reducing catastrophic risks. Many of the best ways to prepare for more probable but less severe outbreaks will also reduce the worst risks.

For example, if we develop broad-spectrum vaccines and therapeutics to prevent and treat a wide range of potential pandemic pathogens, this will be widely beneficial for public health and biosecurity. But it also likely decreases the risk of the worst-case scenarios we’ve been discussing — it’s harder to launch a catastrophic bioterrorist attack on a world that is prepared to protect itself against the most plausible disease candidates. And if any state or other actor who might consider manufacturing such a threat knows the world has a high chance of being protected against it, they have even less reason to try in the first place.

Similar arguments can be made about improved PPE, some forms of disease surveillance, and indoor air purification.

But if your focus is preventing the worst-case outcomes, you may want to focus on particular interventions within biosecurity and pandemic prevention over others.

Some experts in this area, such as MIT biologist Kevin Esvelt, believe that the best interventions for reducing the risk from human-made pandemics will come from the world of physics and engineering, rather than biology.

This is because for every biological countermeasure to reduce pandemic risk, such as vaccines, there may be tools in the biological sciences to overcome these obstacles — just as viruses can evolve to evade vaccine-induced immunity.

And yet, there may be hard limits to the ability of biological threats to overcome physical countermeasures. For instance, it seems plausible that there may just be no viable way to design a virus that can penetrate sufficiently secure personal protective equipment or to survive under far-UVC light. If this argument is correct, then these or similar interventions could provide some of the strongest protection against the biggest pandemic threats.

Two example ways to reduce catastrophe biological risks

We illustrate two specific examples of work to reduce catastrophic biological risks below, though note that many other options are available (and may even be more tractable).

1. Strengthen the Biological Weapons Convention

The principal defence against proliferation of biological weapons among states is the Biological Weapons Convention. The vast majority of eligible states have signed or ratified it.

Yet some states that signed or ratified the convention have also covertly pursued biological weapons programmes. The leading example was the Biopreparat programme of the USSR,²⁴ which at its height spent billions and employed tens of thousands of people across a network of secret facilities.²⁵

Its activities are alleged to have included industrial-scale production of weaponised agents like plague, smallpox, and anthrax. They even reportedly succeeded in engineering pathogens for increased lethality, multi-resistance to therapeutics, evasion of laboratory detection, vaccine escape, and novel mechanisms of disease not observed in nature.²⁶ Other past and ongoing violations in a number of countries are widely suspected.²⁷

The Biological Weapons Convention faces ongoing difficulties:

• The convention lacks verification mechanisms for countries to demonstrate their compliance, and the technical and political feasibility of verification is fraught.
• It also lacks an enforcement mechanism, so there are no consequences even if a state were out of compliance.
• The convention struggles for resources. It has only a handful of full-time staff, and many states do not fulfil their financial obligations. The 2017 meeting of states’ parties was only possible thanks to overpayment by some states, and the 2018 meeting had to be cut short by a day due to insufficient funds.²⁸

Working to improve the convention’s effectiveness, increasing its funding, or promoting new international efforts that better achieve its aims could help reduce the risk of a major biological catastrophe.

2. Govern dual-use research of concern

As discussed above, some well-meaning research has the potential to increase catastrophic risks. Such research is often called ‘dual-use research of concern,’ since the research could be used in either beneficial or harmful ways.

The primary concerns are that dangerous pathogens could be accidentally released or dangerous specimens and information produced by the research could fall into the hands of bad actors.

Gain-of-function experiments by Yoshihiro Kawaoka and Ron Fouchier raised concerns in 2011. They published results showing they had modified avian flu to spread in ferrets — raising fears that it might also be enabled to spread to humans.

The synthesis of horsepox is a more recent case. Good governance of this kind of research remains more aspiration than reality.

Individual investigators often have a surprising amount of discretion when carrying out risky experiments. It’s plausible that typical scientific norms are not well-suited to appropriately managing the dangers intrinsic in some of this work.

Even in the best case, where the scientific community is solely composed of those who only perform work which they sincerely believe is on balance good for the world, we might still face the unilateralist curse. This occurs when only one individual mistakenly concludes that a dangerous course of action should be taken, even when all their peers have ruled it out. This makes the chance of disaster much more likely, because it only takes one person making an incorrect risk assessment to impose major costs on the rest of society.

And in reality, scientists are subject to other incentives besides the public good, such as publications, patents, and prestige. It would be better if safety-enhancing discoveries were found before easier to make dangerous discoveries arise. But the existing incentives may encourage researchers to conduct their work in ways that aren’t always optimal for the social good.

Governance and oversight can mitigate risks posed by individual foibles or mistakes. The track record of such oversight bodies identifying concerns in advance is imperfect. The gain-of-function work on avian flu was initially funded by the NIH (the same body which would subsequently declare a moratorium on gain-of-function experiments), and passed institutional checks and oversight — concerns only began after the results of the work became known.

When reporting the horsepox synthesis to the WHO advisory committee on variola virus research, the scientists noted:

Professor Evans’ laboratory brought this activity to the attention of appropriate regulatory authorities, soliciting their approval to initiate and undertake the synthesis. It was the view of the researchers that these authorities, however, may not have fully appreciated the significance of, or potential need for, regulation or approval of any steps or services involved in the use of commercial companies performing commercial DNA synthesis, laboratory facilities, and the federal mail service to synthesise and replicate a virulent horse pathogen.

One challenge is there is no bright line one can draw to rule out all concerning research. List-based approaches, such as select agent lists or the seven experiments of concern, may increasingly be unsuited to current and emerging practice, particularly in such a dynamic field.

But it’s not clear what the alternative to necessarily incomplete lists would be. The consequences of scientific discovery are often not obvious ahead of time, so it may be difficult to say which kinds of experiments pose the greatest risks or in which cases the benefits outweigh the costs.

Even if a more reliable governance could be constructed, the geographic scope would remain a challenge. Practitioners inclined toward more concerning work could migrate to more permissive jurisdictions. And even if one journal declines to publish a new finding on public safety grounds, a researcher can resubmit to another journal with laxer standards.²⁹

But we believe these challenges are surmountable.

Research governance can adapt to modern challenges. Greater awareness of biosecurity issues can be spread in the scientific community. We can construct better means of risk assessment than blacklists (cf. Lewis et al. (2019)). Broader cooperation can mitigate some of the dangers of the unilateralist’s curse. There is ongoing work in all of these areas, and we can continue to improve practices and policies.

Example reader

What jobs are available?

For our full article on pursuing work in biosecurity, you can read our biosecurity research and policy career review.

If you want to focus on catastrophic pandemics in the biosecurity world, it might be easier to work on broader efforts that have more mainstream support first and then transition to more targeted projects later. If you are already working in biosecurity and pandemic preparedness (or a related field), you might want to advocate for a greater focus on measures that reduce risk robustly across the board, including in the worst-case scenarios.

The world could be doing a lot more to reduce the risk of natural pandemics on the scale of COVID-19. It might be easiest to push for interventions targeted at this threat before looking to address the less likely, but more catastrophic possibilities. On the other hand, potential attacks or perceived threats to national security often receive disproportionate attention from governments compared to standard public health threats, so there may be more opportunities to reduce risks from engineered pandemics under some circumstances.

To get a sense of what kinds of roles you might take on, you can check out our job board for openings related to reducing biological threats. This isn’t comprehensive, but it’s a good place to start:

Our job board features opportunities in biosecurity and pandemic preparedness:

View all opportunities

We thank Gregory Lewis for contributing to this article, and thank Anemone Franz and Elika Somani for comments on the draft.

Learn more

Top recommendations

• The Apollo Program for Biodefense by the Bipartisan Commission on Biodefense
• The Biodefense Budget Breakdown by the Council on Strategic Risks
• Anonymous answers: What are the biggest misconceptions about biosecurity and pandemic prevention?
• Anonymous answers: What are the best ways to fight the next pandemic?

Further recommendations

Resources for general pandemic preparedness

• White House report: American pandemic preparedness
  transforming our capabilities
• The characteristics of pandemic pathogens from CHS (2018)
• Modernizing and expanding outbreak science to support better decision making during public health crises: lessons for COVID-19 and beyond from CHS (2020)
• UK Government’s approach to emerging infectious diseases and bioweapons by Nelson et al. (2019)
• The Global Health Security Agenda
• Some good online sources to keep abreast of developments in the field (primarily from a US perspective) are the GMU Pandora report and the CHS mailing list.

Other resources

• What were the death tolls from pandemics in history? from Our World in Data
• Chapters 3 and 5 of Toby Ord’s book The Precipice: Existential Risk and the Future of Humanity, which discuss the threat of engineered pandemics and other biological threats (2020)
• For a sceptical perspective on the case for extreme risks from engineered pandemics, you can read a series of posts by David Thorstad.
• Engineered pathogens: the opportunities, risks and challenges by Cassidy Nelson in The Biochemist (2019)
• Horsepox synthesis: a case of the unilateralist’s curse? by Gregory Lewis in The Bulletin of the Atomic Scientists (2019)
• Information hazards in biotechnology by Gregory Lewis, Piers Millett, Anders Sandberg, Andrew Snyder-Beattie, and Gigi Gronvall (2019)
• Bridging health and security sectors to address high-consequence biological risks by Cassidy Nelson and Michelle Nalabandian (2019)
• The Precipice‘s appendix, which lists policy and research ideas for reducing existential risks, including from GCBRs
• Greg Lewis’s ultra-rough reading list on GCBRs

Career resources

• Emerging Leaders in Biosecurity Fellowship at CHS
• Marc Lipsitch on choosing a graduate programme or postdoctoral fellowship
• A list of promising places to work, study, and network
• A list of experts to potentially try to work with
• US biosecurity policy resources, think tanks, and fellowships

Podcasts

• 

  Kevin Esvelt on cults that want to kill everyone, stealth vs wildfire pandemics, and how he felt inventing gene drives

  Read more
• 

  The careers and policies that can prevent global catastrophic biological risks, according to world-leading health security expert Dr Tom Inglesby

  Read more
• 

  Dr Greg Lewis on COVID-19 and reducing global catastrophic biological risks

  Read more
• 

  Dr Pardis Sabeti on the Sentinel system for detecting and stopping pandemics

  Read more
• 

  Andy Weber on rendering bioweapons obsolete and ending the new nuclear arms race

  Read more
• 

  Alison Young on how top labs have jeopardised public health with repeated biosafety failures

  Read more
• 

  Marc Lipsitch on whether we’re winning or losing against COVID-19

  Read more
• 

  Tara Kirk Sell on COVID-19 misinformation, who’s done well and badly, and what we should reopen first

  Read more
• 

  Toby Ord on the precipice and humanity’s potential futures

  Read more
• 

  Howie Lempel on why we aren’t worried enough about the next pandemic — and specifically what we can do to stop it.

  Read more
• 

  Beth Cameron fought Ebola for the White House. Now she works to stop something even worse.

  Read more
• 

  Ambassador Bonnie Jenkins on 8 years of combating WMD terrorism

  Read more

The post Preventing catastrophic pandemics appeared first on 80,000 Hours.

Some people say AI research has plateaued. But a lot of evidence from the last year points in the opposite direction:

• New capabilities were developed and emerged
• Research indicates existing AI can accelerate science

And at the same time, important findings about AI safety and risk came out (see below).

AI advances might still stall. Some leaders in the field have warned that a lack of good data, for example, may impede further capability growth, though others disagree. Regardless, growth clearly hasn’t stopped yet.

Meanwhile, the aggregate forecast on Metaculus of when we’ll see the first “general” AI system — which would be highly capable across a wide range of tasks — is 2031.

All of this matters a lot, because AI poses potentially existential risks. We think making sure AI goes well is a top pressing world problem.

If AI advances fast, this work is not only important but urgent.

Here are some of the key developments in AI from the last year:

New AI models and capabilities

OpenAI announced in late December that its new model o3 achieved a large leap forward in capabilities. It builds on the o1 language model (also released in 2024), which has the ability to deliberate about its answers before responding. With this more advanced capability, o3 reportedly:

• Scored a breakthrough 87.5% on ARC-AGI, a test designed to be particularly hard for leading AI systems
• Pushed the frontier of AI software engineering, scoring 71.7% on a key benchmark using real tasks compared to 48.9% accuracy for o1
• Achieved a 25% score on a new (and extremely challenging) FrontierMath benchmark — while previous leading AI models couldn’t get above 2%¹

While not released publicly yet, it seems clear that o3 is the most capable language model we’ve seen. It still has many limitations and weaknesses, but it undermines claims that AI progress stalled in 2024.

It may be the most impressive advance in 2024, but the last year had many other major developments:

• AI video generation gained steam, as OpenAI released Sora for public use and Google DeepMind launched Veo.
• Google DeepMind released AlphaFold 3 — a successor to a Nobel Prize-winning AI system — which can predict how proteins interact with DNA, RNA, and other structures at the molecular level.
• Anthropic introduced the capability for its chatbot Claude to use your computer at your direction.
• AI systems are increasingly able to take audio and visual inputs, and larger amounts of text, while also engaging with users in voice mode.
• By combining the models AlphaProof and AlphaGeometry 2, Google DeepMind was able to use AI to achieve silver medal performance in the International Mathematical Olympiad.
• The Chinese company DeepSeek said that its newest model only cost $5.5 million to train — a dramatic decrease from the reported $100 million OpenAI spent training the comparably capable GPT-4.

And there’s a lot more that could be included here! We won’t be surprised if 2025 and 2026 see many more leaps forward in AI capabilities.

AI helping with science

Recent research indicates that AI can help speed up scientific progress, including AI research itself:

• AI appears to have improved top material science researchers’ output by around 80%, speeding up their ability to generate new ideas.
• AI agents could outperform human experts on ML research engineering tasks when given two hours, though the humans take the lead over longer periods of time.
• Researchers found that language models seem to be able to discover fundamental patterns in the neuroscientific literature, making them better than human experts at predicting the outcomes of new studies in the field.
• AI-discovered molecules appear more likely to pass through phase I clinical trials.

Some key developments in AI risk and safety research

Meanwhile, we’ve seen a mix of encouraging and worrying results in research on AI safety. Here are a few of the important publications this year:

• A recent paper from Anthropic found that AI models can “fake” being aligned with their developers’ intentions when they’re told they’re being trained, only to abandon these behaviours when deployed.
• Apollo Research found that under certain conditions, OpenAI’s o1 model has the capability to scheme to deceive its developers and even pretend to be less capable than it really is.
• In a paper on “Sleeper Agents,” researchers discovered that AI models can have deceptive and unwanted behaviours that standard safety training won’t eliminate.
• A new approach to preventing AI misuse and improving alignment involves using “circuit breakers” to directly target internal representations of harmful outputs, rather than simply trying to train models to refuse to produce harmful outputs.
• Anthropic reported that it successfully identified millions of “features” — patterns of neurons that can be linked to human-understandable concepts — within one of its frontier AI models. This points the way to better understand of how these systems actually work.

What you can do

These developments show the fast pace and potential risks of advancing AI. To help, you can:

• Learn about the the biggest risks AI poses
• Learn about how to carefully communicate about this topic
• Learn how to use AI in your own work
• Consider switching your career to work in AI governance or AI safety
• Consider other potential jobs that can help

We also recommend checking out recent posts from our founder Benjamin Todd on:

• Funding opportunities in AI safety
• How ordinary people can prepare for the possibility of transformative AI in the near future

We plan to continue to cover this topic in the coming year, and we wouldn’t be surprised to see many additional changes and major AI developments. Continue following along with us, and consider sharing this information with your friends by forwarding this email if you find it helpful.

Learn more:

• Can AI Scaling Continue Through 2030? from Epoch AI
• What just happened: A transformative month rewrites the capabilities of AI by Ethan Mollick
• An update from Jack Clark’s “Import AI” newsletter: “OpenAI’s O3 means AI progress in 2025 will be faster than in 2024.”
• Synthetic data is more useful than you think by Lynnette Bye
• Is AI progress slowing down? Making sense of recent technology trends and claims by Arvind Narayanan and Sayash Kapoor
• Things we learned about LLMs in 2024 by Simon Willison
• How A.I. Could Change Science Forever — a video by Cool Worlds

The post What happened with AI in 2024? appeared first on 80,000 Hours.

Here are some of our favourite and most important articles, posts, and podcast episodes we published in 2024:

Articles

Factory farming — There’s a clear candidate for the biggest moral mistake that humanity is currently making: factory farming. We raise and slaughter 1.6-4.5 trillion animals a year on factory farms, causing tremendous amounts of suffering.

The moral status of digital minds — Understanding whether AI systems might suffer, be sentient, or otherwise matter morally is potentially one of the most pressing problems in the world.

Should you work at a frontier AI company? — Working at a frontier AI company is plausibly some people’s highest-impact option, but some roles could be extremely harmful. So it’s critical to be discerning when considering this option — and particularly open to changing course. We’ve previously written about this topic, but explored it in more depth this year while taking account of recent developments, such as prominent departures at OpenAI.

Risks of stable totalitarianism — Some of the worst atrocities have been committed by totalitarian rulers. In the future, the threat posed by these regimes could be even greater.

Nuclear weapons safety and security — Nuclear weapons continue to pose an existential threat to humanity, but there are some promising pathways to reducing the risk.

Other posts

AI for epistemics — Our president and founder, Benjamin Todd, wrote about one of the most exciting ideas he’s heard about recently: using advancing AI technology to improve our decision making and understanding of the world.

Why we get burned out — and what helps — Laura González Salmerón, an 80,000 Hours adviser, wrote about how she experienced burnout in a previous role, and what you can learn from her experience.

What are the biggest misconceptions about biosecurity? — We asked experts about key mistakes they see in the field. They gave us their frank, and sometimes conflicting, answers.

Podcast episodes

Carl Shulman part one and part two — We explored wild answers to the question: what if we develop AI systems that can accomplish everything the most productive humans can?

Meghan Barrett — Can insects suffer? Meghan makes a strong case for taking the possibility very seriously in this genuinely mind-bending episode.

Randy Nesse — Understanding why evolution left many people prone to severe depression and anxiety may help us better manage mood disorders.

Sihao Huang — How can the US and China avoid a destructive AI arms race that no one would win?

Rose Chan Loui — One critic has described OpenAI’s plan to jettison nonprofit status as “the theft of at least the millennium and quite possibly all of human history.” Are they right?

Rachel Gennerster — A pioneer in the field of development economics talked about how we can leverage market forces to drive innovations that can solve climate change, pandemics, and other global problems.

The post 2024 in review: some of our top pieces from this year appeared first on 80,000 Hours.

We spoke to more than a dozen biosecurity experts to understand the space better. We let them give their answers anonymously so that they could feel comfortable speaking their minds.

We don’t agree with everything the experts told us — they don’t even agree with one another! But we think it can be really useful for people who want to learn about or enter this field to understand the ongoing debates and disagreements.

We already published the first article on their answers about misconceptions in biosecurity, and we’re now sharing three more editions, completing this four-part series:

1. AI’s impact on biosecurity

We think one of the world’s most pressing problems is the risk of catastrophic pandemics, and powerful AI could make this risk higher than ever before.

Experts generally agreed that AI developments pose new risks, but there was some disagreement on how big and immediate the threat is.

These are some key quotes from the experts on areas of disagreement:

• “AI may really accelerate biorisk. Unfortunately, I don’t think we have yet figured out great tools to manage that risk.” (Read more)
• “My hot take is that AI is obviously a big deal, but I’m not sure it’s actually as big a deal in biosecurity as it might be for other areas.” (Read more)
• “The timelines in which we will need to tackle major technical challenges have collapsed.” (Read more)
• “Every new technology gets overhyped, and then people realise it’s not as good as we thought it was. We’re in the hype part of that cycle now.” (Read more)

Check out all the answers

2. Navigating information hazards

Experts discussed the tensions around sharing sensitive biosecurity information. While some advocate for more openness to enable better problem solving, others emphasise the need to carefully control potentially dangerous information.

Here are some key quotes from different experts:

• “An area that’s been very neglected is protecting researchers who are out in the field collecting samples from wild animals, whether environmental samples or biomedical samples.” (Read more)
• “Overall, I think there is an unfortunate lack of focus on interventions that will be most valuable in a pandemic scenario with a highly transmissible, high-mortality pathogen.” (Read more)
• “I think that people think a lot about public health, and we all want vaccines for all the diseases. But which kinds of vaccines we should prioritise really matters.” (Read more)
• “[There are] two things that I would prioritise in terms of young people working in this field: models to predict risk and then global biorisk monitoring.” (Read more)

Check out all the answers

3. Neglected interventions in pandemic preparedness

Several key but underappreciated areas emerged from our expert discussions.

For example: better personal protective equipment (PPE) is desperately needed. Multiple experts highlighted this as a severely neglected priority. Current PPE often fits badly and isn’t very effective against highly transmissible pathogens.

But different experts also expressed concern about other neglected areas:

• “An area that’s been very neglected is protecting researchers who are out in the field collecting samples from wild animals, whether environmental samples or biomedical samples.” (Read more)
• “Overall, I think there is an unfortunate lack of focus on interventions that will be most valuable in a pandemic scenario with a highly transmissible, high-mortality pathogen.” (Read more)
• “I think that people think a lot about public health, and we all want vaccines for all the diseases. But which kinds of vaccines we should prioritise really matters.” (Read more)
• “[There are] two things that I would prioritise in terms of young people working in this field: models to predict risk and then global biorisk monitoring.” (Read more)

Check out all the answers

And we give special thanks to Anemone Franz and Tessa Alexanian for working on this series for us, as well as to all the anonymous experts who participated!

Learn more:

• Our article on Preventing catastrophic pandemics
• What are the biggest misconceptions about biosecurity and pandemic risk?
• AI and Biorisk: An Explainer from the Center for Security and Emerging Technology
• The GCBR Organization Updates newsletter from Anemone Franz and Tessa Alexanian

The post What are experts in biosecurity worried about? appeared first on 80,000 Hours.

1. Humanity may soon grapple with many AI systems that could be conscious

In 2020, more than 1,000 professional philosophers were asked whether they believed then-current AI systems¹ were conscious.² Consciousness, in this context, is typically understood as meaning having phenomenal experiences that feel like something, like the experience of perception or thinking.

Less than 1% said that yes, some then-current AI systems were conscious, and about 3% said they were “leaning” toward yes. About 82% said no or leaned toward no.

But when asked about whether some future AI systems would be conscious, the bulk of opinion flipped.

Nearly 40% were inclined to think future AI systems would be conscious, while only about 27% were inclined to think they wouldn’t be.³

A survey of 166 attendees at the Association for the Scientific Study of Consciousness annual conference asked a similar question. Sixty-seven percent of attendees answered “definitely yes” or “probably yes” when asked “At present or in the future, could machines (e.g. robots) have consciousness?”⁴

The plurality of philosophers and majority of conference attendees might be wrong. But we think these kinds of results make it very difficult to rule out the possibility of conscious AI systems, and we think it’s wrong to confidently assert that no AI could ever be conscious.

Why might future AI systems be conscious? This question is wide open, but researchers have made some promising steps toward providing answers.

One of the most rigorous and comprehensive studies we’ve seen into this issue was published in August 2023 with 19 authors, including experts in AI, neuroscience, cognitive science, and philosophy.⁵ They investigated a range of properties⁶ that could indicate that AI systems are conscious.

The authors concluded: “Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.”

They also found that, according to some plausible theories of consciousness, “conscious AI systems could realistically be built in the near term.”

Philosopher David Chalmers has suggested that there’s a (roughly) 25% chance that in the next decade we’ll have conscious AI systems.⁷

Creating increasingly powerful AI systems — as frontier AI companies are currently trying to do — may require features that some researchers think would indicate consciousness. For example, proponents of global workspace theory argue that animals have conscious states when their specialised cognitive systems (e.g. sensory perception, memories, etc.) are integrated in the right way into a mind and share representations of information in a “global workspace.” It’s possible that creating such a “workspace” in an AI system would both increase its capacity to do cognitive tasks and make it a conscious being. Similar claims might be made about other features and theories of consciousness.

And it wouldn’t be too surprising if increasing cognitive sophistication led to consciousness in this way, because humans’ cognitive abilities seem closely associated with our capacity for consciousness.⁸ (Though, as we’ll discuss below, intelligence and consciousness are distinct concepts.)⁹

How soon could conscious AI systems arrive? We’re not sure. But we do seem to be on track to make a huge number of more advanced AI systems in the coming decades.

Another recent survey found that the aggregate forecast of the thousands of AI researchers put a 50% chance that we’ll have AI systems that are better than humans in every possible task by 2047.¹⁰

If we do produce systems that capable, there will be enormous incentives to produce many of them. So we might be looking at a world with a huge number of highly advanced AI systems, which philosophers think could be conscious, pretty soon.

The public may already be more inclined to assign attributes like consciousness to AI systems than experts. Around 18% of US respondents in a 2023 survey believed current AI systems are sentient.¹¹

This phenomenon might already have real effects on people’s lives. Some chatbot services have cultivated devoted user bases that engage in emotional and romantic interactions with AI-powered characters, with many seeming to believe — implicitly or explicitly — that the AI may reciprocate their feelings.¹²

As people increasingly think AI systems may be conscious or sentient, we’ll face the question of whether humans have any moral obligations to these digital minds. Indeed, among the 76% of US survey respondents who said AI sentience was possible (or that they weren’t sure if it was possible), 81% said they expected “the welfare of robots/AIs to be an important social issue” within 20 years.

We may start to ask:

• Are certain methods of training AIs cruel?
• Can we use AIs for our own ends in an ethical way?
• Do AI systems deserve moral and political rights?

These may be really difficult questions, which involve complex issues in philosophy, political theory, cognitive science, computer science, machine learning, and other fields. A range of possible views about these issues could be reasonable. We could also imagine getting the answers to these questions drastically wrong.

And with economic incentives to create these AI systems, and many humans — including experts in the field — prepared to believe they could be conscious, it seems unlikely we will be able to avoid the hard questions.

2. Creating digital minds could go very badly — or very well

One thing that makes this problem particularly thorny is the risk of both over-attributing and under-attributing moral status.

Believing AI systems aren’t worthy of moral consideration when they are and the reverse could both be disastrous. There are potential dangers for both digital minds and for humans.

Dangers for digital minds

If we falsely think digital minds don’t have moral status when they do, we could unknowingly force morally significant beings into conditions of servitude and extreme suffering — or otherwise mistreat them.

Some ways this could happen include:

• The process of aligning or controlling digital minds to act in their creators’ interests could involve suffering, frequent destruction, or manipulation in ways that are morally wrong.¹³
• Our civilisation could choose to digitally simulate its own histories or other scenarios, in which fully simulated digital minds might suffer in extreme amounts — a possibility Nick Bostrom has raised.¹⁴
• Philosophers Eric Schwitzgebel and Mara Garza have argued that even if we avoid creating large-scale suffering, we should be concerned about a future full of “cheerful servant” digital minds. They might in principle deserve rights and freedoms, but we could design them to seem happy with oppression and disregard. On many moral views, this could be deeply unjust.

These bad outcomes seem most likely to happen by accident or out of ignorance, perhaps by failing to recognise digital sentience. But some people might knowingly cause large numbers of digital minds to suffer out of indifference, sadism, or some other reason. And it’s possible some AI systems might cause other AI systems to suffer, perhaps as a means of control or to further their own objectives.

Dangers for humans

There are also dangers to humans. For example, if we believe AI systems are sentient when they are not, and when they in fact lack any moral status, we could do any of the following:

• We could waste resources trying to meet the needs and desires of AI systems even if there’s no real reason to do so.
  • This could be costly, and it could take resources away from causes that genuinely need them.
• We might choose to give AI systems freedom, rather than control them. This plausibly could lead to an existential catastrophe.
  • For example, key decision makers might believe that the possibility discussed in the previous section that AI alignment and AI control could be harmful to digital minds. If they were mistaken, they might forgo necessary safety measures in creating advanced AI, and then that AI could seek to disempower humanity. If the decision makers are correct about the moral risks to digital minds, then the wise choice might be to delay development until we have enough knowledge to pursue AI development safely for everyone.
• Even more speculatively, humanity might decide at some point in the future to “upload” our minds — choosing to be replaced by digital versions of ourselves. If it turned out that these uploaded versions of our minds wouldn’t be conscious, this could turn out to be a severe mistake.¹⁵

It’s hard to be confident in the plausibility of any particular scenario, but these kinds of cases illustrate the potential scale of the risks.

Other dangers

If the world is truly unfortunate, we could even make both kinds of errors at once. We could have charismatic systems (which perhaps act in a humanlike way) that we believe are sentient when they’re not. At the same time, we could have less charismatic but sentient systems whose suffering and interests are completely disregarded. For example, maybe AI systems that don’t talk will be disregarded, even if they are worthy of just as much moral concern as others.⁹

We could also make a moral mistake by missing important opportunities. It’s possible we’ll have the opportunity to create digital minds with extremely valuable lives with varied and blissful experiences, continuing indefinitely. Failing to live up to this potential could be a catastrophic mistake on some moral views. And yet, for whatever reason, we might decide not to.

Things could also go well

This article is primarily about encouraging research to reduce major risks. But it’s worth making clear that we think there are many possible good futures:

• We might eventually create flourishing, friendly, joyful digital minds with whom humanity could share the future.
• Or we might discover that the most useful AI systems we can build don’t have moral status, and we can justifiably use them to improve the world without worrying about their wellbeing.

What should we take from all this? The risks of both over-attribution and under-attribution of sentience and moral status mean that we probably shouldn’t simply stake out an extreme position and rally supporters around it. We shouldn’t, for example, declare that all AI systems that pass a simple benchmark must be given rights equivalent to humans or insist that any human’s interests always come before those of digital minds.

Instead, our view is that this problem requires much more research to clarify key questions, to dispel as much uncertainty as possible, and to determine the best paths forward despite the remaining uncertainty we’ll have. This is the best hope we have of avoiding key failure modes and increasing the chance that the future goes well.

But we face a lot of challenges in doing this, which we turn to next.

3. We don’t know how to assess the moral status of AI systems

So it seems likely that we’ll create conscious digital minds at some point, or at the very least that many people may come to believe AI systems are conscious.

The trouble is that we don’t know how to figure out if an AI system is conscious — or whether it has moral status.

Even with animals, the scientific and philosophical community is unsure. Do insects have conscious experiences? What about clams? Jellyfish? Snails?¹⁶

And there’s also no consensus about how we should assess a being’s moral status. Being conscious may be all that’s needed for being worthy of moral consideration, but some think it’s necessary to be sentient — that is, being able to have good and bad conscious experiences. Some think that consciousness isn’t even necessary to have moral status because an individual agent may, for example, have morally important desires and goals without being conscious.

So we’re left with three big, open questions:

• What characteristics would make a digital mind a moral patient?
• Can a digital mind have those characteristics (for example, being conscious)?
• How do we figure out if any given AI has these characteristics?

These questions are hard, and it’s not even always obvious what kind of evidence would settle them.

Some people believe these questions are entirely intractable, but we think that’s too pessimistic. Other areas in science and philosophy may have once seemed completely insoluble, only to see great progress when people discover new ways of tackling the questions.

Still, the state of our knowledge on these important questions is worryingly poor.

There are many possible characteristics that give rise to moral status

Some think moral status comes from having:¹⁷

• Consciousness: the capacity to have subjective experience, but not necessarily valenced (positive or negative) experience. An entity might be conscious if it has perceptual experiences of the world, such as experiences of colour or physical sensations like heat. Often consciousness is described as the phenomenon of there being something it feels like to be you — to have your particular perspective on the world, to have thoughts, to feel the wind on your face — in a way that inanimate objects like rocks seem to completely lack. Some people think it’s all you need to be a moral patient, though it’s arguably hard to see how one could harm or benefit a conscious being without valenced experiences.¹⁸
• Sentience: the capacity to have subjective experience (that is, consciousness as just defined) and the capacity for valenced experiences, i.e. good or bad feelings. Physical pleasure and pain are the typical examples of valenced, conscious experiences, but there are others, such as anxiety or excitement.
• Agency: the ability to have and act on goals, reasons, or desires, or something like them. An entity might be able to have agency without being conscious or sentient. And some believe even non-conscious beings could have moral status by having agency, since they could be harmed or benefited depending on whether their goals are frustrated or achieved.¹⁹
• Personhood: personhood is a complex and debated term that usually refers to a collection of properties, which often include sentience, agency, rational deliberation, and the ability to respond to reasons. Historically, personhood has sometimes been considered a necessary and sufficient criterion for moral status or standing, particularly in law. But this view has become less favoured in philosophy as it leaves no room for obligations to most nonhuman animals, human babies, and some others.
• Some combination of the above or other traits.²⁰

We think it’s most plausible that any being that feels good or bad experiences — like pleasure or pain — is worthy of moral concern in their own right.²¹

We discuss this more in our article on the definition of social impact, which touches on the history of moral philosophy.

But we don’t think we or others should be dogmatic about this, and we should look for sensible approaches to accommodate a range of reasonable opinions on these controversial subjects.²²

Many plausible theories of consciousness could include digital minds

There are many theories of consciousness — more than we can name here. What’s relevant is that some, though not all, theories of consciousness do imply the possibility of conscious digital minds.

This is only relevant if you think consciousness (or sentience, which includes consciousness as a necessary condition) is required for moral status. But since this is a commonly held view, it’s worth considering these theories and their implications. (Note, though, that there are often many variants of any particular theory.)

Some theories that could rule out the possibility of conscious digital minds:²³

• Biological theories: These hold that consciousness is inherently tied to the biological processes of the brain that can’t be replicated in computer hardware.²⁴
• Some forms of dualism: Dualism, particularly substance dualism, holds that consciousness is a non-physical substance distinct from the physical body and brain. It is often associated with religious traditions. While some versions of dualism would accommodate the existence of conscious digital minds, others could rule out the possibility.²⁵

Some theories that imply digital minds could be conscious:

• Functionalism: This theory holds that mental states are defined by their functional roles — how they process inputs, outputs, and interactions with other mental states. Consciousness, from this perspective, is explained not by what a mind is made of but by the functional organisation of its constituents. Some forms of functionalism, such as computational functionalism, strongly suggest that digital minds could be conscious, as they imply that if a digital system replicates the functional organisation of a conscious brain, it could also have conscious mental experiences.
• Global workspace theory: GWT says that consciousness is the result of integrating information in a “global workspace” within the brain, where different processes compete for attention and are broadcast to other parts of the system. If a digital mind can replicate this global workspace architecture, GWT would support the possibility that the digital mind could be conscious.
• Higher-order thought theory: HOT theory holds that consciousness arises when a mind has thoughts about its own mental states. On this view, it’s plausible that if a digital mind could be designed to have thoughts about its own processes and mental states, it would therefore be conscious.
• Integrated information theory: IIT posits that consciousness corresponds to the level of integrated information within a system. A system is conscious to the extent that it has a high degree of integrated information (often denoted ‘Φ’). Like biological systems, digital minds could potentially be conscious if they integrate information with sufficiently high Φ.²⁶

Some theories that are agnostic or unclear about digital minds:

• Quantum theories of consciousness: Roger Penrose theorises that consciousness is tied to quantum phenomena within the brain.²⁷ If so, digital minds may not be able to be conscious unless their hardware can replicate these quantum processes.
• Panpsychism: Panpsychism is the view that consciousness is a fundamental property of the universe. Panpsychism doesn’t rule out digital minds being conscious, but it doesn’t necessarily provide a clear framework for understanding how or when a digital system might become conscious.
• Illusionism or eliminativism: Illusionists or eliminativists argue that consciousness, as it is often understood, is an illusion or unnecessary folk theory. Illusionism doesn’t necessarily rule out digital minds being “conscious” in some sense, but it suggests that consciousness isn’t what we usually think it is. But many illusionists and eliminativists don’t want to deny that humans and animals can have moral status according to their views — in which case they might also be open to the idea that digital minds could likewise have moral status. (See some discussion of this issue here.)

It can be reasonable, especially for experts with deep familiarity of the debates, to believe much more strongly in one theory than the others. But given the amount of disagreement about this topic among experts, and the lack of solid evidence in one direction or another, and since many widely supported theories imply that digital minds could be conscious (or at least don’t contradict the idea), we don’t think it’s reasonable to completely rule out the possibility of conscious digital minds.²⁸

We think it makes sense to put at least 5% on the possibility. Speaking as the author of this piece, based on my subjective impression of the balance of the arguments, I’d put the chance at around 50% at least.

We can’t rely on what AI systems tell us about themselves

Unfortunately, we can’t just rely on self-reports from AI systems about whether they’re conscious or sentient.

In the case of large language models like LaMDA, we don’t know why it claimed under certain conditions to Blake Lemoine that it was sentient,²⁹ but it resulted in some way from having been trained on a huge body of existing texts.³⁰

LLMs essentially learn patterns and trends in these texts, and then respond to questions on the basis of these extremely complex patterns of associations. The capabilities produced by this process are truly impressive — though we don’t fully understand how this process works, the outputs end up reflecting human knowledge about the world. As a result, the models can perform reasonably well at tasks involving human-like reasoning and making accurate statements about the world. (Though they still have many flaws!)

However, the process of learning from human text and fine-tuning might not have any relationship with what it’s actually like to be a language model. Rather, the responses seem more likely to mirror our own speculations and lack of understanding about the inner workings and experiences of AI systems.³¹

That means we can’t simply trust an AI system like LaMDA when it says it’s sentient.³²

Researchers have proposed methods to assess the internal states of AI systems and whether they might be conscious or sentient, but all of these methods have serious drawbacks, at least at the moment:

• Behavioural tests: we might try to figure out if an AI system is conscious by observing its outputs and actions to see if they indicate consciousness. The familiar Turing Test is one example; researchers such as Susan Schneider have proposed others. But since such tests can likely be gamed by a smart enough AI system that is nevertheless not conscious, even sophisticated versions may leave room for reasonable doubt.
• Theory-based analysis: another method involves assessing the internal structure of AI systems and determining whether they show the “indicator properties” of existing theories of consciousness. The paper discussed above by Butlin et al. took this approach. While this method avoids the risk of being gamed by intelligent but non-conscious AIs, it is only as good as the (highly contested) theories it relies on and our ability to discern the indicator properties.
• Animal analogue comparisons: we can also compare the functional architecture of AI systems to the brains and nervous systems of animals. If they’re closely analogous, that may be a reason to think the AI is conscious. Bradford Saad and Adam Bradley have proposed a test along these lines. However, this approach could miss out on conscious AI systems with internal architectures that are totally different, if such systems are possible. It’s also far from clear how close the analogue would have to be in order to indicate a significant likelihood of consciousness.
• Brain-AI interfacing: This is the most speculative approach. Schneider suggests an actual experiment (not just a thought experiment) where someone decides to replace parts of their brain with silicon chips that perform the same function. If this person reports still feeling conscious of sensations processed through the silicon portions of their brain, this might be evidence of the possibility of conscious digital minds. But — even if we put aside the ethical issues — it’s not clear that such a person could reliably report on this experience. And it wouldn’t necessarily be that informative about digital minds that are unconnected to human brains.

We’re glad people are proposing first steps toward developing reliable assessments of consciousness or sentience in AI systems, but there’s still a long way to go. We’re also not aware of any work that assesses whether digital minds might have moral status on a basis other than being conscious or sentient.³³

The strongest case for the possibility of sentient digital minds: whole brain emulation

What’s the best argument for thinking it’s possible that AIs could be conscious, sentient, or otherwise worthy of moral concern?

Here’s the case in its simplest form:

1. It is possible to emulate the functions of a human brain in a powerful enough computer.
2. Given it’d be functionally equivalent, this brain emulation would plausibly report being sentient, and we’d have at least some reason to think it was correct given the plausibility of functionalist accounts of consciousness.
3. Given this, it would be reasonable to regard this emulation as morally worthy of concern comparable to a human.
4. If this is plausible, then it’s also plausible that there are other forms of artificial intelligence that would meet the necessary criteria for being worthy of moral concern. It would be surprising if artificial sentience was possible, but only by imitating the human mind exactly.

Any step in this reasoning could be false, but we think it’s more likely than not that they’re each true.

Emulating a human brain³⁴ still seems very far away, but there have been some initial steps. The project OpenWorm has sought to digitally emulate the function of every neuron of the C. elegans worm, a tiny nematode. If successful, the emulation should be able to recreate the behaviour of the actual animals.

And if the project is successful, it could be scaled up to larger and more complex animals over time.³⁵ Even before we’re capable of emulating a brain on a human scale, we may start to ask serious questions about whether these simpler emulations are sentient. A fully emulated mouse brain, which could show behaviour like scurrying toward food and running away from loud noises (perhaps in a simulated environment or in a robot), may intuitively seem sentient to many observers.

And if we did have a fully emulated human brain, in a virtual environment or controlling a robotic body, we expect it would insist — just like a human with a biological brain — that it was as conscious and feeling as anyone else.

Of course, there may remain room for doubt about emulations. You might think that only animal behaviour generated by biological brains, rather than computer hardware, would be a sign of consciousness and sentience.³⁶

But it seems hard to be confident in that perspective, and we’d guess it’s wrong. If we can create AI systems that display the behaviour and have functional analogues of anything that would normally indicate sentience in animals, then it would be hard to avoid thinking that there’s at least a decent chance that the AI is sentient.

And if it is true that an emulated brain would be sentient, then we should also be open to the possibility that other forms of digital minds could be sentient. Why should strictly brain-like structures be the only possible platform for sentience? Evolution has created organisms that display impressive abilities like flight that can be achieved technologically via very different means, like helicopters and rockets. We would’ve been wrong to assume something has to work like a bird in order to fly, and we might also be wrong to think something has to work like a brain to feel.

4. The scale of this issue might be enormous

As mentioned above, we might mistakenly grant AI systems freedom when it’s not warranted, which could lead to human disempowerment and even extinction. In that way, the scale of the risk can be seen as overlapping with some portion of the total risk of an AI-related catastrophe.

But the risks to digital minds — if they do end up being worthy of moral concern — are also great.

There could be a huge number of digital minds

With enough hardware and energy resources, the number of digital minds could end up greatly outnumbering humans in the future.³⁷ This is for many reasons, including:

• Resource efficiency: Digital minds may end up requiring fewer physical resources compared to biological humans, allowing for much higher population density.
• Scalability: Digital minds could be replicated and scaled much more easily than biological organisms.
• Adaptability: The infrastructure for digital minds could potentially be adapted to function in many more environments and scenarios than humans can.
• Subjective time: We may choose to run digital minds at high speeds, and if they’re conscious, they may be able to experience the equivalent of a human life in a much shorter time period — meaning there could be effectively more “lifetimes” of digital minds even with the same number of individuals.³⁸
• Economic incentives: If digital minds prove useful, there will be strong economic motivations to create them in large numbers.

According to one estimate, the future could hold up to 10^43 human lives, but up to 10^58 possible human-like digital minds.³⁹ We shouldn’t put much weight on these specific figures, but they give a sense for just how comparatively large future populations of digital minds could be.

Our choices now might have long-lasting effects

It’s possible, though far from certain, that the nature of AI systems we create could be determined by choices humanity makes now and persist for a long time. So creating digital minds and integrating them into our world could be extremely consequential — and making sure we get it right may be urgent.

Consider the following illustrative possibility:

This kind of story seems more plausible than it might otherwise be in part because the rise of factory farming has followed a similar path. Humanity never collectively decided that a system of intensive factory farming, inflicting vast amounts of harm and suffering on billions and potentially trillions of animals a year, was worth the harm or fundamentally just. But we built up such a system anyway, because individuals and groups were incentivised to increase production efficiency and scale, and they had some combination of ignorance of and lack of concern for animal suffering.

It’s far from obvious that we’ll do this again when it comes to AI systems. The fact that we’ve done it in the case of factory farming — not to mention all the ways humans have abused other humans — should alarm us, though. When we are in charge of beings that are unlike us, our track record is disturbing.

The risk of persistently bad outcomes in this kind of case suggests that humanity should start laying the groundwork to tackle this problem sooner rather than later, because delayed efforts may come too late.

Why could such a bad status quo persist? One reason for doubt is that a world that is creating many new digital minds, especially in a short time period, is one that is likely experiencing a lot of technological change and social disruption. So we shouldn’t expect the initial design of AI systems and digital minds to be that critical.

But there are reasons that suffering digital minds might persist, even if there are alternative options that could’ve avoided such a terrible outcome (like designing systems that can’t suffer).⁴⁰ Possible reasons include:

1. A stable totalitarian regime might prevent attempts to shift away from a status quo that keeps them in power and reflects their values.
2. Humans might seek to control digital minds and maintain a bad status quo in order to avoid an AI takeover.

It’s far from obvious that a contingent, negative outcome for digital minds would be enduring. Understanding this question better could be an important research avenue. But the downsides are serious enough, and the possibility plausible enough, that we should take it seriously.

Adding it up

There could be many orders of magnitude more digital minds than humans in the future. And they could potentially matter a lot.

Because of this, and because taking steps to better understand these issues and inform the choices we make about creating digital minds now might have persistent effects, the scale of the problem is potentially vast. It is plausibly similar in scale to factory farming, which also involves the suffering of orders of magnitude more beings than humans.

If the choices we make now about digital minds can have persisting and positive effects for thousands or millions of years in the future, then this problem would be comparable to existential risks. It’s possible that our actions could have such effects, but it’s hard to be confident. Finding interventions with effects that persist over a long time is rare. I wouldn’t put the likelihood that the positive effects of those trying to address this problem will persist that long at more than 1 in 1,000.

Still, even with a low chance of having persistent effects, the value in expectation of improving the prospects for future digital minds could be as high or even greater than at least some efforts to reduce existential risks. However, I’m not confident in this judgement, and I wouldn’t be surprised if we change our minds in either direction as we learn more. And even if the plausible interventions only have more limited effects, they could still be very worthwhile.

5. Work on this problem is neglected but seems tractable

Despite the challenging features of this problem, we believe there is substantial room for progress.

There is a small but growing field of research and science dedicated to improving our understanding of the moral status of digital minds. Much of the work we know of is currently being done in academia, but there may also at some point be opportunities in government, think tanks, and AI companies — particularly those developing the frontier of AI technology.

Some people focus their work primarily at addressing this problem, while others work on it along with a variety of other related problems, such as AI policy, catastrophic risk from AI, mitigating AI misuse, and more.

As of mid-2024, we are aware of maybe only a few dozen people working on this issue with a focus on the most impactful questions.⁴¹ We expect interest in these issues to grow over time as AI systems become more embedded in our lives and world.

Here are some of the approaches to working on this problem that seem most promising:

Impact-guided research

Most of the most important work to be done in this area is probably research, with a focus on the questions that seem most impactful to address.

Philosophers Andreas Mogensen, Bradford Saad, and Patrick Butlin have detailed some of the key priority research questions in this area:

• How can we assess AI systems for consciousness?
• What indications would suggest that AI systems or digital minds could have valenced (good or bad) experiences?
• How likely is it that non-biological systems could be conscious?
• What principles should govern the creation of digital minds, ethically, politically, and legally (given our uncertainty on these questions)?
• Which mental characteristics and traits are related to moral status, and in what ways?
• Are there any ethical issues with efforts to align AI systems?

The Sentience Institute has conducted social science research aimed at understanding how the public thinks about digital minds. This can inform efforts to communicate more accurately about what we know about their moral status and inform us about what kinds of policies are viable.

We’re also interested to see more research on the topic of human-AI cooperation, which may be beneficial for both reducing AI risk and reducing risks to digital minds.

Note, though, that there are many ways to pursue all of these questions badly — for example, by simply engaging in extensive and ungrounded speculation. If you’re new to this field, we recommend reading the work of the most rigorous and careful researchers working on the topic and trying to understand how they approach these kinds of questions. If you can, try to work with these researchers or others like them so you can learn from and build on their methods. And when you can, try to ground your work in empirical science.

Technical approaches

While there are important conceptual issues that need to be addressed in this problem area, we think much, if not most, of the top priority work is technical.

So people with experience in machine learning and AI will have a lot to contribute.

For example, research in the AI sub-field of interpretability — which seeks to understand and explain the decisions and behaviour of advanced AI models — may be useful for getting a better grasp on the moral status of these systems. This research has mostly focused on questions about model behaviour rather than questions that are more directly related to moral status, but it’s possible that could change.

Some forms of technical AI research could be counterproductive, however. For example, efforts to intentionally create new AI systems that might instantiate plausible theories of consciousness could be very risky. This kind of research could force us to confront the problem we’re faced with — how should we treat digital minds that might merit moral concern? — with much less preparation than we might otherwise have.

So we favour doing research that increases our ability to understand how AI systems work and assess their moral status, as long as it isn’t likely to actively contribute to the development of conscious digital minds.

One example of this kind of work is a paper from Robert Long and Ethan Perez. They propose techniques to assess whether an AI system can accurately report on its own internal states. If such techniques were successful, they might help us use an AI system’s self-reports to determine whether it’s conscious.

We also know some researchers are excited about using advances in AI to improve our epistemics and our ability to know what’s true. Advances in this area could shed light on important questions, like whether certain AI systems are likely to be sentient.

Policy approaches

At some point, we may need policy, both at companies and from governments, to address the moral status of digital minds, perhaps by protecting the welfare and rights of AI systems.

But because our understanding of this area is so limited at the moment, policy proposals should likely be relatively modest and incremental.

Some researchers have already proposed a varied range of possible and contrasting policies and practices:

• Jeff Sebo and Robert Long have proposed that we should “extend moral consideration to some AI systems by 2030” — and likely start preparing to do so now.
• Ryan Greenblatt, who works at Redwood Research, proposed several practices for safeguarding AI welfare, including communication with AIs about their preferences, creating “happy” personas when possible, and limiting the uses of more intelligent AIs and running them for less time on the margin.
• Jonathan Birch has proposed a licensing scheme for companies that might create digital minds that could plausible be sentient, even if they aren’t intending to do so. To get a licence, they would have to agree to a code of conduct, which would include transparency standards.
• Thomas Metzinger has proposed an outright ban until 2050 on any research that directly intends to or knowingly takes the risk of creating artificial consciousness.
• Joanna Bryson thinks we should have a legal system that prevents the creation of AI systems with their own needs and desires.⁴²
• Susan Schneider thinks there should be regular testing of AI systems for consciousness. If they’re conscious, or if it’s unclear but there’s some reason to think they might be conscious, she says we should give them the same protections we’d give other sentient beings.⁴³

In its 2023 survey, the Sentience Institute found that:

• Nearly 70% of respondents favoured banning the development of sentient AIs.
• Around 40% favoured a bill of rights to protect sentient AIs, and around 43% said they favour creating welfare standards to protect the wellbeing of all AIs.

There is some precedent for restricting the use of technology in certain ways if it raises major ethical risks, including the bans on human cloning and human germline genome editing.

We would likely favour:

• Government-funded research into the questions above: the private sector is likely to under-invest in efforts to better understand the moral status of digital minds, so government and philanthropic resources may have to fill the gap.
• Recognising the potential welfare of AI systems and digital minds: policy makers could follow the lead of the UK’s Animal Welfare (Sentience) Act of 2022, which created an Animal Sentience Committee to report on how government policies “might have an adverse effect on the welfare of animals as sentient beings.” Similar legislation and committees could be established to consider problems relating to the moral status of digital minds, while recognising that questions about their sentience are unresolved in this case.

We’re still in the early stages of thinking about policy on these matters, though, so it’s very likely we haven’t found the best ideas yet. As we learn more and make progress on the many technical and other issues, we may develop clear ideas about what policies are needed. Policy-focused research aimed at navigating our way through the extreme uncertainty could be valuable now.

Some specific AI policies might be beneficial for reducing catastrophic AI risks as well as improving our understanding of digital minds. External audits and evaluations might, for instance, assess both the risk and moral status of AI systems. And some people favour policies that would altogether slow down progress on AI, which could be justified to reduce AI risk and reduce the risk that we create digital minds worthy of moral concern before we understand what we’re doing.

Summing up so far

To sum up:

1. Humanity will likely soon have to grapple with the moral status of a growing number of increasingly advanced AI systems
2. Creating digital minds could go very badly or very well
3. We don’t know how to assess the moral status of AI systems
4. The scale of the problem might be enormous
5. Work on this problem is neglected but tractable

We think this makes it a highly pressing problem, and we’d like to see a growing field of research devoted to working on it.

We also think this problem should be on the radar for many of the people working on similar and related problems. In particular, people working on technical AI safety and AI governance should be aware of the important open questions about the moral status of AI systems themselves, and they should be open to including considerations about this issue in their own deliberations.

Arguments against the moral status of digital minds as a pressing problem

Two key cruxes

We think the strongest case against this being a pressing problem would be if you believe both that:

• It’s highly unlikely that digital minds could ever be conscious or have moral status.
• It’s highly unlikely society and decision makers will come to mistakenly believe that digital minds have moral status in a way that poses a significant risk to the future of humanity.

If both of those claims were correct, then the argument of this article would be undermined. However, we don’t think they’re correct, for all the reasons given above.

The following objections may also have some force against working on this problem. We think some of them do point to difficulties with this area. However, we don’t think they’re decisive.

What can you do to help?

There aren’t many specific job openings in this area yet, though we’ve known of a few. And there are several ways you can contribute to this work and position yourself for impact.

Take concrete next steps

Early on in your career, you may want to spend several years doing the following:

• Further reading and study
  • Explore comprehensive reading lists on consciousness, AI ethics, and moral philosophy. You can start with the learn more section at the bottom of this article.
  • Stay updated on advancements in AI, the study of consciousness, and their potential implications for the moral status of digital minds.
• Gain relevant experience
  • Seek internships or research assistant positions with academics working on related topics.
  • Contribute to AI projects and get experience with machine learning techniques.
  • Participate in online courses, reading groups, and workshops on AI safety, AI ethics, and philosophy of mind.
• Build your network
  • Attend conferences and seminars on AI safety, consciousness studies, and related fields.
  • Engage with researchers and organisations working on these issues, for example:
    • Robert Long and Kathleen Finlinson at Eleos AI
    • Kyle Fish, an AI welfare researcher at Anthropic
    • Jeff Sebo at the NYU Center for Mind, Ethics, and Policy
    • Jonathan Birch at the London School of Economics and Political Science
    • Patrick Butlin at the Global Priorities Institute
    • Derek Shiller at Rethink Priorities, which has announced a project on digital consciousness
    • Joe Carlsmith at Open Philanthropy
    • Any of the other researchers referenced in the learn more section of this article
• Start your own research
  • Begin writing essays or blog posts exploring issues around the moral status of digital minds.
  • Propose research projects to your academic institution or seek collaborations with established researchers.
  • Consider submitting papers to relevant conferences or journals to establish yourself in the field.

Aim for key roles

You may want to eventually aim to:

• Become a researcher
  • Develop a strong foundation in a relevant field, such as philosophy, cognitive science, cognitive neuroscience, machine learning, neurobiology, public policy, and ethics.
  • Pursue advanced degrees in these areas and establish your credibility as an expert.
  • Familiarise yourself with the relevant debates and literature on consciousness, sentience, and moral philosophy, and the important details of the disciplines you’re not an expert in.
  • Build strong analytical and critical thinking skills, and hone your ability to communicate complex ideas clearly and persuasively.
  • Read our article on developing your research skills for more.
• Help build the field
  • Identify gaps in current research and discourse.
  • Network with other researchers and professionals interested in this area.
  • Organise conferences, workshops, or discussion groups on the topic.
  • Consider roles in organisation-building or earning to give to support research initiatives.
  • To learn more, read our articles on organisation-building and communication skills.

If you’re already an academic or researcher with expertise in a relevant field, you could consider spending some of your time on this topic, or perhaps refocusing your work on particular aspects of this problem in an impact-focused way.

If you are able to establish yourself as a key expert on this topic, you may be able to deploy this career capital to have a positive influence on the broader conversation and affect decisions made by policy makers and industry leaders. Also, because this field is so neglected, you might be able to do a lot to lead the field relatively early on in your career.

Pursue AI technical safety or AI governance

Because this field is underdeveloped, you may be best off to pursue a career in the currently more established (though also still relatively new) paths of AI safety and AI governance work, and use the experience you gain there as a jumping off point (or work at the intersection of the fields).

You can read our career reviews of each to find out how to get started:

• AI governance and policy
• AI safety technical research

Where to work

• Academia
  • You can pursue research and teaching positions in philosophy, technology policy, cognitive science, AI, or related fields.
• AI companies
  • With the right background, you might want to work at leading AI companies developing frontier models.
  • There might even be roles that are specifically focused on better understanding the status of digital minds and their ethical implications, such as Kyle Fish’s role at Anthropic, mentioned above.
    • We think it’s possible a few other similar roles will be filled at other AI companies, and there may be more in the future.
  • You may also seek to work in AI safety, policy, and security roles. But deciding to work for frontier AI companies is a complex topic, so we’ve written a separate article that tackles that issue in more depth.
• Governments, think tanks, and nonprofits
  • Join organisations focused on AI governance and policy, and contribute to developing ethical and policy frameworks for safe AI development and deployment.
  • Eleos AI is a nonprofit launched in October 2024 that describes itself as “dedicated to understanding and addressing the potential wellbeing and moral patienthood of AI systems.” It was founded by Robert Long, a researcher in this area who appeared on The 80,000 Hours Podcast.

We list many relevant places you might work in our AI governance and AI technical safety career reviews.

Support this field in other ways

You could also consider earning to give to support this field. If you’re a good fit for high-earning paths, it may be the best way for you to contribute.

This is because as a new field, and one that’s in part about nonhuman interests, there are few (if any) major funders supporting it and not much commercial or political interest. This can make it difficult to start new organisations and commit to research programmes that might not be able to rely on a steady source of funding. Filling this gap can make a huge difference in whether a thriving research field gets off the ground at all.

We expect there will be a range of organisations and different kinds of groups people will set up to address work on better understanding the moral status of digital minds. In addition to funding, you might join or help start these organisations. This is a particularly promising choice if you have a strong aptitude for organisation-building or founding a high-impact organisation.

Important considerations if you work on this problem

1. Field-building focus: Given the early stage of this field, much of the work involves building credibility and establishing the topic as a legitimate area of inquiry.
2. Interdisciplinary approach: Recognise that understanding digital minds requires insights from multiple disciplines, so cultivate a broad knowledge base. Do not dismiss fields — like philosophy, ML engineering, or cognitive science — as irrelevant just because they’re not your expertise.
3. Ethical vigilance: Approach the topic with careful consideration of the ethical implications of your work and its potential impact on both biological and potential digital entities.
4. Cooperation and humility: Be cooperative in your work and acknowledge your own and others’ epistemic limitations, and the need to find our way through uncertainty.
5. Patience and long-term thinking: Recognise that progress in this field may be slow and difficult.

Learn more

Podcasts

• Jeff Sebo on digital minds, and how to avoid sleepwalking into a major moral catastrophe
• Robert Long on why large language models like GPT (probably) aren’t conscious
• Jonathan Birch on the edge cases of sentience and why they matter
• Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared
• Anil Seth on the predictive brain and how to study consciousness
• Peter Godfrey-Smith on interfering with wild nature, accepting death, and the origin of complex civilisation
• Carl Shulman on the economy and national security after AGI
• Eric Schwitzgebel on the weirdness of the world on Hear This Idea

Research and reports

• Taking AI Welfare Seriously by Robert Long, Jeff Sebo, et al.
• The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI by Jonathan Birch (free PDF from Oxford University Press)
• Consciousness in artificial intelligence: Insights from the science of consciousness by Patrick Butlin, Robert Long, et al.
• Moral consideration for AI systems by 2030 by Jeff Sebo and Robert Long
• Propositions concerning digital minds and society and Sharing the world with digital minds by Nick Bostrom and Carl Shulman
• Digital minds: Importance and key research Questions by Andreas Mogensen, Bradford Saad, and Patrick Butlin
• Digital people would be an even bigger deal and Digital people FAQ by Holden Karnofsky
• Towards evaluating AI systems for moral status using self-reports by Robert Long and Ethan Perez
• Nervous systems, functionalism, and artificial minds by Peter Godfrey-Smith
• Improving the Welfare of AIs: A Nearcasted Proposal by Ryan Greenblatt
• The 2020 PhilPapers survey, edited by David Bourget and David Chalmers
• Reality+: Virtual Worlds and the Problems of Philosophy by David Chalmers
• Could a large language model be conscious? by David Chalmers
• Digital suffering: why it’s a problem and how to prevent it by Bradford Saad and Adam Bradley
• Amanda Askell on consciousness in current AI systems
• To understand AI sentience, first understand it in animals by Kristin Andrews and Jonathan Birch
• A defense of the rights of artificial intelligences and Designing AI with rights, consciousness, self-Respect, and freedom by Eric Schwitzgebel and Mara Garza
• Patiency is not a virtue: the design of intelligent systems and systems of ethics by Joanna J. Bryson
• Folk psychological attributions of consciousness to large language models by Clara Colombatto and Stephen Fleming

Read next:  Explore other pressing world problems

Want to learn more about global issues we think are especially pressing? See our list of issues that are large in scale, solvable, and neglected, according to our research.

Continue →

The post Understanding the moral status of digital minds appeared first on 80,000 Hours.

There’s still a lot of uncertainty about which AI governance strategies would be best. Many have proposed policies and strategies aimed at reducing the largest risks, which we discuss below.

But there’s no roadmap here. There’s plenty of room for debate about what’s needed, and we may not have found the best ideas yet in this space. In any case, there’s still a lot of work to figure out how promising policies and strategies would work in practice. We hope to see more people enter this field to develop expertise and skills that will contribute to risk-reducing AI governance and policy.

“What you’re doing has enormous potential and enormous danger.” — US President Joe Biden, to the leaders of the top AI companies

Why this could be a high-impact career path

Artificial intelligence has advanced rapidly. In 2022 and 2023, new language and image generation models gained widespread attention for their abilities, blowing past previous benchmarks.

And the applications of these models are still new; with more tweaking and integration into society, the existing AI systems may become easier to use and more ubiquitous.

We don’t know where all these developments will lead us. There’s reason to be optimistic that AI will eventually help us solve many of the world’s problems, raising living standards and helping us build a more flourishing society.

But there are also substantial risks. Advanced AI could be used to do a lot of harm. And we worry it could accidentally lead to a major catastrophe — and perhaps even cause human disempowerment or extinction. We discuss the arguments that these risks exist in our in-depth problem profile.

Because of these risks, we encourage people to work on finding ways to reduce the danger through technical research and engineering.

But we need a range of strategies for risk reduction. Public policy and corporate governance in particular may be necessary to ensure that advanced AI is broadly beneficial and low risk.

Governance generally refers to the processes, structures, and systems that carry out decision making for organisations and societies at a high level. In the case of AI, we expect the governance structures that matter most to be national governments and organisations developing AI — as well as some international organisations and perhaps subnational governments.

Some aims of AI governance work could include:

• Preventing the deployment of any AI systems that pose a significant and direct threat of catastrophe
• Mitigating the negative impact of AI technology on other catastrophic risks, such as nuclear weapons and biotechnology
• Guiding the integration of AI technology into our society and economy with limited harms and to the advantage of all
• Reducing the risk of an “AI arms race” between nations and between companies
• Ensuring that advanced AI developers are incentivised to be cooperative and concerned about safety
• Slowing down the development and deployment of new systems if the advancements are likely to outpace our ability to keep them safe and under control

We need a community of experts who understand modern AI systems and policy, as well as the severe threats and potential solutions. This field is still young, and many of the paths within it aren’t clear and are not sure to pan out. But there are relevant professional paths that will provide you valuable career capital for a variety of positions and types of roles.

The rest of this article explains what work in this area might involve, how you can develop career capital and test your fit, and some promising places to work.

What kinds of work might contribute to AI governance?

There are a variety of ways to pursue AI governance strategies, and as the field becomes more mature, the paths are likely to become clearer and more established.

We generally don’t think people early in their careers should aim for a specific high-impact job. They should instead aim to develop skills, experience, knowledge, judgement, networks, and credentials — what we call career capital — that they can use later to have an impact.

This may involve following a standard career trajectory or moving around in different kinds of roles. Sometimes, you just have to apply to many different roles and test your fit for various types of work before you know what you’ll be good at. Most importantly, you should try to get excellent at something for which you have strong personal fit and that will let you contribute to solving pressing problems.

In the AI governance space, we see at least six broad categories of work that we think are important:

• Government work
• Research on AI policy and strategy
• Industry work
• Advocacy and lobbying
• Third-party auditing and evaluation
• International work and coordination

Thinking about the different kinds of career capital that are useful for the categories of work that appeal to you may suggest some next steps in your path. (We discuss how to assess your fit and enter this field below.)

You may want to move between these different categories of work at different points in your career. You can also test out your fit for various roles by taking internships, fellowships, entry-level jobs, temporary placements, or even doing independent research, all of which can serve as career capital for a range of paths.

We have also reviewed career paths in AI technical safety research and engineering, information security, and AI hardware expertise, which may be crucial to reducing risks from AI. These fields may also play a significant role in an effective governance agenda. People serious about pursuing a career in AI governance should familiarise themselves with these subjects as well.

Government work

Taking a role within an influential government could help you play an important role in the development, enactment, and enforcement of AI policy.

We generally expect that the US federal government will be the most significant player in AI governance for the foreseeable future. This is because of its global influence and its jurisdiction over much of the AI industry, including the most prominent AI companies such as Anthropic, OpenAI, and Google DeepMind. It also has jurisdiction over key parts of the AI chip supply chain. Much of this article focuses on US policy and government.¹

But other governments and international institutions matter too. For example, the UK government, the European Union, China, and others may present opportunities for impactful AI governance work. Some US state-level governments, such as California, may have opportunities for impact and gaining career capital.

What would this work involve? Sections below discuss how to enter US policy work and which areas of the government that you might aim for.

In 2023, the US and UK governments both announced new institutes for AI safety — both of which should provide valuable opportunities for career capital and potential impact.

But at the broadest level, people interested in positively shaping AI policy should gain skills and experience to work in areas of government with some connection to AI or emerging technology policy.

This can include roles in: legislative branches, domestic regulation, national security, diplomacy, appropriations and budgeting, and other policy areas.

If you can get a role already working directly on this issue, such as in one of the AI safety institutes or working for a lawmaker focused on AI, that could be a great opportunity.

Otherwise, you should seek to learn as much as you can about how policy works and which government roles might allow you to have the most impact. Try to establish yourself as someone who’s knowledgeable about the AI policy landscape. Having almost any significant government role that touches on some aspect of AI, or having some impressive AI-related credential, may be enough to go quite far.

One way to advance your career in government on a specific topic is what some call “getting visibility.” This involves using your position to learn about the landscape and connect with the actors and institutions in the policy area. You’ll want to engage socially with others in the policy field, get invited to meetings with other officials and agencies, and be asked for input on decisions. If you can establish yourself as a well-regarded expert on an important but neglected aspect of the issue, you’ll have a better shot at being included in key discussions and events.

Career trajectories within government can be broken down roughly as follows:

• Standard government track: This involves entering government at a relatively low level and climbing the seniority ladder. For the highest impact, you’d ideally reach senior levels by sticking around, forming relationships, gaining skills and experience, and getting promoted. You may move between agencies, departments, or branches.
• Specialisation career capital: You can also move in and out of government throughout your career. People on this trajectory also work at nonprofits, think tanks, the private sector, government contractors, political parties, academia, and other organisations. But they will primarily focus on becoming an expert in a topic — such as AI. It can be harder to get seniority this way, but the value of expertise can sometimes be greater than the value of seniority.
• Direct-impact work: Some people move into government jobs without a longer plan to build career capital because they see an opportunity for direct, immediate impact. This might involve getting tapped to lead an important commission or providing valuable input on an urgent project. This isn’t necessarily a strategy you can plan a career around, but it’s good to be aware of it as an option that might be worth taking at some point.

Read more about how to evaluate your fit and get started building relevant career capital in our article on policy and political skills.

Research on AI policy and strategy

There’s still a lot of research to be done on AI governance strategy and implementation. The world needs more concrete policies that would really start to tackle the biggest threats; developing such policies and deepening our understanding of the strategic needs of the AI governance space are high priorities.

Other relevant research could involve surveys of public and expert opinion, legal research about the feasibility of proposed policies, technical research on issues like compute governance, and even higher-level theoretical research into questions about the societal implications of advanced AI.

Some research, such as that done by Epoch AI, focuses on forecasting the future course of AI developments, which can influence AI governance decisions.

However, several experts we’ve talked to warn that a lot of research on AI governance may prove to be useless. So it’s important to be reflective and seek input from others in the field about what kind of contribution you can make. We list several research organisations below that we think pursue promising research on this topic and could provide useful mentorship.

One approach for testing your fit for this work — especially when starting out — is to write up analyses and responses to existing work on AI policy or investigate some questions in this area that haven’t received much attention. You can then share your work widely, send it out for feedback from people in the field, and evaluate how you enjoy the work and how you might contribute to this field.

But don’t spend too long testing your fit without making much progress, and note that some are best able to contribute when they’re working on a team. So don’t over-invest in independent work, especially if there are few signs it’s working out especially well for you. This kind of project can make sense for maybe a month or a bit longer — but it’s unlikely to be a good idea to spend much more than that without funding or some really encouraging feedback from people working in the field.

If you have the experience to be hired as a researcher, work on AI governance can be done in academia, nonprofit organisations, and think tanks. Some government agencies and committees, too, perform valuable research.

Note that universities and academia have their own priorities and incentives that often aren’t aligned with producing the most impactful work. If you’re already an established researcher with tenure, it may be highly valuable to pivot into work on AI governance — your position may even give you a credible platform from which to advocate for important ideas.

But if you’re just starting out a research career and want to focus on this issue, you should carefully consider whether your work will be best supported inside academia. For example, if you know of a specific programme with particular mentors who will help you pursue answers to critical questions in this field, it might be worth doing. We’re less inclined to encourage people on this path to pursue generic academic-track roles without a clear idea of how they can do important research on AI governance.

Advanced degrees in policy or relevant technical fields may well be valuable, though — see more discussion of this in the section on how to assess your fit and get started.

You can also learn more in our article about how to become a researcher.

Industry work

Internal policy and corporate governance at the largest AI companies themselves is also important for reducing risks from AI.

At the highest level, deciding who sits on corporate boards, what kind of influence those boards have, and the incentives the organisation faces can have a major impact on a company’s choices. Many of these roles are filled by people with extensive management and organisational leadership experience, such as founding and running companies.

If you’re able to join a policy team at a major company, you can model threats and help develop, implement, and evaluate proposals to reduce risks. And you can build consensus around best practices, such as strong information security, using outside evaluators to find vulnerabilities and dangerous behaviours in AI systems (red teaming), and testing out the latest techniques from the field of AI safety.

And if, as we expect, AI companies face increasing government oversight, ensuring compliance with relevant laws and regulations will be a high priority. Communicating with government actors and facilitating coordination from inside the companies could be impactful work.

In general, it seems better for AI companies to be highly cooperative with each other² and with outside groups seeking to minimise risks. And this doesn’t seem to be an outlandish hope — many industry leaders have expressed concern about catastrophic risks and have even called for regulation of the frontier technology they’re creating.

That said, cooperation will likely take a lot of effort. Companies creating powerful AI systems may resist some risk-reducing policies, because they’ll have strong incentives to commercialise their products. So getting buy-in from the key players, increasing trust and information-sharing, and building a consensus around high-level safety strategies will be valuable.

Advocacy and lobbying

People outside of government or AI companies can influence the shape of public policy and corporate governance with advocacy and lobbying.

Advocacy is the general term for efforts to promote certain ideas and shape the public discourse, often around policy-related topics. Lobbying is a more targeted effort aimed at influencing legislation and policy, often by engaging with lawmakers and other officials.

If you believe AI companies may be disposed to advocate for generally beneficial regulation, you might work with them to push the government to adopt specific policies. It’s plausible that AI companies have the best understanding of the technology, as well as the risks, failure modes, and safest paths — and so are best positioned to inform policymakers.

On the other hand, AI companies might have too much of a vested interest in the shape of regulations to reliably advocate for broadly beneficial policies. If that’s right, it may be better to join or create advocacy organisations unconnected from the industry — perhaps supported by donations — that can take stances opposed to commercial interests.

For example, some believe it might be best to deliberately slow down or halt the development of increasingly powerful AI models. Advocates could make this demand of the companies themselves or of the government. But pushing for this step may be difficult for those involved with the companies creating advanced AI systems.

It’s also possible that the best outcomes will result from a balance of perspectives from inside and outside industry.

Advocacy can also:

• Highlight neglected but promising approaches to governance that have been uncovered in research
• Facilitate the work of policymakers by showcasing the public’s support for governance measures
• Build bridges between researchers, policymakers, the media, and the public by communicating complicated ideas in an accessible way
• Pressure companies to proceed more cautiously
• Change public sentiment around AI and discourage irresponsible behaviour by individual actors

However, note that advocacy can sometimes backfire because predicting how information will be received isn’t straightforward. Be aware that:

• Drawing attention to a cause area can sometimes trigger a backlash
• Certain styles of rhetoric can alienate people or polarise public opinion
• Spreading mistaken messages can discredit yourself and others

It’s important to keep these risks in mind and consult with others (particularly those who you respect but might disagree with tactically). And you should educate yourself deeply about the topic before explaining it to the public.

You can read more in the section about doing harm below. We also recommend reading our article on ways people trying to do good accidentally make things worse and how to avoid them. And you may find it useful to read our article on the skills needed for communicating important ideas.

Third-party auditing and evaluation

If regulatory measures are put in place to reduce the risks of advanced AI, some agencies and outside organisations will need to audit companies and systems to make sure that regulations are being followed.

Governments often rely on third-party auditors when regulating because the government lacks much of the expertise that the private sector has. There aren’t many such opportunities available in this type of role for AI-related auditing that we know of, but such roles play a critical part of an effective AI governance framework.

AI companies and the AI systems they create may be subject to audits and evaluations out of safety concerns.

One nonprofit, Model Evaluation and Threat Research (METR, formally known as ARC Evals), has been at the forefront of work to evaluate the capabilities of advanced AI models.³ In early 2023, the organisation partnered with two leading AI companies, OpenAI and Anthropic, to evaluate the capabilities of the latest versions of their chatbot models prior to their release. They sought to determine if the models had any potentially dangerous capabilities in a controlled environment.

The companies voluntarily cooperated with METR for this project, but at some point in the future, these evaluations may be legally required.

Other types of auditing and evaluation may be required as well. METR has said it intends to develop methods to determine which models are appropriately aligned — that is, that they will behave as their users intend them to behave — prior to release.

Governments may also want to employ auditors to evaluate the amount of compute that AI developers have access to, their information security practices, the uses of models, the data used to train models, and more.

Acquiring the technical skills and knowledge to perform these types of evaluations, and joining organisations that will be tasked to perform them, could be the foundation of a highly impactful career. This kind of work will also likely have to be facilitated by people who can manage complex relationships across industry and government. Someone with experience in both sectors could have a lot to contribute.

Some of these types of roles may have some overlap with work in AI technical safety research.

International work and coordination

US-China

For someone with the right fit, working to improve coordination with China on the safe development of AI could be a particularly impactful career path.

The Chinese government has been a major funder in the field of AI, and the country has giant tech companies that could potentially drive forward advances.

Given tensions between the US and China, and the risks posed by advanced AI, there’s a lot to be gained from increasing trust, understanding, and coordination between the two countries. The world will likely be much better off if we can avoid a major conflict between great powers and if the most significant players in emerging technology can avoid exacerbating any global risks.

We have a separate career review that goes into more depth on China-related AI safety and governance paths.

Other governments and international organisations

As we’ve said, we focus most on US policy and government roles. This is largely because we anticipate that the US is now and will likely continue to be the most pivotal actor when it comes to regulating AI, with a major caveat being China, as discussed in the previous section.

But many people interested in working on this issue can’t or don’t want to work in US policy — perhaps because they live in another country and don’t intend on moving.

Much of the advice above still applies to these people, because roles in AI governance research and advocacy can be done outside of the US.⁴ And while we don’t think it’s generally as impactful in expectation as US government work, opportunities in other governments and international organisations can be complementary to the work to be done in the US.

The United Kingdom, for instance, may present another strong opportunity for AI policy work that would complement US work. Top UK officials have expressed interest in developing policy around AI, a new international agency, and reducing extreme risks. And the UK government announced the creation of its own AI Safety Institute in 2024 to develop evaluations for AI systems and coordinate globally on AI policy.

The European Union has shown that its data protection standards — the General Data Protection Regulation (GDPR) — affect corporate behaviour well beyond its geographical boundaries. EU officials have also pushed forward on regulating AI, and some research has explored the hypothesis that the impact of the EU’s AI regulations will extend far beyond the continent — the so-called “Brussels effect.”

And any relatively wealthy country could fund some AI safety research, though much of it requires access to top talent and state of the art tech. Any significant advances in AI safety research could inform researchers working on the most powerful models.

Other countries might also develop liability standards for the creators of AI systems that could incentivise corporations to proceed cautiously before releasing models.

And at some point, there may be AI treaties and international regulations, just as the international community has created the International Atomic Energy Agency, the Biological Weapons Convention, and Intergovernmental Panel on Climate Change to coordinate around and mitigate other global catastrophic threats.

Efforts to coordinate governments around the world to understand and share information about threats posed by AI may end up being extremely important in some future scenarios. The Organisation for Economic Cooperation and Development, for instance, has already created the AI Policy Observatory.

Third-party countries may also be able to facilitate cooperation and reduce tensions betweens the United States and China, whether around AI or other potential flashpoints.

What policies and practices would reduce the largest risks?

People working in AI policy have proposed a range of approaches to reducing risk as AI systems get more powerful.

We don’t necessarily endorse all the ideas below, but what follows is a list of some prominent policy approaches that could be aimed at reducing the largest dangers from AI:⁵

• Responsible scaling policies: some major AI companies have already begun developing internal frameworks for assessing safety as they scale up the size and capabilities of their systems. These frameworks introduce safeguards that are intended to become increasingly stringent as AI systems become potentially more dangerous, and they ensure that AI systems’ capabilities don’t outpace companies’ abilities to keep systems safe. Many argue that these internal policies are not sufficient for safety, but they may represent a promising step for reducing risk. You can see versions of such policies from Anthropic, Google DeepMind, and OpenAI.
• Standards and evaluation: governments might develop industry-wide benchmarks and testing protocols to assess whether AI systems pose major risks. The non-profit METR and the UK AI Safety Institute are among the organisations currently developing these evaluations to test AI models before and after they are released. This can include creating standardised metrics for an AI system’s capabilities and potential to cause harm, as well as its propensity for power-seeking or misalignment.
• Safety cases: this practice involves requiring AI developers to provide comprehensive documentation demonstrating the safety and reliability of their systems before deployment. This approach is similar to safety cases used in other high-risk industries like aviation or nuclear power.⁶ You can see discussion of this idea in a paper from Clymer et al and in a post from Geoffrey Irving at the UK AI Safety Institute.
• Information security standards: we can establish robust rules for protecting AI-related data, algorithms, and infrastructure from unauthorised access or manipulation — particularly the AI model weights. Rand released a detailed report analysing the security risks to major AI companies, particularly from state actors.
• Liability law: existing law already imposes some liability on companies that create dangerous products or cause significant harm to the public, but its application to AI models and their risks in particular is unclear. Clarifying how liability applies to companies that create dangerous AI models could incentivise them to take additional steps to reduce risk. Law professor Gabriel Weil has written about this idea.
• Compute governance: governments may regulate access to and use of high-performance computing resources necessary for training large AI models. The US restrictions on exporting state-of-the-art chips to China is one example of this kind of policy, and others are possible. Companies could also be required to install hardware-level safety features directly into AI chips or processors. These could be used to track chips and verify they’re not in the possession of anyone who shouldn’t have them or for other purposes. You can learn more about this topic in our interview with Lennart Heim and in this report from the Center for a New American Security.
• International coordination: Fostering global cooperation on AI governance to ensure consistent standards may be crucial. This could involve treaties, international organisations, or multilateral agreements on AI development and deployment. We discuss some related considerations in our article on China-related AI safety and governance paths.
• Societal adaptation: it may be critically important to prepare society for the widespread integration of AI and the potential risks it poses. For example, we might need to develop new information security measures to protect crucial data in a world with AI-enabled hacking. Or we may want to implement strong controls to prevent handing over key societal decisions to AI systems.⁷
• Pausing scaling if appropriate: some argue that we should currently pause all scaling of larger AI models because of the dangers the technology poses. We have featured some discussion of this idea on our podcast. It seems hard to know if or when this would be a good idea. If carried out, it could involve industry-wide agreements or regulatory mandates to pause scaling efforts.

The details, benefits, and downsides of many of these ideas have yet to be fully worked out, so it’s crucial that we do more research and get more input from informed stakeholders. And this list isn’t comprehensive — there are likely other important policy interventions and governance strategies worth pursuing.

You can also check out a list of potential policy ideas from Luke Muehlhauser of Open Philanthropy,⁸ an article about AI policy proposals from Vox‘s Dylan Matthews, and a survey of expert opinion on best practices in AI safety and governance.

Examples of people pursuing this path

How to assess your fit and get started

If you’re early on in your career, you should focus first on getting skills and other career capital to successfully contribute to the beneficial governance and regulation of AI.

You can gain career capital for roles in many ways. Broadly speaking, working in or studying fields such as politics, law, international relations, communications, and economics can all be beneficial for going into policy work.

And expertise in AI itself, gained by studying and working in machine learning and technical AI safety, or potentially related fields such as computer hardware and information security, should also give you a big advantage.

Testing your fit

Try to find relatively “cheap” tests to assess your fit for different paths. This could mean, for example, taking a policy internship, applying for a fellowship, doing a short bout of independent research, or taking classes or courses on technical machine learning or computer engineering.

It can also involve talking to people doing a job and finding out what the day-to-day experience of the work is and what skills are needed.

All of these factors can be difficult to predict in advance. While we grouped “government work” into a single category above, that label covers a wide range of roles. Finding the right fit can take years, and it can depend on factors out of your control, such as the colleagues you work closely with. That’s one reason it’s useful to build broadly valuable career capital that gives you more options.

Don’t underestimate the value of applying to many relevant openings in the field and sector you’re aiming for to see what happens. You’ll likely face a lot of rejection with this strategy, but you’ll be able to better assess your fit for roles after you see how far you get in the process. This can give you more information than guessing about whether you have the right experience.

Try to rule out certain types of work if you gather evidence that you’re not a strong fit. For example, if you invest a lot of effort trying to get into reputable universities or nonprofit institutions to do AI governance research, but you get no promising offers and receive little encouragement, this might be a significant signal that you’re unlikely to thrive in that path.

That wouldn’t mean you have nothing to contribute, but your comparative advantage may lie elsewhere.

Read the section of our career guide on finding a job that fits you.

Types of career capital

A mix of people with technical and policy expertise — and some people with both — is needed in AI governance.

While anyone involved in this field should work to maintain an understanding of both the technical and policy details, you’ll probably start out focusing on either policy or technical skills to gain career capital.

This section covers:

• Generally useful career capital
• Policy-related career capital
• Technical career capital
• Other specific forms of career capital

Much of this advice is geared toward roles in the US, though it may be relevant in other contexts.

Generally useful career capital

The chapter of the 80,000 Hours career guide on career capital lists five key components that will be useful in any path: skills and knowledge, connections, credentials, character, and runway.

For most jobs touching on policy, social skills, networking, and — for lack of a better word — political skills will be a huge asset. This can probably be learned to some extent, but some people may find they don’t have these kinds of skills and can’t or don’t want to acquire them.

That’s OK — there are many other routes to having a fulfilling and impactful career, and there may be some roles within this path that demand these skills to a much lesser extent. That’s why testing your fit is important.

Read the full section of the career guide on career capital.

Policy-related career capital

To gain skills in policy, you can pursue education in many relevant fields, such as political science, economics, and law.

Many master’s programmes offer specific coursework on public policy, science and society, security studies, international relations, and other topics; having a graduate degree or law degree will give you a leg up for many positions.

In the US, a master’s, a law degree, or a PhD is particularly useful if you want to climb the federal bureaucracy. Our article on US policy master’s degrees provides detailed information about how to assess the many options.

Internships in DC are a promising route to test your fit for policy and get career capital. Many academic institutions now offer a strategic “Semester in DC” programme, which can let you explore placements in Congress, federal agencies, or think tanks.

The Virtual Student Federal Service (VSFS) also offers part-time, remote government internships. Students in this program work alongside their studies.

Once you have a suitable background, you can take entry-level positions within parts of the government and build a professional network while developing key skills. In the US, you can become a congressional staffer, or take a position at a relevant federal department, such as the Department of Commerce, Department of Energy, or the Department of State. Alternatively, you can gain experience in think tanks (a particularly promising option if you have an aptitude for research). Some government contractors can also be a strong option.

Many people say Washington, D.C. has a unique culture, particularly for those working in and around the federal government. There’s a big focus on networking, bureaucratic politics, status-seeking, and influence-peddling. We’ve also been told that while merit matters to a degree in US government work, it is not the primary determinant of who is most successful. People who think they wouldn’t feel able or comfortable to be in this kind of environment for the long term should consider whether other paths would be best.

If you find you can enjoy government and political work, impress your colleagues, and advance in your career, though, you may be a good fit. Just being able to thrive in government work can be a valuable comparative advantage.

Technical career capital

Technical experience in machine learning, AI hardware, and related fields can be a valuable asset for an AI governance career. So it will be very helpful if you’ve studied a relevant subject area for an undergraduate or graduate degree, or did a particularly productive course of independent study.

We have a guide to technical AI safety careers, which explains how to learn the basics of machine learning.

Working at an AI company or lab in technical roles, or other companies that use advanced AI systems and hardware, may also provide significant career capital in AI policy paths. Read our career review discussing the pros and cons of working at a top AI company.

We also have a separate career review on how becoming an expert in AI hardware could be very valuable in governance work.

Many politicians and policymakers are generalists, as their roles require them to work in many different subject areas and on different types of problems. This means they’ll need to rely on expert knowledge when crafting and implementing policy on AI technology that they don’t fully understand. So if you can provide them this information, especially if you’re skilled at communicating it clearly, you can potentially fill influential roles.

Some people who may have initially been interested in pursuing a technical AI safety career, but who have found that they either are no longer interested in that path or find more promising policy opportunities, might also decide that they can effectively pivot into a policy-oriented career.

It is common for people with STEM backgrounds to enter and succeed in US policy careers. People with technical credentials that they may regard as fairly modest — such as a computer science bachelor’s degree or a master’s in machine learning — often find their knowledge is highly valued in Washington, DC.

Most DC jobs don’t have specific degree requirements, so you don’t need to have a policy degree to work in DC. Roles specifically addressing science and technology policy are particularly well-suited for people with technical backgrounds, and people hiring for these roles will value higher credentials like a master’s or, better even, a terminal degree like a PhD or MD.

There are many fellowship programmes specifically aiming to support people with STEM backgrounds to enter policy careers; some are listed below.

Policy work won’t be right for everybody — many technical experts may not have the right disposition or skills. People in policy paths often benefit from strong writing and social skills, as well as being comfortable navigating bureaucracies and working with people holding very different motivations and worldviews.

Other specific forms of career capital

There are other ways to gain useful career capital that could be applied in this career path.

• If you have or gain great communication skills as, say, a journalist or an activist, these skills could be very useful in advocacy and lobbying around AI governance.
  • Especially since advocacy around AI issues is still in its early stages, it will likely need people with experience advocating in other important cause areas to share their knowledge and skills.
• Academics with relevant skill sets are sometimes brought into government for limited stints to serve as advisors in agencies such as the US Office of Science and Technology. This may or may not be the foundation of a longer career in government, but it should give an academic deeper insight into policy and politics.
• You can work at an AI company or lab in non-technical roles, gaining a deeper familiarity with the technology, the business, and the culture.
• You could work on political campaigns and get involved in party politics. This is one way to get involved in legislation, learn about policy, and help impactful lawmakers, and you can also potentially help shape the discourse around AI governance. Note, though, there are downsides of potentially polarising public opinion around AI policy (discussed more below); and entering party politics may limit your potential for impact whenever the party you’ve joined doesn’t hold power.
• You could even try to become an elected official yourself, though it’s competitive. If you take this route, make sure you find trustworthy and informed advisors to build expertise in AI since politicians have many other responsibilities and can’t focus as much on any particular issue.
• You can focus on developing specific skill sets that might be valuable in AI governance, such as information security, intelligence work, diplomacy with China, etc.
  • Other skills: Organisational, entrepreneurial, management, diplomatic, and bureaucratic skills will also likely prove highly valuable in this career path. There may be new auditing agencies to set up or policy regimes to implement. Someone who has worked at high levels in other high-stakes industries, started an influential company, or coordinated complicated negotiations between various groups, would bring important skills to the table.

Where can this kind of work be done?

Since successful AI governance will require work from governments, industry, and other parties, there will be many potential jobs and places to work for people in this path. The landscape will likely shift over time, so if you’re just starting out on this path, the places that seem most important might be different by the time you’re pivoting to using your career capital to make progress on the issue.

Within the US government, for instance, it’s not clear which bodies will be most impactful when it comes to AI policy in five years. It will likely depend on choices that are made in the meantime.

That said, it seems useful to give our understanding of which parts of the government are generally influential in technology governance and most involved right now to help you orient. Gaining AI-related experience in government right now should still serve you well if you end up wanting to move into a more impactful AI-related role down the line when the highest-impact areas to work in are clearer.

We’ll also give our current sense of important actors outside government where you might be able to build career capital and potentially have a big impact.

Note that this list has by far the most detail about places to work within the US government. We would like to expand it to include more options over time. (Note: the fact that an option isn’t on this list shouldn’t be taken to mean we recommend against it or even that it would necessarily be less impactful than the places listed.)

We have more detail on other options in separate (and older) career reviews, including the following:

• China-related AI safety and governance paths
• Party politics (UK-focused)
• Policy-oriented government jobs (UK-focused)

Here are some of the places where someone could do promising work or gain valuable career capital:

Our job board features opportunities in AI safety and policy:

View all opportunities

How this career path can go wrong

Doing harm

As we discuss in an article on accidental harm, there are many ways to set back a new field that you’re working in when you’re trying to do good, and this could mean your impact is negative rather than positive. (You may also want to read our article on harmful careers.)

There’s a lot of potential to inadvertently cause harm in the emerging field of AI governance. We discussed some possibilities in the section on advocacy and lobbying. Some other possibilities include:

• Pushing for a given policy to the detriment of a superior policy
• Communicating about the risks of AI in a way that ratchets up geopolitical tensions
• Enacting a policy that has the opposite impact of its intended effect
• Setting policy precedents that could be exploited by dangerous actors down the line
• Funding projects in AI that turn out to be dangerous
• Sending the message, implicitly or explicitly, that the risks are being managed when they aren’t or that they’re lower than they in fact are
• Suppressing technology that would actually be extremely beneficial for society

We have to act with incomplete information, so it may never be very clear when or if people in AI governance are falling into these traps. Being aware that they are potential ways of causing harm will help you keep alert for these possibilities, though, and you should remain open to changing course if you find evidence that your actions may be damaging.

And we recommend keeping in mind the following pieces of general guidance from our article on accidental harm:

1. Ideally, eliminate courses of action that might have a big negative impact.
2. Don’t be a naive optimizer.
3. Have a degree of humility.
4. Develop expertise, get trained, build a network, and benefit from your field’s accumulated wisdom.
5. Follow cooperative norms.
6. Match your capabilities to your project and influence.
7. Avoid hard-to-reverse actions.

Burning out

We think this work is exceptionally pressing and valuable, so we encourage our readers who might be interested to test their fit for governance work. But going into government, in particular, can be difficult. Some people we’ve advised have gone into policy roles with the hope of having an impact, only to burn out and move on.

At the same time, many policy practitioners find their work very meaningful, interesting, and varied.

Some roles in government may be especially challenging for the following reasons:

• The work can be very fast-paced, involving relatively high stress and long hours. This is particularly true in Congress and senior executive branch positions and much less so in think tanks or junior agency roles.
• It can take a long time to get into positions with much autonomy or decision-making authority.
• Progress on the issues you care about can be slow, and you often have to work on other priorities. Congressional staffers in particular typically have very broad policy portfolios.
• Work within bureaucracies faces many limitations, which can be frustrating.
• It can be demotivating to work with people who don’t share your values. Though note that policy can select for altruistic people — even if they have different beliefs about how to do good.
• The work isn’t typically well-paid relative to comparable positions outside of government.

So we recommend speaking to people in the kinds of positions you might aim to have in order to get a sense of whether the career path would be right for you. And if you do choose to pursue it, look out for signs that the work may be having a negative effect on you and seek support from people who understand what you care about.

If you end up wanting or needing to leave and transition into a new path, that’s not necessarily a loss or a reason for regret. You will likely make important connections and learn a lot of useful information and skills. This career capital can be useful as you transition into another role, perhaps pursuing a complementary approach to AI governance.

What the increased attention on AI means

We’ve been concerned about risks posed by AI for years. Based on the arguments that this technology could potentially cause a global catastrophe, and otherwise have a dramatic impact on future generations, we’ve advised many people to work to mitigate the risks.

The arguments for the risk aren’t completely conclusive, in our view. But the arguments are worth taking seriously, and given the fact that few others in the world seemed to be devoting much time to even figuring out how big the threat was or how to mitigate it (while at the same time progress in making AI systems more powerful was accelerating) we concluded it was worth ranking among our top priorities.

Now that there’s increased attention on AI, some might conclude that it’s less neglected and thus less pressing to work on. However, the increased attention on AI also makes many interventions potentially more tractable than they had been previously, as policymakers and others are more open to the idea of crafting AI regulations.

And while more attention is now being paid to AI, it’s not clear it will be focused on the most important risks. So there’s likely still a lot of room for important and pressing work positively shaping the development of AI policy.

Questions or feedback about this article? Email us

Read next

If you’re interested in this career path, we recommend checking out some of the following articles next.

• 

  US policy master's degrees

  These degrees are highly valuable for those hoping to take on important roles in the US federal government.

  Read more
• 

  Working in US AI policy

  The US government is likely to be a key actor in how advanced AI is developed and used in society, whether directly or indirectly.

  Read more
• 

  Should you work at a frontier AI company?

  Working at a leading AI company is an important career option to consider, but the impact of any given role is complex to assess.

  Read more
• 

  Preventing an AI-related catastrophe

  AI might bring huge benefits — if we avoid the risks.

  Read more

Learn more

Top recommendations

• AI Governance Course – AGI Safety Fundamentals from BlueDot Impact
• Podcast: Tantum Collins on what he’s learned as an AI policy insider
• US AI policy resources, think tanks, fellowships, and more

Further recommendations

Resources from 80,000 Hours

• Article: Working in US AI policy
• Podcast: Tom Kalil on how to have a big impact in government & huge organisations, based on 16 years’ experience in the White House
• Podcast: Holden Karnofsky on how AIs might take over even if they’re no smarter than humans, and his four-part playbook for AI risk
• Podcast: Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared
• Podcast: Allan Dafoe on why technology is unstoppable & how to shape AI development anyway
• Podcast: Tom Davidson on how AI-enabled coups could allow a tiny group to seize power
• Podcast: Nick Joseph on whether Anthropic’s AI safety policy is up to the task
• Podcast: Nathan Calvin on California’s AI bill SB 1047 and its potential to shape US AI policy
• Podcast: Rose Chan Loui on OpenAI’s gambit to ditch its nonprofit
• Podcast: Carl Shulman on the economy and national security and government and society after AGI
• Podcast: Zvi Mowshowitz on sleeping on sleeper agents, and the biggest AI updates since ChatGPT
• Podcast: Lennart Heim on the compute governance era and what has to come after
• Podcast: Sella Nevo on who’s trying to steal frontier AI models, and what could they do with them
• Podcast: Nathan Labenz on the final push for AGI, understanding OpenAI’s leadership drama, and red-teaming frontier models
• Podcast: Vitalik Buterin on defensive acceleration and how to regulate AI when you fear government
• Career review: China-related AI safety and governance paths
• Podcast collection: The 80,000 Hours Podcast on Artificial Intelligence

Resources from others

• Emerging Tech Policy Careers
• Jobs that can help with the most important century by Holden Karnofsky
• 12 tentative ideas for US AI policy by Luke Muehlhauser of Open Philanthropy
• Why and how governments should monitor AI development by Jess Whittlestone and Jack Clark
• AGI safety career advice by Richard Ngo of OpenAI
• The longtermist AI governance landscape: a basic overview on the Effective Altruism forum
• Four Battlegrounds: Power in the Age of Artificial Intelligence by Paul Scharre
• The New Fire: War, Peace, and Democracy in the Age of AI by Ben Buchanan and Andrew Imbrie
• Think tank reports, such as from CSET, CNAS, CSIS
• Government strategies, such as the White House’s 2023 US National Artificial Intelligence R&D Strategic Plan, NIST’s 2023 AI Risk Management Framework, the DOD’s 2022 Responsible AI Strategy and Implementation Pathway, and the 2021 Final Report of the National Security Commission on AI
• Lessons from the Development of the Atomic Bomb by Toby Ord
• Collection of work on ‘Should you should focus on the EU if you’re interested in AI governance for longtermist/x-risk reasons?’ on the Effective Altruism Forum

Read next:  Learn about other high-impact careers

Want to consider more paths? See our list of the highest-impact career paths according to our research.

Continue →

The post AI governance and policy appeared first on 80,000 Hours.

Would we be ready for another pandemic?

It became clear in 2020 that the world hadn’t done enough to prepare for the rapid, global spread of a particularly deadly virus. Four years on, our resilience faces new tests.

Two viruses have raised global concerns:

• Mpox — formerly known as monkeypox — has been declared a public health emergency of international concern by the World Health Organisation based on its spread from the Democratic Republic of Congo (DRC) to nearby countries.
• H5N1 — a strain of bird flu — has been spreading among animals in the United States and elsewhere, with a small number of infections reported in humans.

Here’s what we know about each:

Mpox

Mpox drew international attention in 2022 when it started spreading globally, including in the US and the UK. During that outbreak, around 95,000 cases and about 180 deaths were reported. That wave largely subsided in much of the world, in part due to targeted vaccination campaigns, but the spread of another strain of the virus has sharply accelerated in Central Africa.

The strain driving the current outbreak may be significantly more deadly. Around 22,000 suspected mpox infections and more than 1,200 deaths have been reported in the DRC since January 2023..

These numbers may be artificially low because of insufficient tracking. The virus has spread to several African countries that haven’t previously seen any infections, and children appear to be particularly at risk.

The Africa CDC criticised the international community for largely “ignoring” the continent’s spike in cases after infections began to decline elsewhere.

“We urge our international partners to seize this moment to act differently and collaborate closely with Africa CDC to provide the necessary support to our Member States,” said Africa CDC Director General Dr. Jean Kaseya.

Thankfully, unlike with COVID, we don’t have to wait for a vaccine to be invented — it already exists. The challenge is producing enough of it in time and getting it where it’s needed most.

The US CDC says it is supporting countries in Central Africa with disease surveillance, testing capacity, infection prevention and control, and a vaccination strategy for the DRC.

For more on the mpox outbreak, read this report from Sentinel, a team working to forecast and prevent the largest-scale pandemics.

H5N1

Meanwhile, the H5N1 bird flu virus is currently a less dire threat, but it still raises serious concerns. It is spreading in the US among wild birds as well as farmed chickens and cows. Thirteen people in close contact with infected animals have reportedly caught the virus, according to the CDC, but it is not believed to be spreading between humans at the moment.

The US CDC says that the risk to humans generally remains low, and isn’t recommending the public take any specific measures right now. But a recent report concluded that the virus has “moderate” pandemic potential, similar to other strains of bird flu. (More details.)

In 2023, researcher Juan Cambeiro wrote a report with the Institute for Progress finding that though the risk of an H5N1 pandemic was low, it would likely be more severe than COVID-19 if it emerged.

While the CDC is monitoring the spread of the virus, some experts have warned that we’re not testing enough for new cases of H5N1. That could mean we’d lose precious time to mitigate a larger outbreak.

“I am very confident there are more people being infected than we know about,” said University of Texas Medical Branch researcher Gregory Gray in a recent article for KFF Health News. “Largely, that’s because our surveillance has been so poor.”

But there are some promising signs in the response:

• The US Department of Health and Human Services has awarded Moderna $176 million to develop an mRNA vaccine for an H5 flu variant, which officials think would protect against H5N1. (More details about existing vaccines.)
• Researchers at WastewaterSCAN have shared data tracking the geographic spread of the virus.
• The World Health Organisation has launched a project aimed at securing access to future H5N1 mRNA vaccines for low- and middle-income countries.

Overall, the risk that H5N1 will become a pandemic soon is probably low, though it’s hard to be confident:

• One prediction site, Metaculus, puts the chance of any reported human-to-human transmission of the virus by the end of the year at only 6%
• On Polymarket, the odds of the World Health Organisation declaring a bird flu pandemic this year have been around 11% recently, though with significant variation over time.
• Manifold — another prediction platform — rated the chance that the World Health Organisation declares an H5N1 pandemic by 2030 to be about 50% at the time of this writing.

What we can do

Given how bad pandemics are, we should be taking even a low risk very seriously. The Institute for Progress article estimated that, even with a 4% chance of a virus causing a pandemic as bad or worse than COVID-19, “the expected cost in terms of potential harms to the U.S. is at least $640 billion.” For comparison, the US spent $832 billion on Medicare in 2023, a program that covers 66 million Americans.

The current outbreaks of mpox and H5N1 will likely be tamed without having major global impacts. But new threats will emerge, and humanity will be much better off if we have a well-honed, consistently applied playbook for controlling these pathogens before they cause a crisis.

Some things we should be doing:

• Ensuring we have robust testing regimes when new outbreaks arise
• Routinely ramping up vaccine productions for emerging pathogens, perhaps with advance market commitments, even if the chance we’ll need them in any given case is low
• Taking more active steps to reduce the pandemic risk we face from zoonotic spillover, potential lab accidents, and intentional biological attacks

We have a lot more details about this problem in our article about preparing for the deadliest pandemics and our content on careers reducing biorisks. If you’re a good fit for this kind of work, it could be your best way to have a positive impact.

Learn more:

• The next Black Death (or worse)
• Rachel Glennerster on how “market shaping” could help solve climate change, pandemics, and other global problems
• Kevin Esvelt on cults that want to kill everyone, stealth vs wildfire pandemics, and how he felt inventing gene drives

The post Mpox and H5N1: assessing the situation appeared first on 80,000 Hours.