What is AI safety?

AI safety is the work of making sure increasingly capable AI systems do what we intend, and that their development goes well for people.

New models are quickly approaching and surpassing human-level performance in a number of fields. Ensuring alignment is unsolved: making sure that increasingly intelligent and agentic models pursue goals consistent with human intentions and values.

Many researchers expect the hardest problems to arrive as systems become more autonomous and more capable than the people overseeing them. The field is small, and much of it is still open.

The work

  • Alignment

    Training systems to pursue the goals we actually intend.

  • Interpretability

    Understanding what is happening inside a model, down to individual features and circuits.

  • Evaluations

    Measuring dangerous capabilities and behaviors before a model is deployed.

  • Control

    Keeping AI systems safe to use even if they turn out to be misaligned.

  • Governance

    Rules for building and deploying frontier AI: compute governance, export controls, safety institutes and legislation.

  • Biosecurity

    Stopping AI from lowering the barrier to building biological weapons.

An under-discussed problem

Generative AI has been adopted faster than the personal computer or the internet, but safety research has lagged behind.

Training compute, FLOP (log scale)
  • Notable models
  • AI 2027 scenario
10¹⁷10¹⁹10²¹10²³10²⁵10²⁷10²⁹02.5×10²⁶5×10²⁶7.5×10²⁶10²⁷2014201620182020202220242026AlexNetTransformerGPT-3GPT-4GPT-6 AstraAI 2027 scenario

Frontier-lab figures are Epoch AI estimates. The dashed line is a scenario from AI 2027.

Length of task AI completes 50% of the time (log scale)
1 sec6 sec1 min10 min1 hr8 hr24 hr20192020202120222023202420252026GPT-2GPT-4o3Claude Mythos Preview
17 hrClaude Mythos Preview

METR software and research tasks. Confidence intervals are wide, and METR rates results above 16 hours as less reliable.

Best model score, %
  • GPQA Diamond
  • SWE-bench Verified
  • Humanity's Last Exam
0%25%50%75%100%202420252026PhD experts, GPQA DiamondGPT-4Gemini 4 Argon

GPQA and SWE-bench scores are from Epoch AI's own evaluations. Humanity's Last Exam scores are external results compiled by Epoch AI.

Each dot is about $500 million

One dot: philanthropic AI safety funding, 2025. All dots: capital spending by Microsoft, Amazon, Alphabet, Meta and Oracle, 2025.

Order-of-magnitude comparison of different kinds of spending. Safety funding counts the largest philanthropic funders only, without lab-internal or government safety budgets.

  1. more compute every year

    The compute used to train frontier AI models grew four to five times a year from 2010 to 2024, and about five times a year for frontier language models since 2020. The largest training run Epoch AI has estimated, GPT-6 Astra in 2026, used about two billion times the compute of AlexNet in 2012.

    Sources: Epoch AI, Notable AI Models; Epoch AI, Training compute of frontier AI models grows by 4-5x per year; Epoch AI, Trends; AI 2027, Compute Forecast (scenario)

  2. for the length of tasks AI can finish to double, since 2023

    AI agents can now complete software tasks that take a skilled person hours. On METR's software and research tasks, the length of task they finish half the time has doubled about every six months since 2019, and about every four months since 2023.

    Sources: METR, Measuring AI Ability to Complete Long Tasks; METR, Time horizons

  3. on PhD-level science questions in three and a half years

    Benchmarks built to stay hard are saturating within a few years. On GPQA Diamond, where PhD-level experts score about 70%, the best model went from 36% (GPT-4, 2023) to 96% (Gemini 4 Argon, 2026). Our ability to measure these systems is falling behind, and reliably steering increasingly agentic systems toward the goals we intend is still an open problem.

    Sources: Epoch AI, Benchmarking Hub; Rein et al., GPQA

  4. safety funding per dollar of big-tech capital spending, 2025

    About 2% of AI research is about safety. Philanthropic funding for AI safety reached at least $500 million in 2025. The five largest cloud companies spent about $450 billion on capital expenditure, mostly for AI, that same year.

    Sources: Emerging Technology Observatory, Still a drop in the bucket; Manifund, AI Safety Funder Bulletin; Epoch AI, Hyperscaler capex trend

There are more open problems than people working on them. Our fellowships are a way in.

Where to start

  1. PrimerStart hereWhat is AI alignment?Adam Jones, BlueDot ImpactA plain-language primer on what alignment means, why it is hard, and the main research approaches.AlignmentOverview
  2. EssayStart hereThe Bitter LessonRich SuttonWhy methods that scale with computation have beaten hand-built approaches across seventy years of AI research, and why that keeps surprising researchers.ScalingHistory of AI
  3. EssayStart hereWhy alignment could be hard with modern deep learningAjeya CotraWhy training AI on human feedback could produce systems that seem helpful while learning the wrong goals.AlignmentDeep learningDeception
  4. EssayStart hereAI could defeat all of us combinedHolden KarnofskyHow AI systems at the level of human researchers could outnumber and outpace us.RiskPower
  5. ResearchGo deeperRecent frontier models are reward hackingMETRReal examples of today's frontier models gaming their tasks, documented by METR.EvaluationsReward hacking
  6. ScenarioGo deeperAI 2027AI Futures ProjectA detailed scenario of how the next few years of AI progress could unfold, with two different endings.ForecastingScenarios

How to keep up

Express interest