We conduct research to improve the quality of life for all. Our current focus areas are outlined below.
AI governance and policy
Advanced AI could have enormous benefits but could also pose catastrophic risks. We’re particularly interested in AI governance and policy areas such as strategic analysis, compute governance, and information security.
Selected publications
- Clifton J. Equilibrium and prior selection problems in multipolar deployment. AI Alignment Forum, April 2020.
- Kokotajlo D. Persuasion Tools: AI takeover without AGI or agency? AI Alignment Forum, November 2020.
- Kokotajlo D. How Roodman’s GWP model translates to TAI timelines. LessWrong, November 2020.
- Kokotajlo D. Against GDP as a metric for timelines and takeoff speeds. AI Alignment Forum, December 2020.
Please note that our non-public research reports in this area are shared directly with AI companies and other stakeholders for internal consumption.
Cooperative AI
We’re interested in promoting cooperation between advanced AI systems.
Selected publications
- Clifton J; Riché M. Towards cooperation in learning games. Working paper, October 2020.
- Taylor M. A Measurement Agenda for AI Conflict. Working paper, June 2024.
- DiGiovanni A; Clifton J; Macé N. Safe Pareto Improvements for Expected Utility Maximizers in Program Games. arXiv preprint arXiv:2403.05103, March 2024.
- Clifton J. Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda. AI Alignment Forum, December 2019.
- Macé N; DiGiovanni A; Clifton J. Individually incentivized safe Pareto improvements in open-source bargaining. LessWrong, July 2024.
- DiGiovanni A. In defense of anthropically updating EDT. LessWrong, March 2024.
- Kokotajlo D. Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain. AI Alignment Forum, January 2021.
Please note that our non-public research reports in this area are shared directly with AI companies and other stakeholders for internal consumption.
Societal Long-Term Risks
Protecting secular democracies
We’re concerned about increasing tribalism, polarization, political dysfunction, and the erosion of secular norms that form the foundation of Western democracies. We support projects upholding core Enlightenment values—such as reason, science, liberty, impartiality, free speech, pluralism, and compassion—against authoritarian and regressive ideologies from both sides of the political spectrum.
Selected publications
- Althaus D; Kokotajlo D. Incentivizing forecasting via social media. EA Forum, December 2020.
Reducing risks from fanatical ideologies and malevolent actors
Extremist ideologies led by authoritarian tyrants—such as totalitarian communism, fascism, and religious fundamentalism—have contributed to many of history’s most catastrophic conflicts. We explore how to reduce the risks posed by such fanatical ideologies and malevolent actors.
New Directions
Improving the world—especially from a long-term perspective—is fraught with extreme uncertainty. It’s entirely possible that our current efforts are misguided.
We actively seek external input, including critiques of our current work, to refine our approach to reducing long-term risks. This may lead us to pursue different research areas.