I Read Every Survey on AI Researchers So You Don’t Have To
This post is mostly unedited thoughts. It has not undergone extensive editing
I’ve clustered AI/AGI surveys into 6 clusters:
AI Researcher views (e.g. published authors in NeurIPS, ICML, etc.)
AI Engineer views
AI Safety researcher views
Policymaker views
General Population views (I mainly ignore this group. AI Impacts has a great review)
Activist views
Here is what I’ve found (mostly from the first cluster):
AI Researcher Timelines are shortening
Note 1: These are judgment based forecasts. Researchers are not asked to review the literature on timelines. They are not building timeline models, just giving their best guess.
Note 2: Katja Grace points out that timelines vary strongly depending on phrasing of the question. Regardless, there is a clear trend downwards in timelines.
Forecasting and Model-based timelines
AI Impacts Blog points out that ML experts were pretty accurate in their 2016 short-run predictions. Experts expected 9 milestones to have happened by now – and 10 milestones have now happened.
Using models to predict timelines: Epoch AI has a review of the major timelines models (e.g. Ajeya Cotra, Samotsvety, etc.).
Using forecasting to predict timelines: The Forecasting Research Institute conducted The Existential Risk Persuasion Tournament, where top superforecasters and domain experts spent four months forecasting the likelihood of an ai catastrophe by 2030, 2050 and 2100.
“The median expert predicted a 20% chance of catastrophe and a 6% chance of human extinction by 2100. Superforecasters saw the chances of both catastrophe and extinction as considerably lower than did experts. The median superforecaster predicted a 9% chance of catastrophe and a 1% chance of extinction.”
A Significant Number of Researchers Expect Extremely Bad Outcomes
According to AI Impacts, ~40-50% of respondents indicated a >10% chance of catastrophic outcomes from AI progress.
Results vary highly based on phrasing of the question. It looks like they are falling for the conjunction fallacy, as section 1 has the lowest probability despite being a superset of all the others. In my opinion, the AI Impacts survey phrases questions and frames statistics in the most alarmist way possible.
When they state the median response indicates a 5% chance of catastrophic risk, it means that 50% of AI researchers believe there is a higher than 5% chance of such risk. This does not imply that only 5% of researchers consider catastrophic risk to be highly likely. I noticed some comments confused about this.
Below is a graph of respondents likelihoods of outcomes ranging from “extremely good” to “extremely bad” from 2016:
Now from 2022:
Finally, they re-ran their survey a third time in 2023:
2023 was an eventful year in AI – including the EU AI act, CAIS statement, a call for a 6 month moratorium, and the release of GPT-4, Gemini and Claude 2. All of this seems to have updated AI researchers towards optimism.
Even AI Researchers are Unfamiliar with Alignment Research
This lack of familiarity makes the last few results a lot more concerning.
One of the most shocking findings from the Arkose Interviews was that only 40% of AI researchers (n=~100) had ever heard of “alignment”. However, this data was collected in the beginning of 2022, and I’d guess this has changed dramatically since (section below for details).
An AI Impacts question asked readers to consider Stuart Russell’s formulation of the alignment problem which states “you get exactly what you ask for, not what you want”.
In 2023, 13% of AI researchers considered this to be “among the most important problems in the field”.
This surprised me as <1% of AI researchers work in alignment at the time of the survey. (source)
One explanation is that AI expertise does not necessarily translate to AI Safety expertise. Many experienced ML practitioners and researchers may have never deeply considered alignment. Much of the alignment literature discusses limits of very advanced systems which are implausible with current state-of-the-art. This may look more like sci-fi or philosophy than empirical research.
Another explanation is that researchers are concerned but are unsure where to start. This seems plausible to me. I am also unsure where to start on alignment.
What AI Safety Material is Compelling to AI Researchers?
In 2022, Vael Gates conducted interviews with AI researchers to see what AI Safety materials they find compelling. She pointed out that ML researchers prefer materials that were aimed at an ML audience, which tended to be written by ML researchers, and which tended to be more technical and less philosophical.
In 2016, Scott Alexander ran a similar experiment: he asked his subscribers to read some AI Safety material and then rate their agreement with it. He found that most ai safety material is similarly convincing.
Pessimists and Optimists Agree on Many AGI Lab Safety Measures
There is significant disagreement among alignment researchers. AI researchers disagree with alignment researchers even more. P(dooms) – the probability of catastrophic outcomes – range from 0.0000001 (Yann Lecun) to 99.99999 (Roman Yampolskiy). Both know more than I do, leaving me confused.
GOV AI has a survey on AGI lab employees, and they appear overwhelmingly in consensus about best practices. E.g. almost everyone is on board with “evaluating dangerous capabilities”. Yay.
One potential issue I noticed with this survey is that it appears the respondents were selectively chosen. Despite this, the results still make me optimistic about people being in favor of sensible regulation.
Many AI Researchers Expect a Slow Take-off
I could be wrong, but this world view seems entirely incoherent.
From AI Impacts 2023:
“If science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047. However, the chance of all human occupations becoming fully automatable was forecast to reach 10% by 2037, and 50% as late as 2116 (compared to 2164 in the 2022 survey).”
In their survey, AI Impacts uses two definitions:
High Level Machine Intelligence (HLMI): Machines capable of all human tasks
Estimated 10% by 2037, 50% by 2116
FAOL (Full automation of labor): Estimated 10% by 2027, 50% by 2047
I’m surprised by this result. The median AI researcher actually thinks that there will be an entire 69 years between a machine capable of all human jobs and full automation of labor? Is it really plausible that we build AI systems with human-level cognitive abilities, and business just continues as usual for an entire 69 years? This does not make sense to me and I don’t think these researchers have been thinking hard about their responses to these questions.
AI researchers may either be pricing in a slow takeoff, or just haven’t thought very thoroughly about this. I don’t think this position withstands much scrutiny. Here’s my reasoning:
Currently AI’s are vastly subhuman at some tasks:
The human brain is significantly more sample efficient than AI’s
You and I do not need 100,000 training examples to learn to stop at stop-signs
The human brain is 20 times more energy efficient than a graphics card
Human brains run on 15 kilacalories (62,000 joules) per hour, so 12 Watts
My PC’s power supply is 750 Watts, and my graphics card draws about 200W of that.
AI’s are subhuman at drawing, generalization, visual stuff (as shown by the ARC-AGI contest)
On the other hand, present AI’s are vastly superhuman in other ways: AI has superhuman memory, they can be copied arbitrarily, and they have the ability to work much faster/cheaper than us.
If AIs keep their advantages over humans while overcoming humans where they are currently behind, it is pretty clear that once a lab trains AI that can fully replace its human employees, it will be able to multiply its workforce 100,000x.
As Zvi points out, it seems like a shockingly large number of researchers anticipate that high level machine intelligence will just be “meh”. I would expect anything but “meh” if you successfully deploy human level intelligence. Also, right after sharing their short timelines towards high level machine intelligence, a lot of researchers think the biggest risk is… deep fakes?
Views Among All are Rapidly Updating
People's timelines, objections to ai safety, familiarity with certain ideas are all rapidly updating. Surveys from 2 years ago are essentially worthless because of the gap between pre-GPT and post-GPT views.
AI Safety Researchers do Not Think We Will Solve Alignment
Alignment researchers are also broadly in favor of an AI Pause:
Objections to AI Safety
I had Claude read all of the Arkose Interviews and then say which objections to AI Safety they brought up. Vael Gates asks interviewees what their thoughts are on the off button problem.
Using a taxonomy from AI Risk Skepticism, I clustered objections into categories. The most common objections were:
1. AI is too far away to be concerned
2. There is no clear path to agi from present ai systems
3. Other AI researchers are not concerned
These interviews are pre-GPT. It is likely the debate has evolved significantly.
All of the objections are:
1.1 Priority objection: AGI is Too Far so it isn't worth worrying about
1.2 Priority objection: A Soft Takeoff is more likely and so we will have Time to Prepare
1.3 Priority objection: There is No Obvious Path to Get to AGI from Current AI
1.4 Priority objection: Something Else is More Important than AI safety / alignment
1.5 Priority objection: Short Term AI Concerns are more important than AI safety
2.1 Technical Objection: AI / AGI Doesn’t Exist, developments in AI are not necessarily progress towards AGI
2.2 Technical Objection: Superintelligence is Impossible
2.3 Technical Objection: Self-Improvement is Impossible
2.4 Technical Objection: AI Can’t be Conscious Proponents argue that in order to be dangerous AI has to be conscious
2.5 Technical Objection: AI Can just be a Tool
2.6 Technical Objection: We can Always just turn it off
2.7 Technical Objection: We can reprogram AIs if we don't like what they do
2.8 Technical Objection: AI Doesn't have a body so it can't hurt us
2.9 Technical Objection: If AI is as Capable as You Say, it Will not Make Dumb Mistakes
2.10 Technical Objection: Superintelligence Would (Probably) Not Be Catastrophic
2.11 Technical Objection: Self-preservation and Control Drives Don't Just Appear They Have to be Programmed In
2.12 Technical Objection: AI can't generate novel plans
3.1 AI Safety Objections: AI Safety Can’t be Done Today
3.2 AI Safety Objections: AI Can’t be Safe
4.1 Ethical Objections: Superintelligence is Benevolence
4.2 Ethical Objections: Let the Smarter Beings Win
5.1 Biased Objections: AI Safety Researchers are Non-Coders
5.2 Biased Objections: Majority of AI Researchers is not Worried
5.3 Biased Objections: Keep it Quiet
5.4 Biased Objections: Safety Work just Creates an Overhead Slowing Down Research
5.5 Biased Objections: Heads in the Sand
5.6 Biased Objections: If we don't do it, Someone else will
5.7 Biased Objections: AI Safety Requires Global Cooperation
6.1 Miscellaneous Objection: So Easy it will be Solved Automatically
6.2 Miscellaneous Objection: AI Regulation Will Prevent ProblemsAI Safety Researchers do Not Think We Will Solve Alignment
Here is a list of what I looked at:
https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/
Not that useful
timelines
katja grace 2024: thousands of authors on the impacts of ai
https://www.lesswrong.com/posts/NfPxAp5uwgZugwovY/ai-impacts-survey-december-2023-edition
Really useful, especially the part where they present stuart russels thing and 13% say this is the most important problem
13%.. Seriously say this is among the most important problem
Useful AI Impacts survey, probably the largest of its kind
what do ai researchers find compelling from safety research? Authors are asked to rate stuff – my key takeaway is that engineers prefer empirical articles and less philosophical articles
Likely a big crux
The authors email 97 researchers, transcribe stuff, etc.
What do AI researchers expect: https://www.lesswrong.com/posts/Zq2HaihaDy7sSarMz/how-bad-a-future-do-ml-researchers-expect
Talking to 100 academics: lessons learned
Surveys of AI Safety Researchers
https://www.lesswrong.com/posts/rXSBvSKvKdaNkhLeJ/takeaways-from-a-survey-on-ai-alignment-resources
What are the most useful strategies
I like this report but it cannot be shared publicly :/
How much do people like different ais resources (eg. rob miles, axrp) https://www.lesswrong.com/posts/rXSBvSKvKdaNkhLeJ/takeaways-from-a-survey-on-ai-alignment-resources
https://forum.effectivealtruism.org/posts/uJioXCz5Foo9eqpJ9/big-picture-ai-safety-introduction Key takeaways for most useful things
the technical solutions we might come up with,
spreading a safety mindset through AI research,
promoting sensible AI regulation,
and helping build a fundamental science of AI Safety
Need concrete problems to work on
Bensinger risk survey: https://www.alignmentforum.org/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results
What ai safety researchers did to become ai safety researchers
https://www.lesswrong.com/posts/XTdByFM6cmgB3taEN/key-takeaways-from-our-ea-and-alignment-research-surveys#Alignment_researchers_support_a_pause not insanely useful
https://arxiv.org/pdf/2305.07153 survey on what leading experts think AGI labs should do (both AI Safety experts and AI experts)
I found this HIGHLY useful:
Even those outside of ai safety are likely OVERWHELMINGLY in support of “run dangerous capabilities evaluations” or “allow third party audits”
Existential risk among experts:
https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results
Surveys of the general public
I generally think these are LESS useful, and there are a lot of them
I list two relevant ones
Still practical, relevant, worth considering, worth presenting
https://slatestarcodex.com/2016/10/24/ai-persuasion-experiment-results/ asks readers how convinced they are of catastrophic risk, assigns reading, etc.
This AI risk education research indicates that most AI risk communication strategies are effective and are not counter-productive
https://rethinkpriorities.org/publications/us-public-opinion-of-ai-policy-and-risk polls the public opinion on A pause on certain kinds of AI research
Should AI be regulated (akin to the FDA)?
Worry about negative effects of AI
https://futureoflife.org/ai/superintelligence-survey/
Max tegmark post life3.0 survey, I really like these questions
https://aiindex.stanford.edu/report/
Not really useful, talks about surveying different countries
Surveying Content Creators
https://onlinelibrary.wiley.com/doi/full/10.1111/risa.14299 Framing societal threat and efficacy in YouTube videos about artificial intelligence
Surveys on policymakers
A few interesting points:
Most influential areas impacting people's ai perceptions are:
Tech companies
Pop culture
Academia
This should probably be flipped in a good world?
Or maybe I’m in the west coast academic ivory tower and should be more sympathetic? Either way, tech companies probably shouldnt drive the narrative around ai
FURTHER point: this is a little concerning given how deeply ingrained tech companies are in academia, they pretty much control academia. Every cs phd wants to intern at FAANG, etc
Not very useful
Findings: more in favor of using ai in natural disaster planning than in prison sentences
https://www.lesswrong.com/posts/2sLwt2cSAag74nsdN/speaking-to-congressional-staffers-about-ai-risk
Akash Wasil speaking to congressional staffers
Inspiration for good surveys:
Seriously anything Rob Bensinger is involved in