A Beginner’s Guide to Reducing S-Risk

Suffering risks, or s-risks, are defined as “risks of events that bring about suffering in cosmically significant amounts” (Althaus and Gloor 2016). This guide explores why reducing s-risks is a crucial priority for altruistic endeavors aiming to shape the long-term future, particularly considering the potential for astronomical numbers of sentient beings in the future and the forces that could lead to severe suffering. S-risks can arise from unintended consequences of large-scale goals (incidental s-risks), intentional harm by powerful intelligent beings (agential s-risks), or processes occurring without agent intervention (natural s-risks) (Baumann 2018a).

Efforts to mitigate s-risks generally involve researching factors that exacerbate these mechanisms, focusing on emerging technologies, social institutions, and values. This research informs the development of recommendations, such as principles for the safe design of artificial intelligence, and aims to empower future generations to prevent s-risks.

1. Understanding S-Risks and Longtermism

Humans and their descendants may become capable of substantial technological and civilizational advancements, including extensive space travel and developments in artificial intelligence. These capabilities can have a significant impact on the well-being of many beings. For example, the Industrial Revolution drastically accelerated economic growth, but led to the suffering of billions of animals through factory farming.

In the long-term future, the universe may contain a number of sentient beings far greater than the current population on Earth. This could result from widespread space settlement and access to resources far exceeding those on Earth, either by biological organisms or digitally emulated minds. Digital minds, depending on their mental architectures, might be capable of suffering. Therefore, the total moral weight of their suffering could be highly significant for altruistic cause prioritization.

Prioritizing the reduction of s-risks, where large numbers of future beings experience intense involuntary suffering, is a compelling consideration. Even if one is optimistic about the long-term future, avoiding these worst-case scenarios may be paramount.

Effectively reducing s-risks requires identifying potential causes of massive suffering. While catastrophes of this scale are unprecedented, the rise of factory farming serves as an example of a discrete event that caused a large fraction of total historical suffering on Earth. Since 1970, billions of land animals have been killed annually for global meat production. Unlike the suffering caused by factory farming, some s-risks might be intentional. Future actors, like historical dictators, might deliberately cause harm.

Advanced future technologies used by agents willing to cause massive suffering have been hypothesized as the most likely foreseeable causes of s-risks. However, research focused on s-risks and interventions to reduce them is relatively recent, requiring further investigation to identify other likely sources.

Arguments for prioritizing s-risk reduction typically rely on three premises:

1. Longtermism: We should focus on influencing the long-term future because it could affect a majority of beings with moral standing, and it is feasible to have a positive influence on them.
2. Reducing the expected amount of intense suffering is a fundamental moral responsibility and a top priority among longtermist causes.
3. The most effective way to reduce expected long-term intense suffering is to avoid the worst plausible outcomes of such suffering.

Some approaches to reducing s-risks are broad enough to reduce near-term suffering, promote other values, or improve non-worst-case futures.

1.1. Addressing the “Cluelessness” Objection

Longtermism asserts the moral importance of future beings and our ability to help them. The normative premise has been defended extensively. However, the empirical premise faces the “cluelessness” objection: the probability of s-risks might be sensitive to factors about which present generations remain largely uncertain. It may be difficult to determine which actions will truly reduce s-risks in the long term. Compounding positive influence could be stopped or turned into negative influence by unpredictable factors.

One response is to focus on affecting potential persistent states, which are world states that, once entered, are not exited for a long time (if ever). These states could have different probabilities of s-risks. Interventions that foreseeably make a less s-risk-prone state more likely would be less vulnerable to the cluelessness objection.

Another approach is to build favorable conditions for future generations to reduce s-risks, as they will have more information about exacerbating factors. This requires identifying features that would enable future people to reduce s-risks, rather than hinder reduction or enable increases. Promoting norms against actions that risk immense suffering is a potentially robust candidate.

1.2. The Role of AI in S-Risks

Artificial intelligence (AI) could enable the risk factors for s-risks. This includes space settlement, the deployment of vast computational resources by agents willing to cause suffering, and value lock-in. AI systems automate complex tasks that can be scaled up far more than human labor, surpass human problem-solving, and optimize certain goals consistently.

Machine learning algorithms, including reinforcement learning agents and large language models, have demonstrated capabilities that generalize across a wide variety of tasks. If these advances continue, AI systems could develop into generally intelligent agents, which implement long-term plans culminating in the use of resources on scales larger than those of current civilizations on Earth.

Influencing the development and use of AI is a potentially effective way to reduce s-risks. AI agents with goals directed at increasing suffering or vengeful motivations could efficiently create enough suffering to constitute an s-risk if they acquire enough power. Even without “wanting” to cause an s-risk, an AI might be willing and able to do so if it is instrumental to its goals.

AI is important to influence given a focus on persistent states because human-level general AI is likely to be developed this century. This provides an opportunity to shape the initial conditions of the next iteration—superhuman AI—that could cause s-risks. Experts estimate that machines will accomplish every task better and more cheaply than human workers within the next few decades.

Furthermore, a general AI with certain terminal goals will have strong incentives to stabilize these goals if technically capable. This suggests that developing the first AI capable of both winning a competition and locking in its goals could initiate a persistent state. Therefore, avoiding training AIs with goals that motivate the creation of cosmic-scale suffering is a potential priority.

AI could pose a significant risk to humanity if its development is not guided by ethical principles and safety measures.

2. The Moral Importance of Preventing Intense Suffering

The premise that we should focus on preventing intense suffering is both normative and empirical. It is a claim about moral aims and how effective suffering reduction is at satisfying those aims.

2.1. Normative Considerations

Various moral views hold that we have a strong responsibility to prevent atrocities involving extreme, widespread suffering. Suffering-focused ethics prioritizes reducing intense suffering significantly higher than other goals. Some views hold that suffering is measurably more important, while others, such as negative utilitarianism, give intense suffering lexical priority.

Alternatively, there are views holding that avoiding futures where many beings have lives not worth living is a basic duty. The Asymmetry states that creating an individual whose life has more suffering than happiness is bad, but creating an individual whose life has more goods than suffering is at best neutral. Other views suggest a responsibility to avoid foreseeable risks of extremely bad outcomes for other beings, as outlined by the concept of “minimal morality.”

These views recommend focusing on preventing the existence of lives dominated by suffering. Views like classical utilitarianism and some forms of pluralist consequentialism may prioritize ensuring profoundly positive experiences over reducing suffering risks if those risks are relatively improbable. Other longtermist projects include reducing risks of stable totalitarianism, improving global cooperation, and promoting moral reflection.

Normative reasons for prioritizing s-risk reduction may guide action even for those who don’t consider suffering-focused views more persuasive. Decision-making under moral uncertainty suggests choosing options that are robustly positive across a range of plausible views. Efforts to improve the quality of future experiences, rather than increasing their number, may be favored.

Accounting for moral uncertainty could favor other causes, such as preserving option value for humans and descendants by prioritizing reducing risks of human extinction. However, futures with s-risks also tend to be futures where typical human values have lost control, so the option value argument does not necessarily privilege extinction risk reduction.

2.2. Empirical Considerations

Even without endorsing the moral views discussed, one might believe it is easier to reduce severe suffering than to increase goods or decrease other bads. This makes reducing suffering a higher practical priority. Long-term intense suffering could be easier to prevent because less effort is devoted to s-risk reduction compared to other efforts, such as preventing human extinction.

However, the most effective means to reduce s-risks can converge with interventions towards other goals. For example, reducing political polarization reduces s-risks, but many work on reducing polarization for other reasons. Opportunities to reduce s-risks might not be taken by those focusing on other goals.

Creating a truly utopian long-term future requires a conjunction of desirable conditions, whereas a massive increase in suffering is a relatively simple condition and thus easier to prevent. Even if human extinction is prevented, the future’s optimization for flourishing depends on which values gain power.

S-risk reduction could be a tractable longtermist goal to the extent that plausible causes of human extinction are very difficult to prevent. Aligning AI with human intent is a crucial source of extinction risk, but also a fundamentally challenging technical problem. Reducing s-risks from AI may require solving only certain subsets of the problem of controlling AI behavior. Preventing s-risks caused by malevolent human actors may be easier than influencing AI predictably.

3. Focusing on Worst-Case Outcomes

The premise of focusing on worst-case outcomes relies on a model of the future in which:

a. Large fractions of expected future suffering are due to a small set of factors over which present generations can have some influence.

b. Compared to steering from the median future toward one with no suffering, steering from worst-case outcomes toward the median future could reduce many times more suffering.

If (a) were false, we would not expect to find singular “events” responsible for cosmically significant suffering that agents could prevent. If (b) were rejected, there would be a greater focus on abolishing sources of presently existing suffering.

Two arguments support (b):

• If no s-risks occur, eliminating severe suffering is likely by default, particularly if AI alignment succeeds.

• Even assuming that contemporary causes of suffering persist indefinitely, the amount of suffering in the observable universe is arguably relatively low compared to what it could become.

The worst cases of potential future suffering are not so extremely unlikely as to be practically irrelevant, meaning that the expected suffering from s-risks is large. Trends of technological progress could enable space settlement and increase the potential of powerful agents to vastly increase suffering, conditional on incentives to do so (without necessarily wanting more suffering).

4. Potential Causes of S-Risks

Researchers have developed a typology of s-risks to clarify which are possible and the considerations favoring focusing on one cluster of scenarios over others.

4.1. Incidental S-Risks

An s-risk is incidental if it is a side effect of the actions of some agent(s) who were not trying to cause large amounts of suffering. Plausible cases involve agents with significant influence over astronomical resources finding that one of the most efficient ways to achieve some goal also causes large-scale suffering.

Inexpensive ways to produce desirable resources might entail severe suffering. Slavery is an example. Future agents in power could force astronomical numbers of digital beings to do the computational work necessary for an intergalactic civilization. S-risks from this cause are unlikely if designing these minds to experience little or no suffering is very easy.

If interstellar civilizations interested in achieving difficult goals exist in the future, they may have strong incentives to improve their understanding of the universe. They could cause suffering to beings used for scientific research, analogous to current animal testing and historical nonconsensual experimentation on humans. If future civilizations create highly detailed world simulations for purposes such as scientific research, they might instantiate sentient beings without preventing their suffering by default.

An s-risk could result if Earth-originating agents spread wildlife throughout the accessible universe without protecting the animals that evolve on the seeded planets from suffering. The amount of suffering experienced over the course of Earth’s evolutionary history would be replicated across large numbers of planets.

4.2. Agential S-Risks

An s-risk is agential if it is intentionally caused by some intelligent being. While deliberate creation of suffering appears unlikely, potential mechanisms exist.

Powerful actors might intrinsically value causing suffering and deploy advanced technologies to satisfy this goal. Malevolent traits known as the Dark Tetrad correlate with each other, suggesting that individuals who want to increase suffering may be disproportionately effective at social manipulation and inclined to seek power.

S-risks may also arise via retribution. People commonly believe that those who violate fundamental norms deserve to suffer. Hostility to one’s “outgroup” could amplify retributive sentiments. S-risks may be perpetrated by dictators who inflict disproportionate punishments on rulebreakers.

AI agents could have creation of suffering as either an intrinsic or instrumental goal. By mechanisms similar to those of the evolution of human malevolent traits, the processes by which an AI is trained may select for preferences to cause harm to others, either unconditionally or for retribution. AI training on human data could result in imitation of vengeful tendencies. If an AI’s goals include reducing suffering, an error in training could make the AI want to increase suffering. Conflicts between AIs with different goals could also result in s-risks.

Conflict between AI systems could potentially lead to the creation of scenarios causing immense suffering.

4.3. Natural S-Risks

An s-risk is natural if it occurs without the intervention of agents. A significant share of future suffering could be experienced by beings in the accessible universe who lack access to advanced technology. There may be sentient beings on the many planets capable of sustaining life who are unable to produce abundant resources.

The reasons humans currently do not relieve wild animal suffering might persist. Intelligent agents may be morally indifferent to extraterrestrial beings’ suffering, prioritize other goals, or consider intervention too intractable. Concern for wild animal suffering might remain low because it is not actively caused by human intervention. As civilization’s technological ability to intervene increases, support for reducing natural suffering may increase.

Some attempts to reduce natural s-risks may unintentionally increase incidental and agential s-risks. The most robust approaches would entail increasing future agents’ willingness to assist extraterrestrial beings conditional on already conducting space settlement.

5. Approaches to S-Risk Reduction

Effective interventions against s-risks need to reliably prevent some precondition in a way that is sensitive to near-term factors rather than inevitable (contingent) and does not easily change to some other state over time (persistent). Shaping the goals of our descendants and shaping the development of AI are two ways to prevent lock-in of increases in suffering.

Given the complexity of human societies and AI, current s-risk reduction research is devoted to identifying interventions robust to errors in our understanding. Both targeted and broad classes of interventions are considered. Targeted approaches are less likely to have backfire risks, while broad approaches may rely less strongly on a specific model of a path to impact.

Social dilemmas can arise where collective decisions make things worse than some alternative. Attempts to reduce incidental s-risks could be wasted if efforts cancel each other out.

5.1. Targeted Approaches

Targeted approaches aim to reduce s-risks by building clear models of specific pathways and finding interventions that would likely block them. To prevent lock-in of scenarios where astronomically many digital beings are subjected to suffering for economic or scientific expediency, a targeted option is to work on aligning AI with the values of its human designers. However, there are limitations and potential backfire risks. It is not clear that human values will avoid causing great suffering to digital minds. Progress in AI alignment could enable malevolent actors or increase the risk of near-miss failures.

Research in Cooperative AI is a more robust approach. Agential s-risks could result from failures of cooperation between powerful AI systems. This work entails identifying the possible causes of AI cooperation failure and proposing changes to AI design. Research progress includes:

• Research on conflict-avoiding bargaining mechanisms.
• The framework of safe Pareto improvements.
• Research on the different ways that cooperative outcomes can be made game-theoretically rational.

Cooperative AI overlaps with research in decision theory, such as Evidential Cooperation in Large Worlds (ECL).

To address incidental or agential s-risks from AI, improving coordination between, and risk-awareness within, labs developing advanced AI is an alternative to technical interventions. Coordination can reduce risks of alignment and cooperation failures, malevolent actors gaining control, and conflict between independently trained AIs.

Extraterrestrial intelligent civilizations could influence long-term suffering. Understanding their likelihood of settling space and their values is helpful for assessing counterfactual risks.

5.2. Broad Approaches

Broad s-risk reduction efforts aim to intervene on factors that are likely involved in several different pathways to s-risks. A necessary condition for any s-risk is that agents with the majority of power in the long-term future are not sufficiently motivated to prevent or avoid causing s-risks. Thus, calling attention to and developing nuanced arguments for views that highly prioritize avoiding causing severe suffering might reduce the probability of all kinds of s-risks. Efforts should promote philosophical reflection in ways that are endorsed by a wide variety of open-minded people.

Increasing concern for the suffering of more kinds of sentient beings (“moral circle expansion”) is another broad approach. In practice, this might make future space-settling civilizations more likely to create these beings on large scales, potentially leading to more suffering. AI agents that mistakenly increase the suffering of beings they are trained to care about would also cause potentially more suffering.

Shaping social institutions is helpful if lock-in is unlikely to occur soon. Changes to political systems could increase the likelihood of compromise and international cooperation. More global cooperation would slow down technological races, and greater stability of democracies may also reduce risks of malevolent actors taking power.

An alternative is to build the capacity of future people to reduce s-risks by:

• Writing detailed resources on risk factors.

• Gaining knowledge about fields useful for shaping the future.

• Promoting norms that are likely to ensure the sustainability of the s-risk reduction project.

Those interested in reducing s-risks can contribute with donations to relevant organizations or by pursuing careers in related fields.