Superintelligence: Paths, Dangers, Strategies

Superintelligence: Paths, Dangers, Strategies by Nick Bostrom

Book: Superintelligence: Paths, Dangers, Strategies by Nick Bostrom Read Free Book Online
Authors: Nick Bostrom
Tags: science, Non-Fiction, Philosophy
chances of a beneficial outcome.
Instrumental convergence
    According to the orthogonality thesis, intelligent agents may have an enormous range of possible final goals. Nevertheless, according to what we may term the “instrumental convergence” thesis, there are some
goals likely to be pursued by almost any intelligent agent, because there are some objectives that are useful intermediaries to the achievement of almost any final goal. We can formulate this thesis as follows:
    The instrumental convergence thesis
Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents.
    In the following we will consider several categories where such convergent instrumental values may be found. 8 The likelihood that an agent will recognize the instrumental values it confronts increases (
ceteris paribus
) with the agent’s intelligence. We will therefore focus mainly on the case of a hypothetical superintelligent agent whose instrumental reasoning capacities far exceed those of any human. We will also comment on how the instrumental convergence thesis applies to the case of human beings, as this gives us occasion to elaborate some essential qualifications concerning how the instrumental convergence thesis should be interpreted and applied. Where there are convergent instrumental values, we may be able to predict some aspects of a superintelligence’s behavior even if we know virtually nothing about that superintelligence’s final goals.
    If an agent’s final goals concern the future, then in many scenarios there will be future actions it could perform to increase the probability of achieving its goals. This creates an instrumental reason for the agent to try to be around in the future—to help achieve its future-oriented goal.
    Most humans seem to place some
value on their own survival. This is not a necessary feature of artificial agents: some may be designed to place no final value whatever on their own survival. Nevertheless, many agents that do not care intrinsically about their own survival would, under a fairly wide range of conditions, care instrumentally about their own survival in order to accomplish their final goals.
Goal-content integrity
    If an agent retains its present goals into the future, then its present goals will be more likely to be achieved by its future self. This gives the agent a presentinstrumental reason to prevent alterations of its final goals. (The argument applies only to final goals. In order to attain its final goals, an intelligent agent will of course routinely want to change its
in light of new information and insight.)
    Goal-content integrity for final goals is in a sense even more fundamental than survival as a convergent instrumental motivation. Among humans, the opposite may seem to hold, but that is because survival is usually part of our final goals. For software agents, which can easily switch bodies or create exact duplicates of themselves, preservation of self as a particular implementation or a particular physical object need not be an important instrumental value. Advanced software agents might also be able to swap memories, download skills, and radically modify their cognitive architecture and personalities. A population of such agents might operate more like a “functional soup” than a society composed of distinct semi-permanent persons. 9 For some purposes, processes in such a system might be better individuated as
teleological threads
, based on their values, rather than on the basis of bodies, personalities, memories, or abilities. In such scenarios, goal-continuity might be said to
a key aspect of survival.
    Even so, there are

Similar Books

Blood and Mistletoe

E. J. Stevens

Portrait of A Novel



Jennifer Bene

The Year of the Lumin

Andrew Ryan Henke

House Rules

Jodi Picoult


Anonymous Author

Mature Themes

Andrew Durbin