转载自VernorVinge的主页，93年的文章

The original version of this article was presented at the VISION-21

Symposium sponsored by NASA Lewis Research Center and the Ohio

Aerospace Institute, March 30-31, 1993. A slightly changed version

appeared in the Winter 1993 issue of Whole Earth Review. 

Abstract

Within thirty years, we will have the technological means to create

superhuman intelligence. Shortly after, the human era will be ended. 

Is such progress avoidable? If not to be avoided, can events be guided

so that we may survive? These questions are investigated. Some possible

answers (and some further dangers) are presented. 

What is The Singularity?

The acceleration of technological progress has been the central feature

of this century. I argue in this paper that we are on the edge of change

comparable to the rise of human life on Earth. The precise cause of this

change is the imminent creation by technology of entities with greater

than human intelligence. There are several means by which science may

achieve this breakthrough (and this is another reason for having

confidence that the event will occur): 
There may be developed computers that are "awake" and superhumanly

intelligent. (To date, there has been much controversy as to whether we

can create human equivalence in a machine. But if the answer is "yes, we

can", then there is little doubt that beings more intelligent can be

constructed shortly thereafter.) 
Large computer networks (and their associated users) may "wake up" as

a superhumanly intelligent entity. 
Computer/human interfaces may become so intimate that users may

reasonably be considered superhumanly intelligent. 
Biological science may provide means to improve natural human intellect.

 

The first three possibilities depend in large part on improvements in

computer hardware. Progress in computer hardware has followed an

amazingly steady curve in the last few decades [17]. Based largely on this

trend, I believe that the creation of greater than human intelligence will

occur during the next thirty years. (Charles Platt [20] has pointed out that

AI enthusiasts have been making claims like this for the last thirty years.

Just so I'm not guilty of a relative-time ambiguity, let me more specific: I'll

be surprised if this event occurs before 2005 or after 2030.) 

What are the consequences of this event? When greater-than-human

intelligence drives progress, that progress will be much more rapid. In

fact, there seems no reason why progress itself would not involve the

creation of still more intelligent entities -- on a still-shorter time scale. The

best analogy that I see is with the evolutionary past: Animals can adapt to

problems and make inventions, but often no faster than natural selection

can do its work -- the world acts as its own simulator in the case of

natural selection. We humans have the ability to internalize the world and

conduct "what if's" in our heads; we can solve many problems thousands

of times faster than natural selection. Now, by creating the means to

execute those simulations at much higher speeds, we are entering a

regime as radically different from our human past as we humans are

from the lower animals. 

From the human point of view this change will be a throwing away of all

the previous rules, perhaps in the blink of an eye, an exponential runaway

beyond any hope of control. Developments that before were thought

might only happen in "a million years" (if ever) will likely happen in the

next century. (In [5], Greg Bear paints a picture of the major changes

happening in a matter of hours.) 

I think it's fair to call this event a singularity ("the Singularity" for the

purposes of this paper). It is a point where our old models must be

discarded and a new reality rules. As we move closer to this point, it will

loom vaster and vaster over human affairs till the notion becomes a

commonplace. Yet when it finally happens it may still be a great surprise

and a greater unknown. In the 1950s there were very few who saw it: Stan

Ulam [28] paraphrased John von Neumann as saying: 

One conversation centered on the ever accelerating progress of

technology and changes in the mode of human life, which gives the

appearance of approaching some essential singularity in the history of

the race beyond which human affairs, as we know them, could not

continue. 

Von Neumann even uses the term singularity, though it appears he is

thinking of normal progress, not the creation of superhuman intellect.

(For me, the superhumanity is the essence of the Singularity. Without that

we would get a glut of technical riches, never properly absorbed (see

[25]).) 

In the 1960s there was recognition of some of the implications of

superhuman intelligence. I. J. Good wrote [11]: 

Let an ultraintelligent machine be defined as a machine that can far

surpass all the intellectual activities of any any man however clever. Since

the design of machines is one of these intellectual activities, an

ultraintelligent machine could design even better machines; there would

then unquestionably be an "intelligence explosion," and the intelligence of

man would be left far behind. Thus the first ultraintelligent machine is the

_last_ invention that man need ever make, provided that the machine is

docile enough to tell us how to keep it under control. ... It is more

probable than not that, within the twentieth century, an ultraintelligent

machine will be built and that it will be the last invention that man need

make. 

Good has captured the essence of the runaway, but does not pursue its

most disturbing consequences. Any intelligent machine of the sort he

describes would not be humankind's "tool" -- any more than humans are

the tools of rabbits or robins or chimpanzees. 

Through the '60s and '70s and '80s, recognition of the cataclysm spread

[29] [1] [31] [5]. Perhaps it was the science-fiction writers who felt the first

concrete impact. After all, the "hard" science-fiction writers are the ones

who try to write specific stories about all that technology may do for us.

More and more, these writers felt an opaque wall across the future.

Once, they could put such fantasies millions of years in the future [24].

Now they saw that their most diligent extrapolations resulted in the

unknowable ... soon. Once, galactic empires might have seemed a Post-

Human domain. Now, sadly, even interplanetary ones are. 

What about the '90s and the '00s and the '10s, as we slide toward the

edge? How will the approach of the Singularity spread across the human

world view? For a while yet, the general critics of machine sapience will

have good press. After all, till we have hardware as powerful as a human

brain it is probably foolish to think we'll be able to create human

equivalent (or greater) intelligence. (There is the far-fetched possibility

that we could make a human equivalent out of less powerful hardware, if

we were willing to give up speed, if we were willing to settle for an

artificial being who was literally slow [30]. But it's much more likely that

devising the software will be a tricky process, involving lots of false starts

and experimentation. If so, then the arrival of self-aware machines will not

happen till after the development of hardware that is substantially more

powerful than humans' natural equipment.) 

But as time passes, we should see more symptoms. The dilemma felt by

science fiction writers will be perceived in other creative endeavors. (I

have heard thoughtful comic book writers worry about how to have

spectacular effects when everything visible can be produced by the

technologically commonplace.) We will see automation replacing higher

and higher level jobs. We have tools right now (symbolic math programs,

cad/cam) that release us from most low-level drudgery. Or put another

way: The work that is truly productive is the domain of a steadily smaller

and more elite fraction of humanity. In the coming of the Singularity, we

are seeing the predictions of _true_ technological unemployment finally

come true. 

Another symptom of progress toward the Singularity: ideas themselves

should spread ever faster, and even the most radical will quickly become

commonplace. When I began writing science fiction in the middle '60s, it

seemed very easy to find ideas that took decades to percolate into the

cultural consciousness; now the lead time seems more like eighteen

months. (Of course, this could just be me losing my imagination as I get

old, but I see the effect in others too.) Like the shock in a compressible

flow, the Singularity moves closer as we accelerate through the critical

speed. 

And what of the arrival of the Singularity itself? What can be said of its

actual appearance? Since it involves an intellectual runaway, it will

probably occur faster than any technical revolution seen so far. The

precipitating event will likely be unexpected -- perhaps even to the

researchers involved. ("But all our previous models were catatonic! We

were just tweaking some parameters....") If networking is widespread

enough (into ubiquitous embedded systems), it may seem as if our

artifacts as a whole had suddenly wakened. 

And what happens a month or two (or a day or two) after that? I have

only analogies to point to: The rise of humanki(This article may be

reproduced for noncommercial purposes if it is copied in its entirety,

including this notice.) 

The original version of this article was presented at the VISION-21

Symposium sponsored by NASA Lewis Research Center and the Ohio

Aerospace Institute, March 30-31, 1993. A slightly changed version

appeared in the Winter 1993 issue of Whole Earth Review. 

Abstract

Within thirty years, we will have the technological means to create

superhuman intelligence. Shortly after, the human era will be ended. 

Is such progress avoidable? If not to be avoided, can events be guided

so that we may survive? These questions are investigated. Some possible

answers (and some further dangers) are presented. 

What is The Singularity?

The acceleration of technological progress has been the central feature

of this century. I argue in this paper that we are on the edge of change

comparable to the rise of human life on Earth. The precise cause of this

change is the imminent creation by technology of entities with greater

than human intelligence. There are several means by which science may

achieve this breakthrough (and this is another reason for having

confidence that the event will occur): 
There may be developed computers that are "awake" and superhumanly

intelligent. (To date, there has been much controversy as to whether we

can create human equivalence in a machine. But if the answer is "yes, we

can", then there is little doubt that beings more intelligent can be

constructed shortly thereafter.) 
Large computer networks (and their associated users) may "wake up" as

a superhumanly intelligent entity. 
Computer/human interfaces may become so intimate that users may

reasonably be considered superhumanly intelligent. 
Biological science may provide means to improve natural human intellect.

 

The first three possibilities depend in large part on improvements in

computer hardware. Progress in computer hardware has followed an

amazingly steady curve in the last few decades [17]. Based largely on this

trend, I believe that the creation of greater than human intelligence will

occur during the next thirty years. (Charles Platt [20] has pointed out that

AI enthusiasts have been making claims like this for the last thirty years.

Just so I'm not guilty of a relative-time ambiguity, let me more specific: I'll

be surprised if this event occurs before 2005 or after 2030.) 

What are the consequences of this event? When greater-than-human

intelligence drives progress, that progress will be much more rapid. In

fact, there seems no reason why progress itself would not involve the

creation of still more intelligent entities -- on a still-shorter time scale. The

best analogy that I see is with the evolutionary past: Animals can adapt to

problems and make inventions, but often no faster than natural selection

can do its work -- the world acts as its own simulator in the case of

natural selection. We humans have the ability to internalize the world and

conduct "what if's" in our heads; we can solve many problems thousands

of times faster than natural selection. Now, by creating the means to

execute those simulations at much higher speeds, we are entering a

regime as radically different from our human past as we humans are

from the lower animals. 

From the human point of view this change will be a throwing away of all

the previous rules, perhaps in the blink of an eye, an exponential runaway

beyond any hope of control. Developments that before were thought

might only happen in "a million years" (if ever) will likely happen in the

next century. (In [5], Greg Bear paints a picture of the major changes

happening in a matter of hours.) 

I think it's fair to call this event a singularity ("the Singularity" for the

purposes of this paper). It is a point where our old models must be

discarded and a new reality rules. As we move closer to this point, it will

loom vaster and vaster over human affairs till the notion becomes a

commonplace. Yet when it finally happens it may still be a great surprise

and a greater unknown. In the 1950s there were very few who saw it: Stan

Ulam [28] paraphrased John von Neumann as saying: 

One conversation centered on the ever accelerating progress of

technology and changes in the mode of human life, which gives the

appearance of approaching some essential singularity in the history of

the race beyond which human affairs, as we know them, could not

continue. 

Von Neumann even uses the term singularity, though it appears he is

thinking of normal progress, not the creation of superhuman intellect.

(For me, the superhumanity is the essence of the Singularity. Without that

we would get a glut of technical riches, never properly absorbed (see

[25]).) 

In the 1960s there was recognition of some of the implications of

superhuman intelligence. I. J. Good wrote [11]: 

Let an ultraintelligent machine be defined as a machine that can far

surpass all the intellectual activities of any any man however clever. Since

the design of machines is one of these intellectual activities, an

ultraintelligent machine could design even better machines; there would

then unquestionably be an "intelligence explosion," and the intelligence of

man would be left far behind. Thus the first ultraintelligent machine is the

_last_ invention that man need ever make, provided that the machine is

docile enough to tell us how to keep it under control. ... It is more

probable than not that, within the twentieth century, an ultraintelligent

machine will be built and that it will be the last invention that man need

make. 

Good has captured the essence of the runaway, but does not pursue its

most disturbing consequences. Any intelligent machine of the sort he

describes would not be humankind's "tool" -- any more than humans are

the tools of rabbits or robins or chimpanzees. 

Through the '60s and '70s and '80s, recognition of the cataclysm spread

[29] [1] [31] [5]. Perhaps it was the science-fiction writers who felt the first

concrete impact. After all, the "hard" science-fiction writers are the ones

who try to write specific stories about all that technology may do for us.

More and more, these writers felt an opaque wall across the future.

Once, they could put such fantasies millions of years in the future [24].

Now they saw that their most diligent extrapolations resulted in the

unknowable ... soon. Once, galactic empires might have seemed a Post-

Human domain. Now, sadly, even interplanetary ones are. 

What about the '90s and the '00s and the '10s, as we slide toward the

edge? How will the approach of the Singularity spread across the human

world view? For a while yet, the general critics of machine sapience will

have good press. After all, till we have hardware as powerful as a human

brain it is probably foolish to think we'll be able to create human

equivalent (or greater) intelligence. (There is the far-fetched possibility

that we could make a human equivalent out of less powerful hardware, if

we were willing to give up speed, if we were willing to settle for an

artificial being who was literally slow [30]. But it's much more likely that

devising the software will be a tricky process, involving lots of false starts

and experimentation. If so, then the arrival of self-aware machines will not

happen till after the development of hardware that is substantially more

powerful than humans' natural equipment.) 

But as time passes, we should see more symptoms. The dilemma felt by

science fiction writers will be perceived in other creative endeavors. (I

have heard thoughtful comic book writers worry about how to have

spectacular effects when everything visible can be produced by the

technologically commonplace.) We will see automation replacing higher

and higher level jobs. We have tools right now (symbolic math programs,

cad/cam) that release us from most low-level drudgery. Or put another

way: The work that is truly productive is the domain of a steadily smaller

and more elite fraction of humanity. In the coming of the Singularity, we

are seeing the predictions of _true_ technological unemployment finally

come true. 

Another symptom of progress toward the Singularity: ideas themselves

should spread ever faster, and even the most radical will quickly become

commonplace. When I began writing science fiction in the middle '60s, it

seemed very easy to find ideas that took decades to percolate into the

cultural consciousness; now the lead time seems more like eighteen

months. (Of course, this could just be me losing my imagination as I get

old, but I see the effect in others too.) Like the shock in a compressible

flow, the Singularity moves closer as we accelerate through the critical

speed. 

And what of the arrival of the Singularity itself? What can be said of its

actual appearance? Since it involves an intellectual runaway, it will

probably occur faster than any technical revolution seen so far. The

precipitating event will likely be unexpected -- perhaps even to the

researchers involved. ("But all our previous models were catatonic! We

were just tweaking some parameters....") If networking is widespread

enough (into ubiquitous embedded systems), it may seem as if our

artifacts as a whole had suddenly wakened. 

And what happens a month or two (or a day or two) after that? I have

only analogies to point to: The rise of humankind. We will be in the Post

-Human era. And for all my rampant technological optimism, sometimes I

think I'd be more comfortable if I were regarding these transcendental

events from one thousand years remove ... instead of twenty. 

Can the Singularity be Avoided?

Well, maybe it won't happen at all: Sometimes I try to imagine the

symptoms that we should expect to see if the Singularity is not to

develop. There are the widely respected arguments of Penrose [19] and

Searle [22] against the practicality of machine sapience. In August of

1992, Thinking Machines Corporation held a workshop to investigate the

question "How We Will Build a Machine that Thinks" [27]. As you might

guess from the workshop's title, the participants were not especially

supportive of the arguments against machine intelligence. In fact, there

was general agreement that minds can exist on nonbiological substrates

and that algorithms are of central importance to the existence of minds.

However, there was much debate about the raw hardware power that is

present in organic brains. A minority felt that the largest 1992 computers

were within three orders of magnitude of the power of the human brain.

The majority of the participants agreed with Moravec's estimate [17] that

we are ten to forty years away from hardware parity. And yet there was

another minority who pointed to [7] [21], and conjectured that the

computational competence of single neurons may be far higher than

generally believed. If so, our present computer hardware might be as

much as _ten_ orders of magnitude short of the equipment we carry

around in our heads. If this is true (or for that matter, if the Penrose or

Searle critique is valid), we might never see a Singularity. Instead, in the

early '00s we would find our hardware performance curves beginning to

level off -- this because of our inability to automate the design work

needed to support further hardware improvements. We'd end up with

some _very_ powerful hardware, but without the ability to push it further.

Commercial digital signal processing might be awesome, giving an

analog appearance even to digital operations, but nothing would ever

"wake up" and there would never be the intellectual runaway which is the

essence of the Singularity. It would likely be seen as a golden age ... and

it would also be an end of progress. This is very like the future predicted

by Gunther Stent. In fact, on page 137 of [25], Stent explicitly cites the

development of transhuman intelligence as a sufficient condition to break

his projections. 

But if the technological Singularity can happen, it will. Even if all the

governments of the world were to understand the "threat" and be in

deadly fear of it, progress toward the goal would continue. In fiction,

there have been stories of laws passed forbidding the construction of "a

machine in the likeness of the human mind" [13]. In fact, the competitive

advantage -- economic, military, even artistic -- of every advance in

automation is so compelling that passing laws, or having customs, that

forbid such things merely assures that someone else will get them first. 

Eric Drexler [8] has provided spectacular insights about how far technical

improvement may go. He agrees that superhuman intelligences will be

available in the near future -- and that such entities pose a threat to the

human status quo. But Drexler argues that we can confine such

transhuman devices so that their results can be examined and used

safely. This is I. J. Good's ultraintelligent machine, with a dose of caution. I

argue that confinement is intrinsically impractical. For the case of physical

confinement: Imagine yourself locked in your home with only limited data

access to the outside, to your masters. If those masters thought at a rate

-- say -- one million times slower than you, there is little doubt that over a

period of years (your time) you could come up with "helpful advice" that

would incidentally set you free. (I call this "fast thinking" form of

superintelligence "weak superhumanity". Such a "weakly superhuman"

entity would probably burn out in a few weeks of outside time. "Strong

superhumanity" would be more than cranking up the clock speed on a

human-equivalent mind. It's hard to say precisely what "strong

superhumanity" would be like, but the difference appears to be

profound. Imagine running a dog mind at very high speed. Would a

thousand years of doggy living add up to any human insight? (Now if the

dog mind were cleverly rewired and _then_ run at high speed, we might

see something different....) Many speculations about superintelligence

seem to be based on the weakly superhuman model. I believe that our

best guesses about the post-Singularity world can be obtained by

thinking on the nature of strong superhumanity. I will return to this point

later in the paper.) 

Another approach to confinement is to build _rules_ into the mind of the

created superhuman entity (for example, Asimov's Laws [3]). I think that

any rules strict enough to be effective would also produce a device

whose ability was clearly inferior to the unfettered versions (and so

human competition would favor the development of the those more

dangerous models). Still, the Asimov dream is a wonderful one: Imagine

a willing slave, who has 1000 times your capabilities in every way. Imagine

a creature who could satisfy your every safe wish (whatever that means)

and still have 99.9% of its time free for other activities. There would be a

new universe we never really understood, but filled with benevolent gods

(though one of _my_ wishes might be to become one of them). 

If the Singularity can not be prevented or confined, just how bad could

the Post-Human era be? Well ... pretty bad. The physical extinction of the

human race is one possibility. (Or as Eric Drexler put it of

nanotechnology: Given all that such technology can do, perhaps

governments would simply decide that they no longer need citizens!). Yet

physical extinction may not be the scariest possibility. Again, analogies:

Think of the different ways we relate to animals. Some of the crude

physical abuses are implausible, yet.... In a Post-Human world there

would still be plenty of niches where human equivalent automation would

be desirable: embedded systems in autonomous devices, self-aware

daemons in the lower functioning of larger sentients. (A strongly

superhuman intelligence would likely be a Society of Mind [16] with some

very competent components.) Some of these human equivalents might be

used for nothing more than digital signal processing. They would be

more like whales than humans. Others might be very human-like, yet with

a one-sidedness, a _dedication_ that would put them in a mental hospital

in our era. Though none of these creatures might be flesh-and-blood

humans, they might be the closest things in the new enviroment to what

we call human now. (I. J. Good had something to say about this, though

at this late date the advice may be moot: Good [12] proposed a "Meta-

Golden Rule", which might be paraphrased as "Treat your inferiors as you

would be treated by your superiors." It's a wonderful, paradoxical idea

(and most of my friends don't believe it) since the game-theoretic payoff

is so hard to articulate. Yet if we were able to follow it, in some sense that

might say something about the plausibility of such kindness in this

universe.)

I have argued above that we cannot prevent the Singularity, that its

coming is an inevitable consequence of the humans' natural

competitiveness and the possibilities inherent in technology. And yet ... we

are the initiators. Even the largest avalanche is triggered by small things.

We have the freedom to establish initial conditions, make things happen

in ways that are less inimical than others. Of course (as with starting

avalanches), it may not be clear what the right guiding nudge really is: 

Other Paths to the Singularity: Intelligence Amplification_
When people speak of creating superhumanly intelligent beings, they are

usually imagining an AI project. But as I noted at the beginning of this

paper, there are other paths to superhumanity. Computer networks and

human-computer interfaces seem more mundane than AI, and yet they

could lead to the Singularity. I call this contrasting approach Intelligence

Amplification (IA). IA is something that is proceeding very naturally, in

most cases not even recognized by its developers for what it is. But every

time our ability to access information and to communicate it to others is

improved, in some sense we have achieved an increase over natural

intelligence. Even now, the team of a PhD human and good computer

workstation (even an off-net workstation!) could probably max any

written intelligence test in existence. 

And it's very likely that IA is a much easier road to the achievement of

superhumanity than pure AI. In humans, the hardest development

problems have already been solved. Building up from within ourselves

ought to be easier than figuring out first what we really are and then

building machines that are all of that. And there is at least conjectural

precedent for this approach. Cairns-Smith [6] has speculated that

biological life may have begun as an adjunct to still more primitive life

based on crystalline growth. Lynn Margulis (in [15] and elsewhere) has

made strong arguments that mutualism is a great driving force in

evolution. 

Note that I am not proposing that AI research be ignored or less funded.

What goes on with AI will often have applications in IA, and vice versa. I

am suggesting that we recognize that in network and interface research

there is something as profound (and potential wild) as Artificial

Intelligence. With that insight, we may see projects that are not as directly

applicable as conventional interface and network design work, but which

serve to advance us toward the Singularity along the IA path. 

Here are some possible projects that take on special significance, given

the IA point of view: 
Human/computer team automation: Take problems that are normally

considered for purely machine solution (like hill-climbing problems), and

design programs and interfaces that take a advantage of humans'

intuition and available computer hardware. Considering all the

bizarreness of higher dimensional hill-climbing problems (and the neat

algorithms that have been devised for their solution), there could be

some very interesting displays and control tools provided to the human

team member. 
Develop human/computer symbiosis in art: Combine the graphic

generation capability of modern machines and the esthetic sensibility of

humans. Of course, there has been an enormous amount of research in

designing computer aids for artists, as labor saving tools. I'm suggesting

that we explicitly aim for a greater merging of competence, that we

explicitly recognize the cooperative approach that is possible. Karl Sims

[23] has done wonderful work in this direction. 
Allow human/computer teams at chess tournaments. We already have

programs that can play better than almost all humans. But how much

work has been done on how this power could be used by a human, to

get something even better? If such teams were allowed in at least some

chess tournaments, it could have the positive effect on IA research that

allowing computers in tournaments had for the corresponding niche in

AI. 
Develop interfaces that allow computer and network access without

requiring the human to be tied to one spot, sitting in front of a computer.

(This is an aspect of IA that fits so well with known economic advantages

that lots of effort is already being spent on it.) 
Develop more symmetrical decision support systems. A popular

research/product area in recent years has been decision support

systems. This is a form of IA, but may be too focussed on systems that

are oracular. As much as the program giving the user information, there

must be the idea of the user giving the program guidance. 
Use local area nets to make human teams that really work (ie, are more

effective than their component members). This is generally the area of

"groupware", already a very popular commercial pursuit. The change in

viewpoint here would be to regard the group activity as a combination

organism. In one sense, this suggestion might be regarded as the goal

of inventing a "Rules of Order" for such combination operations. For

instance, group focus might be more easily maintained than in classical

meetings. Expertise of individual human members could be isolated from

ego issues such that the contribution of different members is focussed on

the team project. And of course shared data bases could be used much

more conveniently than in conventional committee operations. (Note that

this suggestion is aimed at team operations rather than political

meetings. In a political setting, the automation described above would

simply enforce the power of the persons making the rules!) 
Exploit the worldwide Internet as a combination human/machine tool. Of

all the items on the list, progress in this is proceeding the fastest and

may run us into the Singularity before anything else. The power and

influence of even the present-day Internet is vastly underestimated. For

instance, I think our contemporary computer systems would break under

the weight of their own complexity if it weren't for the edge that the

USENET "group mind" gives the system administration and support

people! The very anarchy of the worldwide net development is evidence

of its potential. As connectivity and bandwidth and archive size and

computer speed all increase, we are seeing something like Lynn Margulis'

[15] vision of the biosphere as data processor recapitulated, but at a

million times greater speed and with millions of humanly intelligent

agents (ourselves). 
The above examples illustrate research that can be done within the

context of contemporary computer science departments. There are other

paradigms. For example, much of the work in Artificial Intelligence and

neural nets would benefit from a closer connection with biological life.

Instead of simply trying to model and understand biological life with

computers, research could be directed toward the creation of composite

systems that rely on biological life for guidance or for the providing

features we don't understand well enough yet to implement in hardware.

A long-time dream of science-fiction has been direct brain to computer

interfaces [2] [29]. In fact, there is concrete work that can be done (and is

being done) in this area: 
Limb prosthetics is a topic of direct commercial applicability. Nerve to

silicon transducers can be made [14]. This is an exciting, near-term step

toward direct communication. 
Direct links into brains seem feasible, if the bit rate is low: given human

learning flexibility, the actual brain neuron targets might not have to be

precisely selected. Even 100 bits per second would be of great use to

stroke victims who would otherwise be confined to menu-driven

interfaces. 
Plugging in to the optic trunk has the potential for bandwidths of 1

Mbit/second or so. But for this, we need to know the fine-scale

architecture of vision, and we need to place an enormous web of

electrodes with exquisite precision. If we want our high bandwidth

connection to be _in addition_ to what paths are already present in the

brain, the problem becomes vastly more intractable. Just sticking a grid

of high-bandwidth receivers into a brain certainly won't do it. But

suppose that the high-bandwidth grid were present while the brain

structure was actually setting up, as the embryo develops. That suggests: 
Animal embryo experiments. I wouldn't expect any IA success in the first

years of such research, but giving developing brains access to complex

simulated neural structures might be very interesting to the people who

study how the embryonic brain develops. In the long run, such

experiments might produce animals with additional sense paths and

interesting intellectual abilities. 
Originally, I had hoped that this discussion of IA would yield some clearly

safer approaches to the Singularity. (After all, IA allows our participation

in a kind of transcendance.) Alas, looking back over these IA proposals,

about all I am sure of is that they should be considered, that they may

give us more options. But as for safety ... well, some of the suggestions

are a little scarey on their face. One of my informal reviewers pointed out

that IA for individual humans creates a rather sinister elite. We humans

have millions of years of evolutionary baggage that makes us regard

competition in a deadly light. Much of that deadliness may not be

necessary in today's world, one where losers take on the winners' tricks

and are coopted into the winners' enterprises. A creature that was built

_de novo_ might possibly be a much more benign entity than one with a

kernel based on fang and talon. And even the egalitarian view of an

Internet that wakes up along with all mankind can be viewed as a

nightmare [26]. 

The problem is not simply that the Singularity represents the passing of

humankind from center stage, but that it contradicts our most deeply

held notions of being. I think a closer look at the notion of strong

superhumanity can show why that is. 

Strong Superhumanity and the Best We Can Ask for
Suppose we could tailor the Singularity. Suppose we could attain our

most extravagant hopes. What then would we ask for: That humans

themselves would become their own successors, that whatever injustice

occurs would be tempered by our knowledge of our roots. For those

who remained unaltered, the goal would be benign treatment (perhaps

even giving the stay-behinds the appearance of being masters of godlike

slaves). It could be a golden age that also involved progress

(overleaping Stent's barrier). Immortality (or at least a lifetime as long as

we can make the universe survive [10] [4]) would be achievable. 

But in this brightest and kindest world, the philosophical problems

themselves become intimidating. A mind that stays at the same capacity

cannot live forever; after a few thousand years it would look more like a

repeating tape loop than a person. (The most chilling picture I have seen

of this is in [18].) To live indefinitely long, the mind itself must grow ... and

when it becomes great enough, and looks back ... what fellow-feeling can

it have with the soul that it was originally? Certainly the later being would

be everything the original was, but so much vastly more. And so even for

the individual, the Cairns-Smith or Lynn Margulis notion of new life

growing incrementally out of the old must still be valid. 

This "problem" about immortality comes up in much more direct ways.

The notion of ego and self-awareness has been the bedrock of the

hardheaded rationalism of the last few centuries. Yet now the notion of

self-awareness is under attack from the Artificial Intelligence people

("self-awareness and other delusions"). Intelligence Amplification

undercuts our concept of ego from another direction. The post-

Singularity world will involve extremely high-bandwidth networking. A

central feature of strongly superhuman entities will likely be their ability to

communicate at variable bandwidths, including ones far higher than

speech or written messages. What happens when pieces of ego can be

copied and merged, when the size of a selfawareness can grow or shrink

to fit the nature of the problems under consideration? These are essential

features of strong superhumanity and the Singularity. Thinking about

them, one begins to feel how essentially strange and different the Post-

Human era will be -- _no matter how cleverly and benignly it is brought to

be_. 

From one angle, the vision fits many of our happiest dreams: a time

unending, where we can truly know one another and understand the

deepest mysteries. From another angle, it's a lot like the worst- case

scenario I imagined earlier in this paper. 

Which is the valid viewpoint? In fact, I think the new era is simply too

different to fit into the classical frame of good and evil. That frame is

based on the idea of isolated, immutable minds connected by tenuous,

low-bandwith links. But the post-Singularity world _does_ fit with the

larger tradition of change and cooperation that started long ago

(perhaps even before the rise of biological life). I think there _are_

notions of ethics that would apply in such an era. Research into IA and

high-bandwidth communications should improve this understanding. I

see just the glimmerings of this now [32]. There is Good's Meta-Golden

Rule; perhaps there are rules for distinguishing self from others on the

basis of bandwidth of connection. And while mind and self will be vastly

more labile than in the past, much of what we value (knowledge, memory,

thought) need never be lost. I think Freeman Dyson has it right when he

says [9]: "God is what mind becomes when it has passed beyond the

scale of our comprehension." 

[I wish to thank John Carroll of San Diego State University and Howard

Davidson of Sun Microsystems for discussing the draft version of this

paper with me.] 

Annotated Sources [and an occasional plea for bibliographical help]
[1] Alfve'n, Hannes, writing as Olof Johanneson, _The End of Man?_,

Award Books, 1969 earlier published as "The Tale of the Big Computer",

Coward-McCann, translated from a book copyright 1966 Albert Bonniers

Forlag AB with English translation copyright 1966 by Victor Gollanz, Ltd. 

[2] Anderson, Poul, "Kings Who Die", _If_, March 1962, p8-36. Reprinted in

_Seven Conquests_, Poul Anderson, MacMillan Co., 1969. 

[3] Asimov, Isaac, "Runaround", _Astounding Science Fiction_, March 1942,

p94. Reprinted in _Robot Visions_, Isaac Asimov, ROC, 1990. Asimov

describes the development of his robotics stories in this book. 

[4] Barrow, John D. and Frank J. Tipler, _The Anthropic Cosmological

Principle_, Oxford University Press, 1986. 

[5] Bear, Greg, "Blood Music", _Analog Science Fiction-Science Fact_, June,

1983. Expanded into the novel _Blood Music_, Morrow, 1985. 

[6] Cairns-Smith, A. G., _Seven Clues to the Origin of Life_, Cambridge

University Press, 1985. 

[7] Conrad, Michael _et al._, "Towards an Artificial Brain", _BioSystems_,

vol 23, pp175-218, 1989. 

[8] Drexler, K. Eric, _Engines of Creation_, Anchor Press/Doubleday, 1986. 

[9] Dyson, Freeman, _Infinite in All Directions_, Harper && Row, 1988. 

[10] Dyson, Freeman, "Physics and Biology in an Open Universe", _Review

of Modern Physics_, vol 51, pp447-460, 1979. 

[11] Good, I. J., "Speculations Concerning the First Ultraintelligent

Machine", in _Advances in Computers_, vol 6, Franz L. Alt and Morris

Rubinoff, eds, pp31-88, 1965, Academic Press. 

[12] Good, I. J., [Help! I can't find the source of Good's Meta-Golden

Rule, though I have the clear recollection of hearing about it sometime in

the 1960s. Through the help of the net, I have found pointers to a

number of related items. G. Harry Stine and Andrew Haley have written

about metalaw as it might relate to extraterrestrials: G. Harry Stine, "How

to Get along with Extraterrestrials ... or Your Neighbor", _Analog Science

Fact- Science Fiction_, February, 1980, p39-47.] [13] Herbert, Frank,

_Dune_, Berkley Books, 1985. However, this novel was serialized in

_Analog Science Fiction-Science Fact_ in the 1960s. 

[14] Kovacs, G. T. A. _et al._, "Regeneration Microelectrode Array for

Peripheral Nerve Recording and Stimulation", _IEEE Transactions on

Biomedical Engineering_, v 39, n 9, pp 893-902. 

[15] Margulis, Lynn and Dorion Sagan, _Microcosmos, Four Billion Years

of Evolution from Our Microbial Ancestors_, Summit Books, 1986. 

[16] Minsky, Marvin, _Society of Mind_, Simon and Schuster, 1985. 

[17] Moravec, Hans, _Mind Children_, Harvard University Press, 1988. 

[18] Niven, Larry, "The Ethics of Madness", _If_, April 1967, pp82-108.

Reprinted in _Neutron Star_, Larry Niven, Ballantine Books, 1968. 

[19] Penrose, Roger, _The Emperor's New Mind_, Oxford University Press,

1989. 

[20] Platt, Charles, Private Communication. 

[21] Rasmussen, S. _et al._, "Computational Connectionism within Neurons:

a Model of Cytoskeletal Automata Subserving Neural Networks", in

_Emergent Computation_, Stephanie Forrest, ed., pp428-449, MIT Press,

1991. 

[22] Searle, John R., "Minds, Brains, and Programs", in _The Behavioral

and Brain Sciences_, vol 3, Cambridge University Press, 1980. The essay is

reprinted in _The Mind's I_, edited by Douglas R. Hofstadter and Daniel C.

Dennett, Basic Books, 1981 (my source for this reference). This reprinting

contains an excellent critique of the Searle essay. 

[23] Sims, Karl, "Interactive Evolution of Dynamical Systems", Thinking

Machines Corporation, Technical Report Series (published in _Toward a

Practice of Autonomous Systems: Proceedings of the First European

Conference on Artificial Life_, Paris, MIT Press, December 1991. 

[24] Stapledon, Olaf, _The Starmaker_, Berkley Books, 1961 (but from the

date on forward, probably written before 1937). 

[25] Stent, Gunther S., _The Coming of the Golden Age: A View of the End

of Progress_, The Natural History Press, 1969. 

[26] Swanwick Michael, _Vacuum Flowers_, serialized in _Isaac Asimov's

Science Fiction Magazine_, December(?) 1986 - February 1987.

Republished by Ace Books, 1988. 

[27] Thearling, Kurt, "How We Will Build a Machine that Thinks", a

workshop at Thinking Machines Corporation, August 24-26, 1992.

Personal Communication. 

[28] Ulam, S., Tribute to John von Neumann, _Bulletin of the American

Mathematical Society_, vol 64, nr 3, part 2, May 1958, pp1-49. 

[29] Vinge, Vernor, "Bookworm, Run!", _Analog_, March 1966, pp8-40.

Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen

Books, 1987. 

[30] Vinge, Vernor, "True Names", _Binary Star Number 5_, Dell, 1981.

Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen

Books, 1987. 

[31] Vinge, Vernor, First Word, _Omni_, January 1983, p10. 

[32] Vinge, Vernor, To Appear [ :-) ].

===========我是分隔线==================

简单说来，就是说在预言互联网技术上有很好记录的弗诺文奇在93年预言了30年

后人类科技发展到一个千人无法理解的高度，进入超人世纪。

现在只有不到二十年啦，而严肃认真的科技项目没有一个有这十分之一的野心。

也许我们真的需要一个奇迹，像电脑技术一样从实验室的小玩意突然发展统治全

球。任重而道不远啊。

“可现在，你比从前任何时候更需要奇迹，而且要多，要快。可你手中剩下的只

有迷信。”
天渊，第四十五章