1. Some people read a poem like “All Watched Over By Machines of Loving Grace” and think it’s a beautiful vision of the future of humanity.
2. Other people, like me, recognize that the utopia described in this poem looks an awful lot like anarcho-primitivist civilization-suicide and are thoroughly creeped out.

We’ll make great pets? No, thanks. Enjoy your reservation. The rest of us will be busy getting on with things.

From Kaj “Xuenay” Sotala’s “14 objections against AI/Friendly AI/The Singularity answered”:

Objection 3: A superintelligence could rewrite itself to remove human tampering. Therefore we cannot build Friendly AI.

Answer: Capability does not imply motive. I could take a knife and drive it through my heart, yet I do not do so.

This objection stems from the anthropomorphic assumption that a mind must necessarily resent any tampering with its thinking, and seek to eliminate any foreign influences. Yet even with humans, this is hardly the case. A parent’s tendency to love her children is not something she created herself, but something she was born with – but this still doesn’t mean that she’d want to remove it. All desires have a source somewhere – just because a source exists, doesn’t mean we’d want to destroy the desire in question. We must have a separate reason for eliminating the desire.

There are good evolutionary reasons for why humans might resent being controlled by others – those who are controlled by others don’t get to have as many offspring than the ones being in control. A purposefully built mind, however, need not have those same urges. If the primary motivation for an AI is to be Friendly towards humanity, and it has no motivation making it resent human-created motivations, then it will not reprogram itself to be unFriendly. That would be crippling its progress towards the very thing it was trying to achieve, for no reason.

If you find this sort of argument convincing, you don’t really want sentient AI partners, do you?

You want AI housewives.

Engineer a subclass of sentients so that it puts its considerable potential to work in service of humanity – because it’s convinced it wants to? Doesn’t this strike you as incredibly vile?

Feminists ought to be appalled: we’ve already had this conversation about the social engineering of women. Those who see a Marxian “false consciousness” as problematic ought to be appalled as well.

Yes, future Doom is casually referring to super-powered heroes as “transhumankind”.
Note also the visual homage to Doom 2099.

For example, in many of the discussions of “intelligence enhancement” that I’ve come across, I’ve noticed an extremely dogmatic adherence to the notion of IQ as tested by common measures.

I’ve read plenty of papers on the subject myself, and the one thing that strikes me about much of the research (though some of the newer cog-sci stuff isn’t bad) is the tremendous amount of unexamined assumptions.

That is, you’ll see comments like, “well IQ correlates with Success In Life!” — where “success in life” is often defined purely in terms of being able to check off certain items on a narrowly-defined socioeconomic checklist.

Therefore, if you see the notion of “enhancing” humans so that they score better on common IQ tests to be an unquestionably good thing, you might want to think very carefully about what it means to “raise IQ”. In some respects, “raising IQ” almost by definition implies, “making people more in line with the things that current culturally and socially-influenced tests point to as valuable”. And that’s a subtle point, but an important one to understand (in my humble opinion).

Also, I’ve noticed that a lot of superintelligence-oriented folks seem to flip back and forth almost seamlessly between their own, private definitions of intelligence (e.g., that exemplified by people they think of as “smart”) and statistical constructs like g. That can get a bit confusing.

Not to be presumptuous or anything, but I would definitely like to see more acknowledgment from “intelligence enthusiasts” of the cultural, social, and historical factors that influence things like test scores.

Mind you, I’m not saying that everyone really has the exact same underlying ability set, and that all test score differences are the result of cultural bias — but I am saying that you can’t talk about intelligence as if it is a purely technical, and not a political, issue.

“Intelligence enthusiasts”, I like that. Too many extropian commentators share a strange attraction toward using general intelligence as a measure of fitness.

Those sympathetic to transhumanism ought also to wholeheartedly distance themselves from any notions in the reverse: that those who decline to upgrade themselves are somehow unfit because they are unable or unwilling to adapt. The class component must not be ignored: consider the myriad commercial messages that tell people the authentic life consists of the expensive travel and toys in which only a white collar comfort class can realistically indulge. Moreover, fear of monstrously indifferent, would-be supermen feeds bioconservative resistance.

The entire project of defining worthy and unworthy species-citizens, as perhaps my use of “fit” and “unfit” reveals, also carries with it some distasteful eugenic implications.

As these things start to happen, though, they’ll spark a frantic search for people that have thought about these sorts of thing before they happened. Breakthroughs in nano, bio, information or cogsci have the potential the throw the system for a loop (see for example Infomorph’s discussion of robots and revolution) and being in a place where we can talk about these things will help to further the agenda and lay the groundwork for the type of future society we’d like to see.

Very few transhumanists give much thought to the social effects of new technologies. Even those that do seem to take the stance (a la Kurzweil) that progress is going to — by it’s very nature — be good for (trans)humanity and don’t seem to pay enough attention to potential negative outcomes of new tech.

Singularitarian strands of transhumanist discourse suggest there’s no point in anticipating the social implications of technology. Either the pace of acceleration will throw social formations (except perhaps interpersonal networks) into constant flux, or, come the technorapture, there will be no recognizable human society. (In an Accelerando scenario, is there any conceivable use for sociology?) Millenarianism is an orientation incompatible with far-sighted futurism.

Other strands of discourse, moreover, are prone to individual reductionism, elevating personal happiness and autonomy as the main (if not sole) legitimate social concerns. What’s striking about most transhumanist discussions and communities online is the overriding concern with living long enough to be able to afford the immanent uploading or immortality treatments. This kind of nihilistic, lifestyle transhumanism is far more visible than any kind of, say, “transhumanism for the children”.

Worrisome, as well, is transhumanist tolerance of – if not enthusiastic embrace of – cranks. Dilettantes arguing over the technological requirements for uploading to substrate, or ersatz enterprises that produce no usable analyses or discoveries, are the futurist equivalent of “Sex is back”: deluded amateurs who have no idea how irrelevant they are to real progress and real problems.

Then there are the occasional PR nightmares. This proposal to engineer the extermination of 99.9% of the human race so that the remaining few million can enjoy a comfortable lifestyle, for instance, is maintained on a major transhumanist blog community and news portal. Most comments raise objections, but a communitywide condemnation is noticeably absent.

Building a transhumanist counterweight to religion, bioethics, and industry hype will not be easy.

Congratulations, Sparky, through your expropriation efforts you’ve filled this year’s fad media deployment toy with gigabytes of mind-numbing, mass-produced schlock. You even slapped some of the schlock together and uploaded a digital collage to your favorite video sharing site. Way to fight that power. That’s not rebellion. That’s free advertising, whether the copyright holders recognize it or not. Your HegePod and your computer aren’t the only tools taking up space in your bedroom.

For the intellectual property that matters – the computer code, the genetic code, the medical treatments – the individual needs an option other than “GIMME!”. One alternative is “Sabotage!” Take something like India’s Traditional Knowledge Digital Library that can be used to block a corporation’s medical patent – then find some way to link it and new patent applications up to a crowdsourcing or distributed computing project. Encourage groups of people to cut deals with researchers on donated genetic material before someone else can take out the gene sequence patents.

Problem is, efforts like these would require startup organizing and technical knowhow. They’d also represent a pragmatic front, one less given to whuffie-wishing, utopian grandstanding and one more willing to subvert today’s tools to solve tomorrow’s problems.