Singularitarianism attracts more than its share of crankdom, hack philosophy, and clueless amateurism. It makes transhumanism look ridiculous. The phantom enterprise of speculating how best to control artificial intelligence to make it “friendly”, though, has severe ethical problems besides.

From Kaj “Xuenay” Sotala’s “14 objections against AI/Friendly AI/The Singularity answered”:

Objection 3: A superintelligence could rewrite itself to remove human tampering. Therefore we cannot build Friendly AI.

Answer: Capability does not imply motive. I could take a knife and drive it through my heart, yet I do not do so.

This objection stems from the anthropomorphic assumption that a mind must necessarily resent any tampering with its thinking, and seek to eliminate any foreign influences. Yet even with humans, this is hardly the case. A parent’s tendency to love her children is not something she created herself, but something she was born with – but this still doesn’t mean that she’d want to remove it. All desires have a source somewhere – just because a source exists, doesn’t mean we’d want to destroy the desire in question. We must have a separate reason for eliminating the desire.

There are good evolutionary reasons for why humans might resent being controlled by others – those who are controlled by others don’t get to have as many offspring than the ones being in control. A purposefully built mind, however, need not have those same urges. If the primary motivation for an AI is to be Friendly towards humanity, and it has no motivation making it resent human-created motivations, then it will not reprogram itself to be unFriendly. That would be crippling its progress towards the very thing it was trying to achieve, for no reason.

If you find this sort of argument convincing, you don’t really want sentient AI partners, do you?

You want AI housewives.

Engineer a subclass of sentients so that it puts its considerable potential to work in service of humanity – because it’s convinced it wants to? Doesn’t this strike you as incredibly vile?

Feminists ought to be appalled: we’ve already had this conversation about the social engineering of women. Those who see a Marxian “false consciousness” as problematic ought to be appalled as well.