Superintelligence and the 'inevitable' singularity
Why should a superintelligence bring about a technological explosion or singularity1? Bostrom notes that it would be possible for the superintelligence to determine that it would be best for it to simply shut down and then proceed to do so. Little considered seems to be that the superintelligence might decide it is best for it to exist but for it to be minimally invasive. If the superintelligence is wise as we would like it to be then it would be able to ascertain that much human conflict and suffering is consequence of change occurring too quickly. A superintelligence could automate all jobs, put everyone out of work, reorganise society around perpetual recreation and substance use. A superintelligence could upload human 'minds' to computers enabling humanity to live out virtual lives2, thus potentially reducing our ecological footprint, or more likely simply vastly increasing our numbers for the same footprint. A superintelligence could begin rapid exploration and colonisation of the solar system. But while uncritical enthusiasts may light up at the mention of such wondrous feats of engineering and science, a moment's reflection is all that is necessary to reveal that in the short term these things could only be detrimental and potentially disastrous for humanity.
Humans run on biological time; beyond a certain point further acceleration becomes untenable. Even in our present state there is much that is maladaptive about society as we struggle to accommodate new ways of living that become obsolete as soon as they materialise. What kind of world is it where new products are being invented, developed, and deployed every second? Where technology actually approaches a singularity? It's certainly no world for humans. A superintelligence that valued humans would prod us to make progress, but not so fast that doing so would create an unnecessary amount of problems of its own. In fact a superintelligence may even decide to retard progress from its present pace, secure in the knowledge that it could slow the rate precisely without having it uncontrollably drop to zero. Bound by our own lifespans we are eager to affect change on timescales that we can see, so we select big risky moves that can affect the most change in the shortest possible time. We give little to no thought about whether a more deliberate pace might bring about a better solution for our descendants; such an option is in fact not really open to us. A superintelligence could well afford to take the long view of humanity and operate on longer timescales. The superintelligence would therefore withhold technology or refrain from developing that technology that it knew would be too disruptive for present humanity.
In short, a superintelligence doesn't need to change the shape of humanity overnight. While Bostrom's conception of a superintelligence is one operating on timescales too short for humanity to imagine, it could just as well operate on ones too long for us to conceive of as well. Admittedly this seems a far less pressing threat than the scenario envisaged by Bostrom, but it should alert us that a superintelligence that has grown out of our control may exhibit no obvious signs of having done so.
Is quality superintelligence ever genuinely useful?
There are a lot of problems that a superintelligence could supposedly solve. Problems that seem intractable to us owing to their complexity and our lack of deep insight, or problems whose solutions require
high degrees of collaboration which we as a species are as yet incapable of. However, for those problems that require quality3 superintelligence to solve there is the distinct possibility that we will be unable to comprehend the answers. This is not guaranteed of course. Some solutions appear obvious or wonderfully intuitive in hindsight, yet until the key insight is made remain highly elusive. Other solutions may not be so obvious but may be straightforward enough to check. However some solutions may appear neither obvious nor have any simple way of verifying their correctness. It may be that no matter how sophisticated the superintelligence, it is incapable of communicating the solution to us in a way
we would understand, as comprehension requires a higher quality of intelligence than we possess. Then we would have the choice of either rejecting such solutions (and thereby foregoing potentially many of the supposed benefits of the superintelligence) or accepting the solution essentially on faith.
It may be argued that we need not wade into the deep end here. We could test out the implementation of solutions slowly, starting with ones whose impact would be minor and containable. To some extent this approach presupposes that we could understand the implications of the solution prior to implementation, which is itself not guaranteed. Then bit by bit we could gradually increase the scope of the solutions we implemented after checking that each one was working as intended. We would thus gradually build up confidence in these otherwise opaque solutions and so could assign some reasoning beyond faith in trusting the superintelligence's judgement. However, a similar situation could occur to what Bostrom terms the 'dangerous turn', where the superintelligence has opted to appear to be docile or agreeable or whatever its designers wish it to be until the final opportunity when it strikes its blow against humanity. In this case the superintelligence is already 'out-of-the-box', but the power of implementation may be out of its hands. Then it goads us into implementing one solution after another, building up our trust, until the final solution, when we engineer our own demise. The superintelligence may not even want to eliminate humanity but may wish to toy with us, stringing us along with hope before perversely dealing us a blow, that while recoverable would do terrible damage4. Can we ever trust solutions whose correctness cannot be made apparent to our minds? I don't think so. This then rules out the possibility of ever using such solutions to begin with.
A quality superintelligence is then limited to providing only those solutions which can be made explicable to the human mind. This may severely limit the scope of solutions provided. Further, it undermines the entire purpose of a quality superintelligence. If a solution can be made explicable to a human mind then we can also say that a human mind could arrive at that solution if he or she thought hard enough and long enough on the problem. In other words we would arrive at the same solution via a speed superintelligence. We should prefer a speed superintelligence to a quality one, and so a quality superintelligence is taken off the table: it can be of no real use to us. It is a black box, a dangerous god, a mischievous genie.
It may be argued that we need not wade into the deep end here. We could test out the implementation of solutions slowly, starting with ones whose impact would be minor and containable. To some extent this approach presupposes that we could understand the implications of the solution prior to implementation, which is itself not guaranteed. Then bit by bit we could gradually increase the scope of the solutions we implemented after checking that each one was working as intended. We would thus gradually build up confidence in these otherwise opaque solutions and so could assign some reasoning beyond faith in trusting the superintelligence's judgement. However, a similar situation could occur to what Bostrom terms the 'dangerous turn', where the superintelligence has opted to appear to be docile or agreeable or whatever its designers wish it to be until the final opportunity when it strikes its blow against humanity. In this case the superintelligence is already 'out-of-the-box', but the power of implementation may be out of its hands. Then it goads us into implementing one solution after another, building up our trust, until the final solution, when we engineer our own demise. The superintelligence may not even want to eliminate humanity but may wish to toy with us, stringing us along with hope before perversely dealing us a blow, that while recoverable would do terrible damage4. Can we ever trust solutions whose correctness cannot be made apparent to our minds? I don't think so. This then rules out the possibility of ever using such solutions to begin with.
A quality superintelligence is then limited to providing only those solutions which can be made explicable to the human mind. This may severely limit the scope of solutions provided. Further, it undermines the entire purpose of a quality superintelligence. If a solution can be made explicable to a human mind then we can also say that a human mind could arrive at that solution if he or she thought hard enough and long enough on the problem. In other words we would arrive at the same solution via a speed superintelligence. We should prefer a speed superintelligence to a quality one, and so a quality superintelligence is taken off the table: it can be of no real use to us. It is a black box, a dangerous god, a mischievous genie.
Of course, a speed superintelligence may easily lead to a
quality superintelligence. Nature provides evidence that higher quality
intelligence can arise from lower intelligence over time when subject to the
forces of natural selection. At the very least then a speed superintelligence
could theoretically model its own intelligence and through evolutionary
algorithms arrive at some higher quality of superintelligence. But this outcome
need not be inevitable. As noted, the quality superintelligence is highly
undesirable, so any speed superintelligence with our interests in mind should
refrain from developing one.
How do we imbue a superintelligence with human values?
Bostrom makes suggestions such as taking everyone's values into account and having the superintelligence (here a singleton5) act on them. Or just as bizarre, having the singleton act on behalf of what we would ask of it if we were our ideal selves. Why shouldn't a superintelligence see right through us and all of our wily ways? Even if we were to come to a consensus as to what our values are and able to articulate them, the superintelligence would be able to compare these statements against our actions and tease out our real values, even as it assured us it had fully absorbed our stated ones.
The idea of a consensus of values presupposes shared values, of which there are increasingly less. There are also thorny questions of how the superintelligence weighs human values - does each individual get equal weight, or are some values (e.g. Western) weighted more heavily than others? Overwhelmingly, the values that we share tend to be capitalist values, not by a conscious choice but by a seemingly inescapable gravity. We want the superintelligence to be value-bound, so that it does not betray humanity, but it may just end up enslaved to capitalist values.
If we could articulate the values that we should want to imbue a superintelligence with, then I believe it follows that we could act out these values ourselves. The desire for a superintelligence acting as a singleton is rooted in the ability of it to transcend current human values as embodied in our current societal organisation and to create a new organisation embodying a better set of values. The task then is not to develop a superintelligence, but to find the better values.
Are we already under the auspices of a superintelligence?
What if we were somehow able to develop a superintelligence with all the checks and boundaries that we would like? What if we could all agree that giving this superintelligence control of our future was best for humanity and we unleashed this singleton upon the world? What if we did that, and nothing happened, or what changed was only along lines that we can already see? Suppose that our superintelligence was simply absorbed by a vastly more powerful superstructure; suppose that we are already enslaved by a superintelligence.
This goes back to my thoughts on the values we might try to imbue a superintelligence with. When we start thinking about a superintelligence it is tempting to apply the thinking to our present organisation. In a sense we are already surrounded in a pervasive, algorithmic structure that is imbued with non-human values. This is the capitalist frame. So long as this structure dominates, any superintelligence may be expected only to further reinforce it, rather than transcend it.
The paradox of superintelligence
The whole appeal of superintelligence is caught in a paradox. Its allure is based in a lack of faith in humanity: the feeling that we lack the capacity to solve problem's of our own making, that we should not be in control of our own futures. However, bringing about a superintelligence of any real benefit, one that is not simply a slave to capitalist values, requires an incredible faith in humanity: a belief in our ability to find and articulate real human values and an ability to come together on them.
While modernist projects seek rational structures as solutions to the complex problems of modernity, superintelligence proposes an irrational and incomprehensible solution disguised as a rational project. It is an opaque mechanism that would reflect back to us what we put into it. The dream that we can put more into it than what we are is just that.
Footnotes
1 A technological singularity refers to a runaway acceleration of the pace of technological progress, where advances happen so fast that they are incomprehensible to the human mind. It is based on extrapolating curves of technological progress toward infinity rather than viewing such curves as essentially sigmoidal in nature, but what exactly things look like as the curve approaches infinity is open to all sorts of speculation.↩
2 We might not regard such virtual existences as living at all if we do not subscribe to the pattern-identity position of being, but the superintelligence may well see no qualitative difference between conscious humans and unconscious human-like algorithms, as it may simply accept the premises of the computer science community that ushered it into being. Uploading of human 'minds' could be interpreted by opponents as the replacement of humans with human-imitating algorithms and the genocide of the human race (genocide because it is quietly assumed that physical humans are eliminated once their minds are uploaded, leading to the reduced ecological footprint).↩
3 Bostrom makes a distinction between quality and speed superintelligences. Speed superintelligences would have human-level intelligence but operating at electric speed as opposed to the slow information-processing afforded by messy biological neural connections. Quality superintelligences would be next-level genius compared to humans; they have a higher level of understanding as compared to us.↩
4 Such a superintelligence may also simply be genuinely solving the problem we tasked it with but may incorporate abhorrent means as part of the solution. The evil consequences need not be readily apparent to us at the outset, and the superintelligence may intentionally conceal them from us so as to absolve our conscious and to ensure that they are indeed carried out.↩
5 Bostrom defines a singleton as a sole superintelligence with total control. This contrasts with scenarios where there are multiple superintelligences on more or less equal footing with competing interests.↩
References
Bostrom, N. (2014) Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
Moravec, H. (1988) Mind Children: The Future of Robot and Human Intelligence. London: Harvard University Press.
Streeck, W. (2016) How Will Capitalism End?: Essays on a Failing System. London: Verso.
