In his discussion of a theoretical artificial intelligence “arms race”, Nick Bostrom, Director of the Future of Humanity Institute at Oxford, presents a model of future AI research in which development teams compete to create the first General AI. Under the assumption that the first AI will be very powerful and transformative (a notably arguable one, as per the soft vs. hard takeoff debate), each team is highly incentivised to finish first. Bostrom argues that the level of safety precautions each development team will undertake arises as a reflection of broader policy parameters, specifically those relating to the allowed level of market concentration (i.e. permitted consolidation of research teams), and information accessibility (i.e. degrees of intellectual property protection & algorithm secrecy).
In his work, Bostrom does not reach one specific conclusion regarding AI safety levels, but instead defines a set of Nash equilibria given various numbers of development teams + levels of information accessibility. Specifically, he notes that having additional development teams (and therefore reduced market concentration) may increase the likelihood of an AI disaster, especially if risk-taking is more important than skill in developing the AI. Increased information accessibility also increases risk. The more teams know of each others’ capabilities and methodologies, the greater the velocity of, and enmity in, development; a greater equilibrium danger level follows accordingly.
Rationality's Volition 
Hello blog author,
I found Bostrom’s model interesting. Specifically, his impressive construction of a multi-variable model for such a broad array of concepts. While, I am a novice in the modeling field, I am still blown away (and possibly skeptical) of his ability to mold together multiple variables which seem un-mixable.
Perhaps that is what provokes my question, How would this model come into any potential use with seemingly incalculable variables? I submit, this is a question I can apply to many models. However, my experience with biophysical models has always satisfied this concern. So I ask you, author, do you see a conceivable future in which we are applying statistics to this equation, hoping to yield solutions? Or does the model have real application elsewhere, beyond the simplified approach I believe I’ve presented?
Science,
Scott Borrus
Science begins with the willingness to be puzzled by things that seem obvious. (Noam Chompsky)
LikeLike
Hello! Apologies for my much delayed reply. Happy to be back to blogging!
In this paper, Bostrom’s model is intended to provide guidance towards AI governance, specifically regarding the number of teams allowed to build AGI and the degree of information sharing between them. He mentions a Singleton as a potential solution. I hope Ban Ki-moon likes technology 🙂
LikeLike
Crazy to think that greed and enmity in pursuit of AGI–a landmark representing the obsoletion of human error–could end up being the most catastrophic human errors in history.
Ehwuhdufuuuuuuh?
LikeLike
Hello! Apologies for my long-delayed reply. It has been an extraordinary few months, but happy to be back to blogging.
Note that AGI (or as follows, ASI) would only absolve us of human error if its goal structure perfectly mirrored our own. If its goal structure were say, opposite ours, it could almost certainly put our level of ‘human error’ to shame 🙂
LikeLike