Shadow costs potentially outweigh the benefits of AI development, thus making it a net negative. Governance on a nationwide level is insufficient and can be modelled by the tragedy of the commons. Hence to reduce the negative effects some AI developments should be slowed down and increasing effect-limiting work accelerated.
Russia, India and the USA are blocking talks for an international treaty outlawing lethal autonomous weapon systems (LAWS), as reported by Reuters. Given that these countries are producers of those weapon systems it is not that surprising. [1]
The EU also plans an exception for military uses in the new AIA regulation that regulates the use of AI.[2]
Some are pushing against regulation, like the US Security Commission on AI.
What is in for those countries? To increase accuracy and response time, and reduce the number of casualties more and more weapon systems are being remotely controlled and the operation is extended by cybernetic control algorithms. Advanced cybernetic control enables the new weapon category called “loitering munition” in between unmanned aerial vehicles (UAVs) and cruise missiles.
Fully automated weapon systems allow the industrialised destruction of human lives in suppression and policing or even genocides. Some NGOs[3] are pushing for an international treaty to draw a red line, which is not to cross: Automatic decision to pull the trigger. The idea for an international treaty is to add a protocol to the Convention on Conventional Weapons (CCW).
Would the regulation eliminate the danger of this new technology?
Does it make a big difference when the decision support system presents all information, target-locked and the operator only needs to manually confirm the kill with a press of a button? The human in the loop forms an additional requirement not desired by the military. This single interaction makes drones require a constant radio connection, making stealth systems more vulnerable to detection and a make them a target for anti-radiation missiles. For guided missiles, the additional radio equipment means also additional costs during production and operation. Evaluating the signal by a human takes time, which can be used by the opponent’s system to evade. In the arms race of attack and defence, this requirement draws a line upheld by political or legal power.
One example where more autonomy is helping in war is “fire-and-forget” missiles which have been target locked minutes ago using an infrared signal. This is currently a successful and common tactic in war, as seen again in the Russian-Ukrainian war. To avoid detection by Radar the missiles use sea-skimming, a technique where the missile flies just above the water. Being undetectable is key to making the attack successful. Hence having autonomous missiles can enable this attack, while a missile depending on line-of-sight must get too close and are in danger of being intercepted or the launcher is shot before the missile is launched.
Even with regulation, it is too easy to find ways around the laws to have a real effect and the technological systems will push hard against that legal line. Violations are far easier to cover because it can hardly be proven in the hardware that a missile or turret was using computer vision and other algorithms and not remotely controlled. So in this case a treaty’s final effect is questionable.
The Difference in Growth Rates: an Argument for Differential Technological Development
The growth in humanity’s wisdom is outperformed by the growth in power. Humanity’s wisdom is contained by the institutions and the knowledge passed along generations. In particular, I argue that each generation can only develop a limited amount of new views, hence the rate of progress in civilisation development has an upper bound. Older humans stick to the views they developed in their younger years[4] when their neural pathways were forming at a higher rate. Furthermore, all new information we obtain is set in relation to our knowledge base. Certain impactful experiences like leaving the place of your uprising for the first time are made only once in younger years and the experience itself is influenced by the conditions at that time.
In 1945, after the experiences the world made in both world wars, the United Nations Organisation (UNO) was founded which should reduce the risks of wars. It is a place to foster coordination but it has no power to control all nations as each nation has autonomy. We still have not found a way to deal with problems like self-inflicted existential risks e.g. a global thermonuclear war or climate change. This means that humanity currently wields more power than our maturity. Toby Ord compares humanity with the education of a child.[5] A child is bound by rules to ensure a safe uprising. We must lay down rules to govern ourselves as laid out before in the example of LAWS. As we have seen, the self-governance imposed by nations quickly ends up again in the tragedy of the commons. Everyone else has a nice, wealthy life, while your altruistic country who needs to cut short on wealth by introducing a Pigouvian tax to compensate others from the negative externalities your country caused. International constructs help overcome the limitations of the national idea but have not fully overcome it. The alternative to improving self-governance is reducing the risks we self-impose.
From side effects to net benefit calculation
I once spoke with someone doing a PhD in the area of computer vision where the goal was to identify and separate people on a video stream. I curiously asked about any concerns about how this technology can be used to surveil people. The person answered that he never really thought about it. This indifference is all too common with students under engineers. The most famous example of this is nuclear technology which resulted in the creation of a nuclear arms race. Technology is developed with a problem to solve in mind. Once the technology is applied it solves the problem but the technology is not limited to this application. Side effects are often overlooked.
Let us do a brief ontology on effects. What is an effect? An effect is the ontological category that describes the relationship between two phenomena: It describes the significance of a transformation of some phenomenon from one state to another in the unit of a sign. The state change caused by the appliance can become the significance of another effect. The joined set of the three objects initial state, significance and resulting state together then becomes itself an effect. The side-effect can lead to further effects and so forth. We call these effects first-order or second-order effects etc. These nodes then form a Bayesian network.
The goal is to obtain some ethics so our first-order effect is an action we want to pick.
In economic terms, the calculation of the net benefit or expected utility must also include the shadow costs. The shadow costs are the non-obvious or non-intentional costs associated with the action. The costs are the value we assign to each effect. In our model, there are many connections between each node. Since we do not have total information we can do a reasonable simplification.[6] The net benefit of advancing technology x can be written as the sum of the product of the probability that a problem b is solved by the technology and the utility of the solved problem plus the respective sum of the weighted shadow cost SC[7]:
Deriving DTD
The shadow costs aggregate side effects which include the increase of existential risks. When choosing probabilities ex ante you would most likely end up with much lower probability scores on shadow costs 
but the utility of the SC in case of increasing existential risk are enormous 
. Depending on the outcome of the equation it should matter whether this technology should be worked on or not.
That means for some technologies we should currently just not work on them. The principle of differential technological development, a concept attributed to Nick Bostrom, suggests working on technology which reduces risks. E.g. this could be drone defence technology instead of working on drones.
Time-dependant equation
The whole world is advancing and changing without us taking any action. Without any technological advancements, the world is still changing. The context is changing over time because it is a shared environment of many actors. We can increase the fidelity of the Bayesian network model with a lot of other nodes to include this effect. The simplification from the Bayesian network model suffers from this simplification: It lacks modelling time dependant utilities. The site of all effects, intentional or side effects, would monotonically decrease over time to zero.[8]
We have not figured out how to solve the climate crisis. I would argue that civilisation facing the climate crisis equals humanity being in the phase of having fossil fuel based industrial society as the common form. The amount of energy civilization consumed over time has grown continuously. Not long ago the main energy source has become fossil fuels and started the chapter of industrialization in our history. Only moving beyond that to become an energetically sustainable high-energy civilisation would reduce the risks associated with the climate crisis. This means the faster we move out of this state the better for our risk. Any action taken now has a bigger impact than doing it later.
Toby Ord classifies risks into two categories as state or transition risks. Where transition risks are risks that arise during a transition period to a new technology and state risks are existential risks that exist while remaining in a certain state. I don’t think it makes much sense to differentiate since all state risks but the final state (singularity or collapse) are transitions risks by definition: There will be a certain state in the future where the risk will be eliminated or reduced. Up until then, the other existential risks remain. As long as this state is not achieved we are in transition.
Applying the net benefit analysis to AI
This presented principle of a net benefit analysis of technology is not limited to UAVs or AI and can be performed for all technology. However, I want to take a look at the area I feel more confident to rate potential shadow costs.
Will improving AI be a net benefit for humanity or not and can we now answer the question of which technologies to work on? By using the simplified model we can assign the variables of the probability and the effect. The intended first-order effects get a positive value but then we must subtract the side effects. AI is a dual-use technology. It can be used to solve various problems humanity faces, while also offering destructive faculties like more destructive weapon systems. I covered already the potential for personalised AI assistance as a positive example. One risk of every AI capability improvement is increasing the probability of getting unaligned general AI. In an article by 80k hours[9], different views on this work have been collected. Some of them share the conclusion outlined here. The critical path in a dependency graph is the shortest path from A to B, where every change would affect the total length of the path. Any work on something on the critical path to catastrophe is contributing. I am undecided if working on AI which does not increase capabilities should be also rated negatively because it might be on the critical path. Thousands of companies working on AI systems create the demand in a market to do capabilities research done by others, which might be the limiting factor.
When considering the sentiment that side effects are contributing potentially bigger harm than the first-order effects, we must conclude that most high-tech technological development might be a moral net negative. The sheer greatness of the potential of human civilisations and the imbalance in technological progress over cultural advancements cancels all positive effects. This is also true when considering big effects like creating man-made existential risks or the intended effects of transitioning out of an existential state.
You might not be convinced by the assessment of the maximum effects overshadowing the calculation. Depending on this you might still gain something from a more fine-grained analysis. A full net benefit analysis will follow in another post.
The risks outweigh rushing technological development as these technologies could be developed when needed. For the foreseeable future, these are not blocking humanity’s progress but they would advance the solution to existing problems. Right now it is more important to make sure that we don’t make mistakes which destroy humanity’s potential.
One should not fall into the trap of just looking at first-order effects in the short term. The time dependency is increasing the urgency. However, this is also true for differential technology development, which can be utilised for our gains.
Footnotes
- Given that Israel is quite advanced in the area of high tech military weapon systems it is surprising that Isreal was not joining the blockade.
- https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf
- A list of organisations can be found on https://autonomousweapons.org, further organisations are Amnesty International and Human Rights Watch
- evidence for static views in each generation: https://www.economist.com/graphic-detail/2019/10/31/societies-change-their-minds-faster-than-people-do
- Tobi Ord: The Precipice, p. 201
- The complete calculation is done via discounting on future returns via the Bellman optimality equation. Algorithms in the domain of artificial intelligence use this calculation to train intelligent agents to chose the best action by reinforcement learning.
- This equation is simplified to guide our understanding of the problem. The development of one technology influences the utility of other existing technologies. E.g. the discovery of plane flight changed the impact of improvements in high speed trains. Some technologies might help tackling one existential risk like climate change, but at the same time increase the risks of other existential risks like non-aligned artificial general intelligence.
- Interestingly there is one case where actions do not follow this utility path. When developing a standard to a new technology and it is done too early the standard is not sufficient in capturing the problem it is trying to solve. A new one will then emerge, which will coexist. Then two exists. To unify them a third one is later introduced, which causes more fragmentation etc.
- https://80000hours.org/articles/ai-capabilities/