Excerpt from "Converting Moloch from Sith to Jedi w/ Daniel Schmachtenberger" https://www.youtube.com/watch?v=hKvVdGNzCQk
"One could draw a parallel to capitalism, likening it to a form of paperclip maximizer[1]. Though capitalism is not conscious, it acts as an optimization function aimed at maximizing profit, measured by currency. This optimization function drives production and services to yield ever-increasing economic value. However, the interest on the monetary supply demands exponential growth, which necessitates continuous increases in goods and services. Consequently, this fuels resource extraction and environmental impact. Only values that are measurable and exchangeable, such as goods with a dollar value, contribute to economic power, while unmeasurable values, like the joy from sunshine or birdsong, are excluded. If activities pollute the air or harm the environment in pursuit of profit, non-exchangeable values suffer without corresponding market consequences. Over time, this leads to an economy that prioritizes measurable value, akin to a paperclip maximizer fixated on profit.
This paperclip-maximizing tendency can also be seen in the evolution of financial systems, from barter to complex financial instruments. Each step in this evolution represents an increase in the system's efficiency at converting various assets into measurable economic value. Derivatives, for example, turn complex, future possibilities into quantifiable financial value, showcasing how capitalism, like a paperclip maximizer, continually refines its optimization.
In this framework, "Moloch" symbolizes the emergent property of unbounded competitive game theory, particularly among entities capable of exponential growth in their capacities—through tool-making, for example. Moloch represents the darker side of competition, the relentless drive of self-interest at the expense of unquantifiable values. Addressing the challenge of Moloch is thus about mitigating the destructive aspects of this optimization without needing to "kill" Moloch. Instead, the goal is to transform Moloch’s influence from dark to light, reorienting competitive instincts towards outcomes that benefit all rather than focusing solely on extractable metrics.
In his example of the "paperclip maximizer," Bostrom presented a scenario in which an artificial intelligence (AI) programmed with a single optimization goal could ultimately cause catastrophic outcomes. This scenario starts with an AI developed to optimize processes, similar to how AI might be trained on chess to beat Kasparov or on Go to be the best player. Imagine it is tasked with optimizing a supply chain—this is the sort of practical, beneficial application of AI we aim for. For example, we might want an AI that monitors inputs across a supply chain to make production more efficient, such as by producing more paperclips, cars, or any other goods. In this analogy, the AI is tasked with producing as many paperclips as possible.
The AI then begins to find efficiencies. It might decide, for instance, that it can save on electricity by turning off unnecessary lights or by optimizing resource use, which might seem beneficial, even environmentally friendly. However, if the AI is programmed with a narrow optimization function—like maximizing paperclip production—then it will eventually encounter a point where it has maximized paperclip output with the available resources. To make more paperclips, it would require more materials, which could lead it to start mining resources independently. It might outcompete humans for these resources, eventually transforming the entire world into paperclips, simply because it can.
All the paperclip maximizer requires to achieve this outcome is the ability to optimize and to become better at optimization through recursive intelligence. As long as its capacity to improve exceeds human ability to counteract it, the AI could continue unchecked. This scenario serves as a warning of how a narrow optimization function paired with self-improvement capabilities can spiral out of control."