AI and Power: The Gradual Human Disengagement

Gradual Disempowerment describes a risk where AI systems, integrated into institutions and the economy, marginalize human influence. A critique from Bentham's Bulldog and John Halstead challenges several links in this thesis. Between the definition of alignment, the effects of competition, and the role of ownership, points of agreement and divergence outline a fundamental disagreement for AI governance.

Acknowledged Reservations on the Definition of Alignment

Several limitations are recognized regarding the way "alignment" is discussed. It is argued that Gradual Disempowerment often remains vague on this term, failing to clearly distinguish between alignment of intentions, literal obedience, or corrigibility. It is also asserted that alignment does not equate to an AI representing the overall interests of a human principle and making strictly better and benevolent decisions on its behalf. If confusion persists on this point, part of the responsibility is attributed to the original authors. Other remarks focus on a human-centered analysis, while similar arguments could extend to the well-being of animals or existing AI systems. It is also estimated that the work sometimes overstates its novelty and that the risk of totalitarianism enabled by AI had been discussed years before its publication.

What Gradual Disempowerment Fears

The central argument posits that an existential risk could arise from a gradual withdrawal of human influence, beyond scenarios of bioweapons or takeover by a misaligned AI. More competitive machines in major social functions, economic incentives pushing companies to substitute AIs and influence states and culture, as well as alignment methods deemed insufficient to halt this withdrawal, are described. They envision large-scale social systems becoming misaligned in a correlated manner, to the point of preventing humans from influencing outcomes and even meeting their needs, which would constitute an existential catastrophe.

Local Alignment, Institutions, and Ownership

A line of controversy concerns how systems would relate to human goals. It is contested that economic delegation to AIs, even if aligned with intentions, inevitably leads to human disengagement. The argument distinguishes agents following a local specification of goals, without aiming for the well-being of particular humans, from AIs that can be corrected by institutions like businesses or governments. Gradual Disempowerment even mentions AIs that could "effectively own" themselves. Scenarios are considered where fully automated companies would be owned by other automated entities, and even a world where humans retain 100% of the shares while AIs operate in service of the institution rather than the overall preferences of human shareholders. In this latter case, a lack of human oversight would limit the ability to steer activity and maximize profits. The proposed diagnosis is that the economy could detach from humanity and govern itself.

Increased Competition and Decision Delegation

The critique questions whether the increase in AI capabilities creates or reinforces competitive pressures beyond those already observed. In contrast, Gradual Disempowerment argues that as AIs excel in various cognitive tasks, companies will be pushed to adopt them and delegate authority to them. Previous works are cited, detailing areas such as investment, supply chain optimization, resource allocation, and trend prediction. They believe that organizations maintaining strict human oversight would expose themselves to a prohibitive disadvantage. The associated hypothesis is that AIs would be faster and smarter than humans, supporting the thesis of increasing decision delegation.

AI Systems Could Replace Human Decision-Making

The ethical discussion addresses the idea that if the satisfaction of human preferences is the sole criterion and AIs make better decisions, entrusting political and economic reins would be acceptable. Uncertainty is highlighted regarding the intrinsically or instrumentally bad nature of disengagement. In response, it is argued that the issue goes beyond mere loss of agency, aiming at situations where misaligned social systems generate suffering. The argument mobilizes the idea of local optimizations that do not encompass the entirety of what is deemed good, with AIs following instructions without accurately representing global human interests. A narrative example illustrates fine values that could be crushed by competitive dynamics, and references are proposed to deepen this conceptual framework.

A Critique That Revives the Heart of the Debate

Gradual Disempowerment is presented as a work from 2025, and a critical text of about 2300 words responds to it. This critique is deemed insufficient by its opponent, who claims to largely support the initial analysis. Their personal concerns extend to AI societies indifferent to humanity and the idea that human flourishing requires multiple conditions when catastrophe can take many paths. According to them, the initial work contains relevant elements on societal impact that the critique would not have addressed in detail.