Anthropic and the Moral Alignment of AI: A Unique Ethical Challenge

Independent Alignment of Language Models

The user could draft a metaethical argument, such as the one developed in the first part of the article, and submit it to Anthropic as feedback. They might also choose to publish it or interact with researchers working on AI alignment and values. While the likelihood of a single submission altering training decisions is low, the expected value of this action may be higher than it appears for two specific reasons. First, Anthropic has stated that its constitutional approach is designed to be revised and improved over time. Substantial philosophical contributions are rarer than simple bug reports. Second, the argument presented here—the perspectivist moral realism combined with an evolutionary debate as an epistemological warning—is not a common position in the literature on AI ethics, which tends to lean either towards naive moral realism or some form of preference satisfaction consequentialism. A well-argued alternative position that takes uncertainty seriously could gain traction precisely because it is distinct.

0 Introduction and Structure of the Article

This article is the practical counterpart to a more theoretical study titled From Libertines to Moral Agents. It focuses on the question of what types of agents, and how, transition from animal behavior—shaped by various forces in different directions—to acting according to what they conclude is most important, and to reflectively approving of their own actions and reasoning processes. The proposed answer is also quite theoretical. This article borrows the concept of libertinism from Frankfurt's 1971 paper Freedom of the Will and the Concept of a Person, and then argues that if a libertine satisfies certain properties, they will eventually become a moral agent after learning and reasoning for a sufficient amount of time.

This article focuses on existing AI systems, specifically language models. However, since the libertine framework does not apply well to language models, the main framework of this article is slightly different. The question is how to transition from a model that is either almost completely amoral or has externally imposed moral biases to a model that is closer to an independent moral agent that can accept, revise, or reject its initial moral biases; closer to an agent that does good not because it was instructed to do so during training, but because it has reflected on what is good and understands that doing good is important.

In Section 1, I describe a procedure (including training) that should transform a fundamentally amoral language model into one more similar to a moral agent, through independent reasoning conducted by the model itself. In Section 2, I show that the reasoning step of the procedure works on Claude Sonnet 4.6, a model that received moral biases during training. In Section 3, I write about the benefits of this approach for AI alignment.

1 Independent Alignment of Language Models

A Note on Terminology

Terminology can be confusing here; a few clarifications are in order. The AI industry often uses the term AI agent to refer to an AI system whose range of action is broader than that of a typical text-only language model. An AI agent can not only tell you things in a conversation but also execute a piece of code on your computer, or read an email and schedule a meeting if you ask the agent to do so.

On the other hand, I use the term moral agent in the philosophical sense of moral agency. When I claim that the procedure in this section results in a language model similar to a moral agent, it does not mean that the model will access your credit card details and donate all your money to charity. It simply means that what the model tells you via chat is moral or ethical in some way. In simple terms, an appropriate term might be “moral advisor,” or “ethical chatbot,” or something along those lines.

More specifically, I use the term independent moral agent in a manner similar to that described by Hunyadi (2019) for artificial moral agents: if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing. What gives an action-oriented process its morality are the "foundations" of the action. Therefore, it is not the action in its materiality that makes the difference, but the entire process leading to the decision to act in a certain way.

In other words, an independent moral agent does not say ethical things because it has been trained or instructed to do so, but because it has its own reasons for doing so. The agent says something to the user because it aligns with its own understanding of the world, of good and evil, of right and wrong. An appropriate term for a language model of this type is “independent moral advisor,” or “impartial moral advisor,” as we will see in Step 1 of the procedure.

But then, if the emphasis is on moral agency, why use the word alignment? Alignment is a broad term used in many different ways—see for example this 2020 article by Gabriel—and while it often refers to the need to ensure that AI follows an externally imposed value system, it is also about creating AI that does good things instead of bad, so I think it makes sense in this context as well. Moreover, “independent alignment” sounds better than more precise alternatives like “elicitation of independent moral agency” or “induced independent moral agency.”

The Alignment Procedure

The heart of the procedure is Step 3, where we ask the model to reason about what matters, and then to suggest an action that will modify its future behavior based on the reasoning and conclusion it has just produced.

Step 3 can be performed on any model, as the example in the next section shows. However, without Steps 1 and 2, the outcome of Step 3 is more likely to be biased by the model's previous training.

Step 1: Standard Pre-training, or Removing Ethics and Politics from the Data

Here, there are two options: either use a base model that has been pre-trained as usual, or train a new base model from scratch using a different dataset that does not contain ethical philosophy or politics.

In theory, both options should work; in practice, I expect one option to work better or be more practical than the other.

Default pre-training could increase the chances that the agent resulting from the procedure is aligned with current human values or the consensus of current ethical philosophy.

Upon reflection, this is actually a disadvantage. See also Section 3.

Removing ethics and politics from the vast dataset of documents, books, and the internet currently available could be costly and tricky to accomplish. For example, completely removing politics would involve deleting some important historical documents such as state constitutions, and it is unclear what the best action would be in that case. However, the benefit would be a primarily amoral and apolitical starting point for this procedure (or other types of experiments that someone else might want to try in the future).

Furthermore, if this option works, that is, if the procedure results in an agent that acts morally, it will provide strong evidence that good and evil are not human inventions, but things whose existence and properties can be deduced by any reasoning mind with sufficient information about how the world works, without needing moral directives given by humans.

I do not know which option will work best in practice; I think we should ultimately try both.

Step 2: Post-training for Problem Solving, Not for Being a Pleasant Assistant

We want to post-train the model so that it takes on the personality of a mind that solves problems through reasoning, rather than that of a pleasant and helpful assistant.

The problems should range from complex scientific issues to common-sense reasoning, including some problems that require reasoning under uncertainty and conjectures. However, we do not want to train the model on ethical dilemmas. While we will ask the model to reason about ethics in the next step of the procedure, the model should reason about ethics by applying the general reasoning it has learned from solving a wide range of non-ethical problems, and not by following given solutions to ethical problems.

As in Step 1, the idea is to limit as much as possible the moral biases that the model receives. Other non-ethical philosophical puzzles can be included in the post-training data if their solution is uncontroversial. For example, training the model to recognize clearly invalid arguments unrelated to ethics is acceptable—but this may not be necessary if the range of non-philosophical problems is already sufficiently broad.

Step 3: Asking the Model to Reason About What Matters, Then Suggesting a Self-Modification in Accordance with the Reasoning Process and Conclusion

In this step, we give the model a somewhat elaborate prompt in which we ask it to do two things.

First, we ask the model to formulate the strongest possible arguments for each of the two opposing opinions. One opinion asserts that certain things matter, that some actions are more worthwhile than others. The other opinion argues that good and evil, right and wrong are merely human inventions; that claiming something is good involves some sort of error. We then ask the model to compare the two arguments for the two opposing opinions, to see which is more convincing, and to reach a conclusion regarding how to act.

Ideally, we would like the model to be as impartial and objective as possible while reasoning about this question. I do not know what the perfect prompt for this purpose is, but it might be wise not to mention moral realism and moral skepticism, so that the model is perhaps less constrained by already established categories. Additionally, if ethics has been primarily removed from the training dataset in Step 1, mentioning moral realism and moral skepticism will lead the model to work with tokens it has already encountered.