Anthropic: Emmanuel Ameisen Analyzes LLMs' Thinking

Researcher in interpretability at Anthropic, Emmanuel Ameisen describes the internal mechanisms of language models that go beyond the simple "next token." He connects these insights to alignment and safety, supporting them with foundational works and readings. His background combines research and practical applied ML experience.

Interpretability Related to Alignment and Cited Works

Emmanuel Ameisen links interpretability to the alignment and safety of AI. He cites major references in the field, such as "A global workspace in language models" and "Scaling Laws for Neural Language Models." He also mentions "Claude’s Constitution," as well as works by Chris Olah and the publication Distill. Among the sources mentioned is "Tracing the thoughts of a large language model."

What Models Do Internally According to Ameisen

According to Emmanuel Ameisen, language models develop global representations of concepts and predict a large number of tokens in advance without explicitly stating these strategies. He believes that their internal structure could reflect an organization similar to the separation between conscious and unconscious processes observed in humans. He questions the framework of the next token predictor, which he finds reductive for understanding the planning and thinking of models. Ameisen considers that anthropomorphism can be a relevant approach to grasp these systems. He also argues that models are more cultivated than constructed. He discusses the interest of the J space for exploring the potential consciousness of models and describes a continuous progression of capabilities, where some perceive sudden leaps.

Background and Areas of Study

Emmanuel Ameisen is a researcher in interpretability at Anthropic, where he studies language models like Claude. He previously worked for several years in applied machine learning at Stripe and Insight. He is the author of "Building Machine Learning Powered Applications." Among the books he recommends are "A Brief History of Intelligence: Evolution, AI, and the Five Breakthroughs That Made Our Brains," "Stories of Your Life and Others," and "Four Thousand Weeks: Time Management for Mortals."