Anthropic Cuts First Token Delay by 35% on Long Queries

Anthropic has launched a public beta of a conversation mode that retains the history on the service side while remaining encrypted and ephemeral. This approach does not affect token usage but speeds up the first token for very long requests and is already integrated with major cloud providers.

In-Memory Encryption and Five-Minute Expiration

The conversation thread is kept encrypted in memory between two requests and is deleted after five minutes of inactivity. This feature is available on the Claude platform as well as through AWS, Microsoft Foundry, and Google Cloud. Additional information is available in the documentation.

Delta History Sent and Impact on the First Token

Instead of transmitting the entire history with each call, only the changes since the last message are sent. Until now, each API request had to include the entire conversation. According to Anthropic, this mechanism reduces the time to the first token by approximately 35% for requests containing between 500,000 and one million tokens. Token usage remains unchanged, with Claude still reading the complete history.

Agents and Massive Workflows Among the First Beneficiaries

According to Anthropic, this mode particularly benefits agents specialized in coding, agents handling IT tasks, and processes involving the management of large volumes of documents or images. The Message Threads feature is currently available in public beta on Claude.