TOON Revolutionizes Token Management for LLMs, Beyond JSON

For years, JSON has established itself as the go-to language for APIs, integrations, and communication between applications. Its simplicity and robustness have made it a favored tool among developers. However, with the emergence of Large Language Models (LLMs), the use of JSON reveals a hidden cost, particularly in terms of token management.

LLMs process JSON differently than traditional applications. Each syntax element, each repeated key, and each nested structure consumes tokens, which can quickly saturate the models' context window and increase operational costs. To address this issue, a new method called TOON (Token-Oriented Object Notation) has been introduced.

TOON presents itself as a more efficient solution for managing tokens while maintaining the same data structure as JSON. By using TOON, field names are declared only once, values are organized in lines, and the structure remains understandable for the model. This approach is particularly advantageous for LLMs dealing with uniform schemas with repeated records, such as in RAG retrieval results or agent tool outputs.

The article highlights concrete examples of converting JSON to TOON, illustrating how this method can be integrated into prompts for LLMs. TOON thus offers token optimization without compromising the validation of outputs, which can continue to use JSON.

However, it is crucial to note that TOON should not replace JSON in all contexts. It is recommended to use it only where relevant, carefully assessing performance compared to JSON. Best practices include validating outputs, considering the reliability of tools and models, and managing edge cases. TOON should be viewed as an optimization layer for context representation rather than a substitute for the enterprise contract.