TOON Reduces Token Waste in LLM Models

JSON and Token Waste in LLM Models

The JSON format is widely appreciated for its simplicity and efficiency in APIs, storage, and application logic. However, when it comes to large language model (LLM) pipelines, JSON can become a burden. Indeed, it generates a token overhead that does not provide significant added value to the model. The braces, quotes, commas, and repeated field names on each line are all elements that unnecessarily weigh down LLM prompts.

To illustrate this problem, imagine sending 100 support tickets or user records to a model. Each object contains the same field names, leading to excessive repetition. This is where TOON, or Token-Oriented Object Notation, comes into play. This newer format is designed to maintain the same data model as JSON while using fewer tokens. It also provides models with clearer structural cues.

TOON: A Compact and Efficient Alternative

TOON presents itself as a compact and lossless representation of JSON, specially tailored for LLM input. The official documentation describes TOON as particularly effective for uniform arrays of objects. In other words, TOON allows you to declare fields once and then spread the row values in a compact tabular form. For example, instead of repeating the fields for each user, TOON declares them once, thereby reducing redundancy.

Here’s a concrete example:

{ "id": 1, "name": "Alice", "role": "admin" },
{ "id": 2, "name": "Bob", "role": "user" },
{ "id": 3, "name": "Charlie", "role": "user" }

With TOON, this becomes:

users[3]{id,name,role}:

The structure remains clear, but the repeated keys have disappeared. This is where TOON derives most of its value.

When is TOON Most Useful?

TOON is a serialization format that can represent objects, arrays, strings, numbers, booleans, and null values. It is designed to be lossless compared to JSON, meaning you can convert JSON to TOON and vice versa without losing information. However, it is not necessary to replace JSON in your application. A better approach is to keep JSON in your backend, APIs, and storage, and then convert it to TOON only when you are about to send structured data to an LLM.

TOON is particularly useful when your prompt contains repeated structured records with the same fields. Good examples include retrieved support tickets, catalog lines, analytics records, tool outputs, CRM entries, or memory snapshots for agent systems. However, if your structure is deeply nested, highly irregular, purely flat, or very small, the benefits may diminish or disappear.

How to Get Started with TOON

Step 1: Install the TOON Command Line Interface

The easiest way to try TOON is with the official command line interface (CLI) of the TOON project. The TOON site directly links to its CLI, and the main repository presents the format as part of a larger ecosystem of SDKs and tools.

To install the package, use the following command:

npm install -g @toon-format/cli

Step 2: Convert a JSON File to TOON

Start by creating a folder, then run the following command to create the JSON file:

{ "id": 1, "name": "Alice", "role": "admin" },
{ "id": 2, "name": "Bob", "role": "user" },
{ "id": 3, "name": "Charlie", "role": "user" }

Next, use this command to convert the JSON file to TOON:

npx @toon-format/cli users.json -o users.toon

You will get a compact result similar to this:

[3]{id,name,role}:

This is the basic TOON model: declare the shape once, then list the values line by line. This aligns with the official design goal of tabular arrays for uniform objects.

Step 3: Use TOON as Model Input

The best place to use TOON is on the input side of your pipeline. Instead of pasting a large JSON blob into a prompt, pass the TOON version and keep the instruction simple.

The following data is in TOON format:

users[3]{id,name,role}:

Summarize the users' roles and report anything unusual.

This works well because TOON is designed to help the model read a repeated structure with less overhead. This is also how the official project frames its benchmarks: as a comprehension test across different structured input formats.

Step 4: Keep JSON for Outputs

This is one of the most important practical decisions. TOON is very useful for input, but JSON is generally the best choice for output when another system needs to parse the model's response. This is because JSON benefits from much stronger tool support, and modern APIs can enforce structured JSON output with schemas.

In practice, the safest model is:

• JSON in your application.
• TOON for significant structured prompt context.
• JSON again for machine-parsable model responses.

This gives you efficiency on the input side and reliability on the output side.

Step 5: Evaluate in Your Own Pipeline

Do not change formats solely based on trends.

Conduct a small benchmark in your own workflow:

• Count input tokens for JSON.
• Count input tokens for TOON.
• Compare latency.
• Compare response quality.
• Compare total cost.

The official TOON project positions token savings as one of the main benefits, and third-party coverage echoes these claims, but community discussions also show that results heavily depend on the shape of the data. That’s why the best question is not "Is TOON better than JSON?"

The best question is: "Is TOON better for this specific step of the LLM?"

Conclusion

TOON is not something you need to use everywhere.

It is a targeted optimization for a specific problem: token waste on a repeated JSON structure within LLM prompts. If your pipeline passes many repeated structured records to a model, TOON is worth testing. If your payloads are small, irregular, or highly nested, JSON may still be the best choice.

The smartest way to adopt it is simple: keep JSON where JSON already works well, use TOON when packaging large structured inputs into prompts, and evaluate the results on your own tasks before committing to its use.