Andrej Karpathy Reinvents AI with His Wiki LLM Model

The Challenge of Retrieval-Augmented Generation (RAG)

In the field of artificial intelligence, Retrieval-Augmented Generation (RAG) is a widely adopted method. It involves downloading documents, asking a question, and the system extracts the relevant text to generate a response. Platforms like NotebookLM and ChatGPT utilize this approach. However, Andrej Karpathy, an influential figure in the AI domain, highlights a significant weakness: these systems do not retain acquired knowledge. Each query is treated in isolation, without the accumulation of understanding.

When a question requires synthesizing multiple documents, the system combines them only for that specific query. If the same question is asked again, the process is repeated entirely. This approach makes it difficult to connect information over an extended period, such as linking articles from March to those from October. In summary, while RAG produces answers, it does not build lasting knowledge.

Karpathy's Vision: A Wiki Model for AI

Karpathy proposes an innovative solution that could transform our perception of AI models. Instead of processing documents after each query, he suggests handling them during their initial ingestion. This method allows for the creation of a structured and permanent wiki product, offering greater control over the storage and retrieval of documents.

When a new source is added, the LLM model does not merely create an index for future retrieval. It integrates this source into its knowledge base, updating existing pages and creating new ones as necessary. Contradictions between new and old information are detected and flagged, thereby reinforcing complex relationships throughout the wiki.

According to Karpathy, “With LLMs, knowledge is created and maintained continuously and coherently rather than being generated by individual queries.” This approach stands in stark contrast to the RAG model, where knowledge is processed at the time of each query, while in the Wiki model, it is integrated once during ingestion.

Process of Creating an LLM Wiki

Step 1: Gather Resources

To develop a wiki, it is essential to collect various materials: articles, books, personal notes, discussion transcripts, and even your own historical conversations. These materials serve as the raw material for the wiki, just as ore must be refined before use.

Step 2: Classification Before Extraction

It is crucial not to treat all documents the same way. For example, a 50-page research document requires section-by-section extraction, while a tweet or a social media thread only needs a main idea and the corresponding context. Similarly, a meeting transcript requires the extraction of decisions made, actions to be taken, and key quotes. By first classifying the type of document, you will help extract the right type of information with the appropriate level of detail.

Step 3: Generating Wiki Pages

The source materials are integrated into the LLM via a structured query, allowing for the production of wiki pages that conform to an established template. Each page includes a frontmatter block (YAML), a TLDR sentence, the body of content, and identified counterarguments or data gaps.

Step 4: Creating an Index

A central index.md file serves as a table of contents and will directly link to each page of the wiki. This is how an AI agent can efficiently traverse the entire knowledge base; it starts with the index, reads the TLDRs, and then explores only the pages relevant to its specific question.

Step 5: Recording Questions

This is one of the least appreciated features of the system. When you pose a well-formulated question to the LLM and receive a response that provides valuable information, for example, a comparison between two frameworks or an explanation of the relationship between two concepts, you record this response as a new wiki page labeled with the query-result tag. Over time, your best insights have been collected rather than lost in chat logs.

Step 6: Regular Audits

At appropriate intervals, you ask the LLM to audit the entire wiki for contradictions or inconsistencies between pages and to indicate statements that have been rendered obsolete by a more recent source. Additionally, the LLM will provide suggestions for identifying orphan pages (i.e., pages that have no links pointing to them) and for providing a list of concepts referenced in the existing content but not yet represented by their own respective pages.

Tools for Effective Management

Karpathy uses several tools to optimize this process. Obsidian is a free markdown knowledge management application that uses a local directory as a vault. For Karpathy, it is the visualization interface used for the wiki because it has three distinct features that matter for his system:

• The Graph View provides a graph of all wiki pages represented as nodes, and furthermore, each wiki link ([[wiki-links]]) will be represented as edges connecting all the nodes together. Hub pages will be connected to many nodes and will thus be represented as larger-than-average nodes. Orphan pages will be represented as isolated nodes. This allows for an immediate visual representation of a person's knowledge density and knowledge gaps. No other document display or file browser can provide this visual representation.

• The Dataview Plugin allows users to transform their wiki into a queryable database. All pages must have a YAML frontmatter, so the integration specification is satisfied and allows the user to execute SQL-like queries against all pages of the wiki.

• The Web Clipper browser extension (available for Chrome, Firefox, Safari, Edge, Arc, and Brave) converts web articles into clean Markdown with YAML frontmatter in one click, saving directly into your raw folder. You upload all images to your computer by pressing the Ctrl+Shift+D shortcut after finishing the clipping, as the LLM requires access to the images.

Finally, Qmd enables large-scale searching, making access to wiki content seamless even with a large number of pages. The LLM can use the index.md file to access the wiki content without issue on a small scale. However, the index becomes unreadable in a context window when you have more than 100 pages because the LLM has a context limit. To address this issue, Qmd allows for large-scale searching, facilitating access to a large volume of data.

In conclusion, Karpathy's approach with the Wiki LLM model could transform how AI systems manage and utilize information, making systems smarter and more efficient.