AI Knowledge Bases: 6 Steps to Avoid Mistakes

Building a knowledge base for AI models is not a one-time task, but an iterative process of refinement. An accurate and well-organized knowledge base enhances both the speed and accuracy of the model, areas where current models often fall short. Indeed, a recent study shows that leading AI chatbots make mistakes almost every second query.

Adopting a systematic approach to building a knowledge base helps you create one that is standardized, scalable, and explicit. Any new developer can easily add or update the knowledge base over time to keep it current and reliable.

To ensure you achieve this, you can follow these six steps every time you start creating a knowledge base:

Collect Relevant Data

A common misconception when collecting data for a knowledge base is to assume that more data is better. This leads you into the classic "garbage in, garbage out" problem.

Prioritize value over volume and collect all relevant data for your model. This could take the form of:

• Factual and tutorial content covering facts and procedures
• Problem-solving content in the form of instructional text or videos
• Historical data showing past issues or execution logs
• Real-time data covering the live system status or recent news feeds
• Domain data to provide more context to the model

It is important to understand that your system does not need all information. For example, if you are building a customer support chatbot, your model will only need factual and tutorial content explaining the company's policies and procedures. This ensures that your model does not invent invalid or off-topic responses and is limited to what it is provided.

There is a growing trend to feed AI-generated data when building a knowledge base for new AI models. This practice offers speed, but you must verify the reliability and relevance of the results. Always optimize the content for clear responses and check the output before adding it to the knowledge base.

Clean and Segment Data into Chunks

After preparing the raw data, you can first clean it. The cleaning process typically includes:

• Removing duplicate and outdated content
• Eliminating irrelevant details such as headers, footers, and page numbers
• Standardizing content, both in terms of format and content (consistent terminology)

These cleaned data are then divided into logical chunks, where each chunk contains a clear idea or topic. Each chunk is also assigned metadata that provides quick context about the content it contains. This metadata helps AI models navigate knowledge bases more quickly and reach chunks containing relevant details swiftly.

You can also set role-based access on the chunks to ensure which roles have access to the information in that chunk. While many roles may have access to a model, not everyone can access all data. Segmentation is where you can define security and access control within the model.

A good practice is to segment data based on user queries rather than the document structure. For example, if you have a document on managing logins and access, you can segment it according to common user questions such as "How to change my password?", "What is the password policy?", etc. You can then validate these chunks by testing them against real queries. A safe set might be 10 to 12 questions.

Organize and Index Data

Text chunks are converted into numbers called vectors using an embedding model such as OpenAI v3-Large, BGE-M3, etc. AI models can traverse vectors faster than a large block of text. After vectorization, the metadata attached to the chunk is then associated with the vector. The final chunk will look like this:

[ Vector (numbers) ] + [ Original Text ] + [ Metadata ]

Choose a Platform to Store Data

You can store this vector output in a vector database such as Pinecone, Milvus, or Weaviate for retrieval. You can upload the vector data by writing simple Python code.

To increase upload speed, it is suggested to use the batch insertion option. You can also normalize the vectors (make them all the same size) during the upload phase. After normalization, quantize (compress) to optimize storage. This additional step of normalization and quantization speeds up subsequent retrieval.

Optimize Retrieval

To enable retrieval from the vector database, you can use orchestration frameworks such as LlamaIndex and LangChain.

• LlamaIndex can traverse the vector database faster and reach the exact chunk where the content related to the user's query is located.
• LangChain then takes the data from the chunk and transforms it according to the user's query, for example, by summarizing the text or drafting an email from it.

Hybrid Retrieval

Take advantage of both keyword search and vector similarity.

Where each approach shines:

• Keywords: searches for exact matches but may miss searches with synonyms
• Embeddings: has the advantage of capturing meaning, but there is a possibility of missing the exact keyword

The hybrid approach is a combination of both to get the best of each method.