In the process of building AI question answering systems and knowledge augmentation services, RAG (Retrieval Augmented Generation) architecture has gained traction due to its efficiency and accuracy. However, many teams often overlook the importance of corpus quality and splitting strategies when implementing RAG systems, which are key factors in determining the success of the system. This article will delve into how to improve the accuracy and maintainability of RAG systems through high-quality corpora and scientific splitting strategies.
In recent years, RAG (Retrieval-Augmented Generation) is gradually becoming the mainstream architecture for enterprises to build AI question and answer systems and knowledge enhancement services. It effectively improves the accuracy and controllability of the Q&A system by “retrieving knowledge first, and then calling the large model to generate answers”.
However, in many enterprise-level RAG projects we have participated in, we have found that many teams tend to focus on “upper-level technologies” such as model selection and vector retrieval, but ignore the real “foundation” of the system – corpus quality and splitting strategy.
In fact, high-quality corpus data and scientific content organization are the keys to determining whether the RAG system can be accurate, maintainable, and stable.
This article will focus on two core questions:
- How to build a high-quality corpus that can support the operation of AI systems?
- How to choose a reasonable splitting strategy to improve retrieval accuracy and generation quality?
01 Enterprise Knowledge Data ≠ General Corpus: Build a knowledge base and first identify the object
A high-accuracy RAG system must first be built on:A high-quality, well-structured, semantically complete corpusOn the basis. No matter how advanced the algorithm is, if the underlying corpus data quality is not good, the performance of the system will also be limited. We have proven in multiple projects –By optimizing the corpus content structure alone, the accuracy can be improved by more than 20% under the premise that the model and parameters remain unchanged.
What does a product manager need to do?
In the process of a product from scratch, it is not easy to do a good job in the role of product manager, in addition to the well-known writing requirements, writing requirements, writing requirements, there are many things to do. The product manager is not what you think, but will only ask you for trouble, make a request:
View details >
Compared with the public corpus on the Internet, the internal data of the enterprise has the following significant characteristics:
- Diverse data sources:It involves product manuals, process systems, training materials, email communication, customer service records, etc., which are often distributed in multiple platforms and systems;
- Highly heterogeneous format:There are various formats such as PDF, Word, tables, pictures, JSON, XML, etc.;
- Terminology intensive:It contains a large number of industry terms, abbreviations, and code identifications, which poses a challenge to the understanding of general large models;
- High timeliness requirements:Enterprise knowledge is frequently updated, and policies, products, and processes need to be updated synchronously.
thereforeThe standard of enterprise corpus is not only “content”, but “machine readable, organizable and controllable”.
02 Build a high-quality corpus: from cleaning, structuring to evaluation system
We have summarized a set of knowledge collation processes that are suitable for most enterprises, divided into the following five steps:
1. Data source identification and access
- Clarify key business issues, such as customer service focusing on FAQs, internal training focusing on process systems, etc.;
- sort out the list of data sources and give priority to access the core content;
- Establish a standardized or automated data synchronization mechanism.
2. Content cleaning and pre-treatment
- Remove irrelevant content, correct typography, merge redundant information;
- Spelling grammar correction and unified naming standards;
- Scripting tools are often combined with human review.
3. Format standardization and structured processing
- Convert various formats uniformly to plain text or Markdown;
- Extract title hierarchy, list structure, and key entities for easy indexing and semantic understanding.
4. Metadata and label system construction
- Add meta information such as source, version, author, and scope of application to each piece of knowledge;
- Support subsequent retrieval sorting, permission control, and knowledge evolution management.
5. Version control and update mechanism
- Establish a regular synchronization mechanism to record update logs and keep historical versions.
- Ensure that the RAG system is constantly using “up-to-date and effective” knowledge.
Corpus quality can be assessed regularly from five dimensions:
- Completeness:Are all core business issues covered?
- Accuracy:Is there misleading or outdated information in the content?
- Consistency:Is the terminology uniform? Are there any information conflicts?
- Timeliness:Is the content updated in a timely manner?
- Usability:Can the machine parse and organize correctly?
Through technical means such as automatic detection, statistical analysis, and expert spot checks, problems and user feedback are collected during the operation of the system, and the quality of the knowledge base can be continuously iteratively optimized.
03 Splitting strategy affects accuracy: The smaller the chunk, the better
In RAG systems, the raw corpus must be split into retrievableChunk。 This step may seem like a technical detail, but it actually has a huge impact on the accuracy, response speed and generation effect of the system.
Why split is necessary:
- More accurate search:Avoid returning the entire irrelevant content and reduce the processing burden of large models.
- The context is more focused:Reduce the chance of large models being interfered with by context-independent content.
- Improved search efficiency:Each chunk vector is more accurate, more responsive, and also supports fast indexing and recall.
- Enhanced Semantic Understanding:Organize related content together to form semantically coherent chunks to better understand the relationship between contexts.
In actual projects, we found that the accuracy of the system’s answers can be achieved simply by optimizing the splitting strategyIncrease by 10%~15%.
But splitting also comes with challenges:Too small will destroy the semantic integrity, lose the association between paragraphs, and disassemble too much and the retrieval will be inaccurate. At the same time, different types of documents also have different structures and semantic characteristics, requiring different splitting strategies. Therefore, it is necessary to find a balance between “granularity” and “semantic context”.
The impact of splitting granularity on retrieval performance is multifaceted: the splitting is too coarse, and it is easy to recall irrelevant content; If the splitting is too fine, the vector library will be too large and the semantics will be fragmented, affecting the coherence of generation. The best practice is usuallyParagraph + sentence level mix strategy, and make dynamic adjustments based on actual business scenarios.
04 Selection and optimization of corpus splitting strategy
Three types of splitting strategies
1. Rule-based splitting
- Fixed-length splitting: divided by the number of characters/tokens, easy to implement, high computational efficiency but easy to separate semantics;
- Separate by punctuation/paragraph: respects text structure, suitable for natural language documents; However, it can lead to uneven block size, requiring additional processing to control block size;
- Divide by document structure (title, list): suitable for technical documents and operation manuals.
2. Semantic-based splitting
- Topic-aware type: divide the content based on topic transformation to improve the internal consistency of each piece;
- Semantic similarity splitting: calculates semantic boundaries through text embedding, suitable for complex long texts;
- Entity and Relationship Splitting: Maintains logical integrity between entities, suitable for knowledge-intensive content.
3. Hybrid and industry customized splits
- Multi-level splitting and hierarchical indexing: Chapters→paragraphs→ sentences, and establish multi-level splitting, which not only improves retrieval accuracy but also maintains computational efficiency;
- Adaptive strategy: flexibly adjust the granularity according to the corpus density, adopt finer split granularity for dense content, and use coarser split granularity for narrative type to ensure a balance between efficiency and accuracy;
- Industry-Tailored Rules: Splitting strategies tailored to specific domains or document types. For example, for legal documents, special attention may be needed on terms and citation relationships; For medical documentation, special attention may be needed on the relationship between diseases, symptoms, and treatments.
Selection of practical cases
aboutDocumentation for the financial industry, such as annual reports, prospectuses, etc., adopting a structure-based multi-level splitting strategy. Start by section, then use a more granular split of the financial data section to ensure accurate answers to queries about specific financial metrics. This strategy increased the retrieval accuracy from the initial 70% to 92%.
aboutTechnical documentation, such as API documentation, technical manuals, etc., splitting based on semantic units works best. For example, take each API method with its parameters, return values, sample code, etc. as a complete block, even though this block may be large. This approach ensures the integrity of technical information and improves the accuracy of answers.
aboutCustomer Service FAQ document, using the question and answer pair as the basic splitting unit, ensuring that each question and its answer remain in the same block. We also establish semantic associations between questions, and when the answer to one question references another question, the system can automatically associate these related contents.
05 Written at the end: corpus is the foundation, and the “first principle” cannot be ignored
Building a high-quality, maintainable, and truly production-usable RAG system does not start with “model replacement”, but with “knowledge polishing”.
When many enterprises implement RAG systems, they often focus on the “superstructure” such as model selection and vector library performance, but ignore themThe “foundation” of corpus data and splitting strategy。 And it is this part that often determines the actual answer quality and user experience of the system, and it is also the projectA watershed moment of success or failure。
The quality of the corpus determines whether the system can answer accurately, and the splitting strategy determines whether the system can answer stably.
It is recommended that every team that is building an enterprise AI Q&A system starts from the following three questions to self-check:
- Is our knowledge data processing process clear and standardized enough?
- Can real user questions be accurately covered in our corpus?
- Can the system provide stable and credible answers in 95% of scenarios?
Corpus is an asset, and accuracy is productivity.
The future belongs to those teams that really polish the corpus as a “product”.
