On-demand webinar: navigating the challenges to AI success

Expert Human Intervention: The Appen Advantage in RAG Optimization

Published on
June 6, 2024

RAG Overview

Retrieval-augmented generation (RAG) is a technique that enhances language model generation by incorporating external knowledge. This is typically done by retrieving relevant information from a large corpus of documents and using that information to inform the generation process.

Improving the performance of RAG systems presents a considerable challenge for AI developers. Evaluation and optimization tasks often demand extensive trial-and-error efforts, which offer limited insights into the complex underlying process.

Is there a more efficient way to boost RAG system performance? Before we explore a potential solution, let’s look at the core components of RAG systems and understand why they have become the preferred choice for domain specific generative AI projects.

Integrating Human-in-the-loop in RAG systems is a necessity, not a choice. Here is why:

Ingestion Process

Chunking is the process of dividing prompts and documents into smaller, manageable segments called chunks. These segments can be delineated based on fixed dimensions such as the number of characters, sentences, or paragraphs. In a RAG system, each chunk is converted into an embedding vector, which is then used for retrieval. Optimizing the size of these chunks is critical: smaller, more precise chunks yield a better match between the user's query and the retrieved content, while still allowing the system to strike an appropriate balance between comprehensive coverage and precise retrieval.

Query Process

The RAG query process starts with an initial prompt, which is then refined by a Rewriter to clarify the intent or improve and revise its format. Next, the updated prompt is passed to the Retriever, which extracts relevant chunks of information from a corpus. These chunks are then prioritized by the Reranker to identify the most relevant ones. Finally, the top-ranked chunks undergo LLM Inference to synthesize and generate a coherent and contextually relevant response. This end-to-end process ensures that the final output is a direct answer to the user’s query, optimized for accuracy and relevance.

Why RAG + Humans = Higher Performing AI Applications

By involving human oversight, errors in data can be corrected, and the relevance of retrieved information can be ensured, leading to more accurate and contextually appropriate responses. Human feedback also enables adaptive learning, allowing models to dynamically adjust to complex data scenarios and iteratively improve through continuous refinement.  

Appen's AI Data Platform significantly enhances this process by enabling seamless collaboration between data science and engineering teams, as well as subject matter experts (SMEs) from the business. Our platform supports the collection, preparation, cleansing, annotation, and optimization of high-quality AI training data, which is pivotal for tailoring high-performance RAG models.

Ways Humans Improve RAG Results

Challenge Description
1 - Data Lacks Structure or Clear Format Without clear formatting, identifying meaningful segments or chunks within data becomes challenging.
2 - Absence of Essential Contextual Metadata Without contextual cues to guide chunking, there's a higher likelihood of including irrelevant information within chunks, increasing noise in the retrieval process. In the absence of contextual cues, chunk boundaries may be arbitrarily defined, potentially leading to inaccuracies in chunk selection. This can result in chunks that are too large, containing irrelevant information, or too small, lacking sufficient context for meaningful retrieval.
3 - Data is Out-of-Date or Conflicting Without having a quality assurance/quality control mechanism to ensure the content finding its way into the vector store is up-to-date and accurate, the resulting RAG system can be misguided by invalid context. By building out a workflow whereby internal SMEs can review and validate knowledge base content before it makes its way to a vector store, organizations can ensure their RAG applications have reliable context in generating responses.
4 - Data Segmentation or Granularity Issues Chunking based on character count or sentence breaks may not capture semantic context effectively, leading to mismatches between user queries and retrieved content.
5 - Missing Data Effective RAG systems depend on comprehensive data. When data is missing, it can result in the system failing to retrieve relevant information, leading to incomplete or unsatisfactory answers. To mitigate this, regular data audits and updates should be implemented. Additionally, fallback mechanisms can be designed to prompt for human intervention when data gaps are detected.
6 - Prompt Quality Issues In running an end-to-end evaluation of a RAG system, it is important that the prompts being tested are of high quality and contain sufficient breadth to cover the range of real-world user behavior that is expected to be encountered. Achieving quality in this domain required human SMEs in the loop that understand the knowledge base content and have an intuition around possible usage edge cases.
7 - Rewritten Prompt Quality Issues Prompts that are rewritten to match system expectations may not always capture the user's original intent. This can lead to responses that, while syntactically correct, are semantically misaligned. Human oversight in the rewriting process can ensure that prompts remain true to the user's intent, thus maintaining the effectiveness of the system.
8 - Missing Top Ranked Chunks Occasionally, the most relevant chunks of data are not surfaced by the ranking algorithm. This could be due to issues in the algorithm or gaps in the data. Human involvement in the iterative refinement of ranking algorithms can ensure that top-ranked chunks are not omitted, improving the accuracy of the responses.
9 - Effectiveness of Reranking Reranking mechanisms are crucial for ensuring the best data chunks are presented first. If reranking algorithms are not effectively prioritizing relevant data, response quality diminishes. Human analysts can tune reranking algorithms based on performance reviews to enhance the selection of data chunks.
10 - Response Deviates from Established Guardrails Responses that deviate from established guardrails can lead to the dissemination of misinformation or inappropriate content. Humans can enforce guardrails by periodically reviewing responses and providing corrective feedback to the system to prevent such deviations.
11 - Inaccurate Utilization of Data Chunks by Response If a RAG system inaccurately uses data chunks, it may deliver responses that are contextually irrelevant or factually incorrect. Human oversight can ensure that the system correctly interprets and utilizes data chunks by refining the retrieval algorithms and providing targeted training data.
12 - Response Style and Tone Inconsistencies A RAG system should maintain a consistent style and tone to meet user expectations. However, variations in data can lead to inconsistencies. Human intervention can guide the system towards a standardized response style by editing and curating training datasets that reflect the desired tone.
13 - Incorrect Specificity Responses that are either too vague or overly detailed can impair user experience. Humans can improve specificity by adjusting the system's parameters to better align with the desired level of detail and by adding annotations to the data that highlight the importance of specificity.


Human oversight is crucial for optimizing RAG systems, ensuring they accurately address user queries and maintain up-to-date, relevant responses. Through expert management of data inputs and continuous updates, these systems handle complex real-world demands effectively. Appen's AI Data Platform plays a critical role in this process, providing meticulous data handling—collection, cleansing, annotation, and optimization. This robust platform helps enterprises navigate challenges such as data inconsistencies and outdated information, enabling the creation of precise, reliable, and context-aware AI applications. Thus, Appen stands out as the preferred partner for enterprises poised to maximize their internal data resources in the fast-evolving AI landscape.

Accelerate Your AI Journey

Ready to harness the full power of RAG with Appen's expertise? Contact us for a consultation and see the difference firsthand.

Related posts

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

No items found.