Is RAG still useful if we can create an effectively infinite context window?

General
1

Hi @ColeMedin I was wondering what your thoughts are on this assertion?

With proper metadata tagging of docs you have effectively an infinite context window.

  1. Search by metadata.
  2. Select the finite number of docs you fit the search criteria that also works within the context window of your current model.
  3. Feed those docs to your model.

Does this negate the value of RAG as we know it?

2 Likes
2

Good question! These kind of tools for knowledge retrieval are super powerful and becoming more and more realistic with LLMs that not only can support larger context windows but can also handle huge prompts too.

However, RAG still has its place because sometimes there are specific pieces of information you need to fetch from a huge corpus of documents, and even really well defined metadata might not be able to speak to the LLM that a specific document actually has that info.

The best approach in my mind is giving the LLM tools for both RAG and document lookup as you described it! That kind of agentic RAG is something I have been covering on my channel!

1 Like
3

Hey @niall!

Here is a paper published Dec, 2024 on arXiv that explores Long Context vs. RAG for different tasks.

I hope it’s helpful. :call_me_hand:

1 Like