How does RAG capability get used by popular AI coding assistants like Cursor, Codeium, tabnine, sourcegraph Cody and such ?

Retrieval-Augmented Generation (RAG) is a critical capability in modern AI-powered coding assistants like Cursor, Codeium, Tabnine, Sourcegraph Cody, and others. These assistants leverage RAG to enhance code completion, provide contextual suggestions, and answer complex coding questions more effectively. Here’s how RAG is used in these tools:

How embeddings are relevant and used in applying RAG for coding assistance ?

These embeddings are indexed and stored, allowing the AI to retrieve relevant information before generating suggestions.

✅ Semantic Search for Code & Documentation – Retrieves relevant snippets even when phrasing differs.
✅ Multi-File Code Understanding – Helps AI assistants analyze code across multiple files and modules.
✅ Faster Query Resolution – Quickly fetches relevant functions, classes, and past implementations.
✅ Project-Specific Context – Ensures AI completions are aligned with your project’s unique structure.
✅ Better Code Suggestions & Refactoring – Embeddings help improve accuracy and consistency of AI-generated code.

So, embeddings are used mainly for semantic search to construct proper context. Does popular AI Chat tools like ChatGPT, claude.ai and deepseek.com use RAG by default to enhance our prompt text with additional context (if we don't provide additional documentation or attachment) ? The embeddings are not relevant when we don't use RAG ? Or is it integral part of all LLM chat activities one way or another ? LLM itself operates fundamentally based on embeddings or not ?

Yes, embeddings are fundamental to how all Large Language Models (LLMs) operate, but their role differs depending on whether RAG is being used or not. Let’s break this down step by step.

✅ No, not by default—unless explicitly provided with external knowledge sources like documentation, files, or APIs.

🔹 Example:

👉 So, embeddings for retrieval are NOT always used unless RAG is enabled.

🔹 Yes, but in a different way! Even when RAG is not used, embeddings are still a core part of how LLMs generate responses.

✅ Absolutely. LLMs fundamentally operate using embeddings!

👉 So, embeddings are core to all LLMs, but RAG-based embeddings are an additional layer used for retrieval.

1️⃣ All LLMs (ChatGPT, Claude, DeepSeek, etc.) always use embeddings internally to process and generate text.
2️⃣ RAG (Retrieval-Augmented Generation) is separate and adds retrieval-based embeddings when external knowledge is required.
3️⃣ If you don’t provide extra documentation or attachments, these tools typically do not use RAG—they rely only on their built-in training.
4️⃣ Even without RAG, embeddings are at the core of how LLMs work.

Would you like an example of embeddings in action with a simple Python script? 🚀

If I am implementing AI Code assistance tool, I need to be aware of these. The embedding mechanism of LLM is internal to LLM. My tool retrieves additional context using my own embeddings -- This additional context of say, API documentation is a pure list of words or in embeddings format ? Should my AI code assistance tool pass this additional context as list of words to LLM or in the embedding format itself ? Is different embeddings mechanisms used by LLMs and agents expected to be compatible or independent ?

✅ Pass it as a list of words (text), NOT as embeddings.

LLMs do not accept embeddings directly as input. Instead, they process raw text and internally convert it into embeddings before generating responses.

🔹 They are independent and serve different purposes, but they are complementary.

✔ Use embeddings for retrieval, but always pass text to the LLM.
✔ Optimize context injection – You don’t need to pass an entire API document, just the most relevant parts.
✔ Format the retrieved context clearly – Use bullet points, code blocks, or structured formats.
✔ Fine-tune context length – LLMs have token limits, so pass only what’s necessary.

Would you like a sample Python implementation for embedding-based retrieval using FAISS or Pinecone? 🚀

So RAG based embeddings involve the raw embedding mechanism used to index the domain specific Knowledge (like entire code base of my repository) and the tricks and algorithms to extract relevant information from the domain source. What are the popular choices of the raw embedding technology and the algorithms that extract relevant information using these embeddings today ?

Yes, you’ve got it right! RAG-based embeddings involve two key aspects:
1️⃣ Raw embedding technology – Converts your domain-specific knowledge (e.g., API docs, codebase, previous chat history) into high-dimensional vectors and stores them in a vector database.
2️⃣ Retrieval algorithms – Efficiently extract the most relevant information from the stored embeddings based on a user query.

In this context of trying to create embeddings for the entire code base, does it apply embedding for every word of the code or chunks of the code ? You mentioned OpenAI providing text-embedding-ada-002 that helps with creating embedding for our data. Do they provide API for this and if so, is it usually cheaper than LLM chat completion requests pricing in terms of cost per millions of tokens ? Could this embedding be built incrementally if I encounter a large repository with more new modules ? A vector databased based on embeddings should decide one fixed embeddings and can not mix different embeddings in order to be useful, right ? Vector databases like FAISS expect you to create the embeddings or do they provide integrated service of creating embeddings as well and charge the API cost to you ?

Some opensource coding assistants let you provide your own LLM Model (and API Key) for code assistance. Do any of them also directly expose you to select the embedding-model and Vector Database Providers and let you directly pay for them as well ? If I use FAISS embeddings, do I create a separate API Key with Facebook and pay for the usage ?