Ground LLM answers in your own documents — embeddings, vector search, and the RAG pipeline.
Published September 21, 2026
A large language model only knows what was in its training data. It doesn't know your company's policies, your product docs, or anything that happened after its training cut-off. When asked anyway, it may produce a confident, plausible and wrong answer (a hallucination).
Retrieval-Augmented Generation (RAG) fixes this by giving the model the relevant facts at question time: search your own documents for passages related to the question, paste them into the prompt, and instruct the model to answer from them. The model supplies the language skills, and your documents supply the facts.
| Need | RAG | Fine-tuning |
|---|---|---|
| Answer from private or changing documents | ✅ Update the index, and answers change immediately | ❌ Retrain for every change |
| Cite sources | ✅ You know which passages were used | ❌ Knowledge is blended into weights |
| Control access (user sees only their docs) | ✅ Filter at retrieval time | ❌ Hard |
| Change the model's style or teach a format | ❌ Limited | ✅ What fine-tuning is good at |
Rule of thumb: RAG for knowledge, fine-tuning for behaviour. Many systems use both.
INDEXING (offline, whenever documents change)
documents → clean → split into chunks → embed each chunk → store vectors + text + metadata
QUERYING (online, per question)
question → embed → vector search (top-k similar chunks) → [optional re-rank]
→ build prompt: instructions + retrieved chunks + question → LLM → answer (+ sources)
An embedding model turns text into a vector: a list of hundreds or thousands of numbers positioned so that texts with similar meaning are close together. "How do I reset my password?" and "Forgot login credentials" end up near each other even though they share no words. A vector store (pgvector, Qdrant, Weaviate, Pinecone, Elasticsearch/OpenSearch) finds the chunks nearest to the question's vector, usually by cosine similarity, using approximate nearest-neighbour indexes such as HNSW to stay fast over millions of chunks.
Important: the question and the documents must be embedded with the same model. Switching embedding models means re-embedding everything.
Documents are split into chunks because whole documents are too long to embed meaningfully and too big to paste into prompts.
// Indexing
@Service
@RequiredArgsConstructor
public class DocumentIngestion {
private final VectorStore vectorStore;
public void ingest(String text, Map<String, Object> metadata) {
List<Document> chunks = new TokenTextSplitter().apply(List.of(new Document(text, metadata)));
vectorStore.add(chunks); // embeds each chunk with the configured embedding model and stores it
}
}
// Querying
@Service
@RequiredArgsConstructor
public class SupportAssistant {
private final VectorStore vectorStore;
private final ChatClient chatClient;
public String answer(String question) {
List<Document> hits = vectorStore.similaritySearch(
SearchRequest.builder().query(question).topK(4).similarityThreshold(0.6).build());
String context = hits.stream()
.map(d -> "[source: " + d.getMetadata().get("source") + "]\n" + d.getText())
.collect(Collectors.joining("\n\n---\n\n"));
return chatClient.prompt()
.system("""
Answer using ONLY the context provided. If the context does not contain the answer,
say you don't know. Cite the source of each fact in square brackets.
""")
.user("Context:\n" + context + "\n\nQuestion: " + question)
.call()
.content();
}
}
Spring AI also ships a QuestionAnswerAdvisor that performs this retrieve-then-prompt step automatically when attached to a ChatClient. The explicit version above shows what happens underneath. (Spring AI's API has changed between milestones. Check the version you use.)
Two prompt details matter a lot:
When answers are wrong, the cause is usually retrieval, not the LLM: the right passage never made it into the prompt. The standard upgrades:
Measure the two halves separately:
Build a small labelled test set from real user questions, and re-run it whenever you change chunking, embeddings or prompts.
Q: When would you fine-tune instead of using RAG? A: Fine-tune to change behaviour: a consistent output format, a domain-specific writing style, or better performance on a narrow task. Use RAG to give the model knowledge that is private, large or frequently changing. They combine well: a fine-tuned model that is still fed retrieved context.
Q: How do you choose chunk size? A: Start around a few hundred tokens with some overlap, split on semantic boundaries, then tune empirically with a retrieval test set. Precise factual lookups favour smaller chunks, and questions needing broader context favour larger ones. Some systems store small chunks for matching but pass the surrounding larger section to the LLM.
Q: Why add keyword search if vectors capture meaning? A: Embeddings are weak at exact identifiers (SKUs, error codes, function names, rare proper nouns), where lexical matching is precise. Hybrid search gets the best of both and is one of the most reliable quality improvements.
Q: How do you stop the model from making things up when nothing relevant is found? A: Use a similarity threshold, so weak matches aren't passed as context. Explicitly instruct the model to say it doesn't know when the context lacks the answer. And evaluate faithfulness. Showing sources to users also makes unsupported answers easier to spot.
Q: What's a vector database actually doing that a normal database doesn't? A: Fast approximate nearest-neighbour search over high-dimensional vectors, using specialised indexes such as HNSW graphs or IVF clustering, that find the closest vectors without comparing against every stored one. Relational databases can add this with extensions (pgvector), which is often enough at moderate scale.