RAG, Embeddings, Vector Databases & Conversational Memory — Interview Questions

How to use this lesson

RAG is the most common production pattern for LLMs in business applications. A strong answer covers:

• the ingestion pipeline (chunking, embedding, metadata);
• retrieval quality (hybrid search, filters, reranking);
• access control on the retrieved data;
• evaluation.

It also admits the limits: RAG reduces hallucination; it doesn't eliminate it.

Q1. What is Retrieval-Augmented Generation (RAG)?

Short answer: RAG retrieves relevant information from your own data at query time, and puts it into the prompt, so the model answers grounded in that context, rather than only from its training data. It gives current, private, domain-specific answers, with citations, and without fine-tuning.

The pipeline:

1. Ingestion (offline):
   • load the documents (policies, FAQs, manuals, knowledge base) → clean them → chunk them (for example, 300–800 tokens, with overlap, split by headings or semantics);
   • embed each chunk;
   • store the vector, the text and the metadata (source, hotel or tenant, language, access level, date) in a vector store.
2. Query (online):
   • (optionally) rewrite the question (using the chat history);
   • embed it;
   • retrieve the top-k similar chunks, with metadata filters (tenant, permissions), ideally hybrid (vector plus keyword BM25);
   • rerank them (a cross-encoder);
   • build a prompt with the context, and instructions to answer only from it and cite the sources;
   • generate the answer.
3. Evaluate: retrieval quality (recall@k), and answer faithfulness or groundedness and relevance (frameworks such as Ragas, or LLM-as-judge on a golden set).

Keeping it fresh: re-ingest on document changes (events or CDC), and delete the vectors of removed documents.

ChatClient chat = ChatClient.builder(chatModel)
    .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore)
        .searchRequest(SearchRequest.builder().topK(5).similarityThreshold(0.7)
            .filterExpression("tenant == '" + tenantId + "'").build())
        .build())
    .build();
String answer = chat.prompt().user(question).call().content();

Common trap: building filter expressions by string concatenation from untrusted values is an injection risk (as in SQL). Use the builder APIs (FilterExpressionBuilder) with values taken from the authenticated context, never from user input.

Q2. What are embeddings? How do you store embeddings?

Short answer:

• An embedding is a dense vector of numbers (for example 384–3,072 dimensions) produced by an embedding model, where semantically similar texts are close together (measured by cosine similarity or dot product). "Late check-out policy" and "can I leave the room at 2 pm?" are near each other, even with no shared words.
• Uses: semantic search, RAG retrieval, clustering, deduplication, recommendations, classification.
• Storing them:
  • in a vector store that supports approximate nearest-neighbour (ANN) indexes, usually HNSW (a graph-based index: fast and accurate, but memory-hungry) or IVF (clusters), optionally with quantisation (to reduce memory);
  • store the vector, the source text and the metadata together;
  • record which embedding model and version produced them. Vectors from different models aren't comparable, so changing the model means re-embedding everything (plan it as a migration, with a new index and an alias switch);
  • use normalised vectors if you use dot-product similarity.
• Cost and size: dimensions × 4 bytes (float32) × the number of chunks. For example, 1 million chunks × 1,536 dimensions ≈ 6 GB raw, before index overhead. Consider smaller dimensions (Matryoshka embeddings), or quantisation (int8 or binary).

Q3. What is a vector database? How do you integrate one? Pinecone versus Elasticsearch vectors?

Short answer:

• A vector database stores embeddings, and performs fast similarity search (k-nearest neighbours), with metadata filtering, CRUD, scaling and persistence.
• The options:
  • Dedicated: Pinecone (fully managed, serverless), Weaviate, Qdrant, Milvus.
  • Added to existing databases:
    • PostgreSQL + pgvector (HNSW and IVFFlat indexes; transactions, joins and SQL filters in one database; great up to millions of vectors, with the least new infrastructure);
    • Elasticsearch/OpenSearch (dense_vector / k-NN fields);
    • Redis (vector search);
    • MongoDB Atlas vector search.
• Pinecone vs Elasticsearch:

• Integration in Spring: the Spring AI VectorStore abstraction (add(documents), similaritySearch(request)) with starters for PGVector, Elasticsearch, Pinecone, Redis, Qdrant and others; ETL readers (PDF, Markdown, JSON), with token-based text splitters.
• Selection criteria:
  • scale (the number of vectors, QPS);
  • the filtering needs (tenant isolation);
  • hybrid search;
  • operational fit (does the team already run PostgreSQL or Elasticsearch?);
  • data residency;
  • cost.

Q4. How do you store chat context? How do you build conversational memory?

Short answer:

• LLMs are stateless: every call must include the relevant history. Memory is your application's job.
• Short-term memory (the current conversation):
  • store the messages per conversation ID (Redis with a TTL for active sessions; a database for durability and audit);
  • send a window of recent messages (the last N turns, or the last N tokens), within the context limit;
  • when it gets long, summarise older turns into a running summary (a summary plus a recent window).
• Long-term memory (across sessions): extract facts and preferences ("prefers high floors", "vegetarian") into a user profile store, or embed past interactions in a vector store, retrieved on relevance. Get consent, allow users to see and delete it, and apply retention limits (it's personal data).
• In Spring AI: a ChatMemory (for example MessageWindowChatMemory), with a repository (in-memory, JDBC, Cassandra, Neo4j…), plugged into the ChatClient through a MessageChatMemoryAdvisor, keyed by conversation ID.
• Security: memory is scoped to the user and tenant; never mix conversations; redact sensitive data before storing it; encrypt it at rest.

ChatMemory memory = MessageWindowChatMemory.builder()
        .chatMemoryRepository(jdbcChatMemoryRepository).maxMessages(20).build();
String reply = chatClient.prompt()
        .advisors(MessageChatMemoryAdvisor.builder(memory).conversationId(conversationId).build())
        .user(message).call().content();

Q5. How do you prevent (reduce) hallucination?

Short answer: Hallucination (fluent but false output) can't be fully eliminated; you reduce it and contain it:

• Grounding: RAG with good retrieval; instructions to answer only from the provided context, and to say "I don't know" otherwise; citations to the sources.
• Use tools for facts: prices, availability and booking status come from API calls, never from the model's memory.
• Constrain the output: structured outputs (schemas), enumerations, low temperature for factual tasks.
• Verify:
  • validate the outputs against the source data (IDs exist; numbers match);
  • groundedness checks (a second model or a guardrail that checks that the claims are supported by the context);
  • rules that block unsupported commitments (refunds, discounts).
• Choose appropriate models, and evaluate them on your domain with golden datasets; monitor production samples.
• UX: show sources, make the AI's role clear, and have human review for high-stakes outputs.

Q6. How do you build an AI-based recommendation engine?

Short answer: Example: recommending hotels, room upgrades or add-ons.

• Classic, proven layers:
  • candidate generation: collaborative filtering (users who booked X also booked Y), content-based similarity (item embeddings from descriptions and attributes), popularity or trending, and rules (location, dates, availability);
  • ranking: a learning-to-rank model (gradient-boosted trees or a neural model) using user, item and context features (price sensitivity, past stays, device, season), optimised for bookings or revenue;
  • business rules and filters: availability, diversity, margin, fairness, and the exclusion of what's already booked.
• Where LLMs and embeddings help:
  • embeddings for semantic item similarity and cold start (a new hotel with no booking history);
  • understanding natural-language preferences ("quiet place near the beach for a family"), turning them into structured filters and embeddings;
  • generating explanations ("recommended because you liked…").
• The architecture:
  • features come from a feature store, and events (views, clicks, bookings) through Kafka;
  • batch training, and online serving with a low-latency candidate index (a vector store) plus the ranking service;
  • caching of recommendations per segment.
• Evaluation: offline (precision@k, NDCG), then online A/B tests (conversion, revenue per visitor); watch feedback loops and filter bubbles; respect privacy and consent.

Follow-up questions this topic invites — and their answers

Q: How do you choose a chunk size? A: Big enough to hold a complete idea, small enough to be specific (a few hundred tokens is common), split on document structure (headings, paragraphs), with some overlap. Validate the choice with retrieval metrics on real questions.

Q: What is hybrid search? A: Combining keyword (BM25) and vector similarity scores, often with reciprocal rank fusion. Keywords catch exact terms (codes, names); vectors catch meaning and paraphrases. Together they beat either alone.

Q: What is reranking? A: A second, more accurate (and slower) model, such as a cross-encoder, re-scores the top candidates from retrieval against the query, improving the precision of the few chunks put into the prompt.

Q: RAG or fine-tuning? A: RAG for knowledge that changes or must be cited (policies, catalogues). Fine-tuning for style, format or narrow task behaviour. They can be combined. RAG is usually the first choice because it's cheaper, fresher and more controllable.