LLM Integration in Java — Spring AI, Bedrock, Prompts, Streaming & Orchestration — Interview Questions

How to use this lesson

LLM questions in Java interviews are about engineering, not machine-learning theory. Interviewers want to hear that you treat the model as:

• an unreliable, slow, expensive external dependency, wrapped with the usual discipline: abstractions, timeouts, retries, fallbacks, observability, cost control;
• plus the AI-specific concerns: prompts as versioned artefacts, grounding (RAG), guardrails, and evaluation.

The AI chat backend design itself is covered in the Senior system-design chapter (search, notifications and chat).

Q1. What is AWS Bedrock, and how does it work? What is a "Bedrock-style" API abstraction?

Short answer:

• Amazon Bedrock is a managed service that gives API access to many foundation models (Anthropic Claude, Amazon Nova and Titan, Meta Llama, Mistral, Cohere, and others) without managing any infrastructure. You call a regional endpoint with AWS IAM authentication.
• Its main features:
  • a unified Converse API (the same request and response shape across models, including tool use and streaming);
  • Knowledge Bases (managed RAG: ingestion, chunking, embeddings, a vector store);
  • Agents (tool orchestration);
  • Guardrails (content filters, PII redaction, denied topics, grounding checks);
  • model evaluation;
  • provisioned throughput for guaranteed capacity;
  • private connectivity (VPC endpoints), and data that isn't used to train the models.
• Pricing is mainly per input and output token (on demand), or provisioned throughput.
• The abstraction idea: your application codes against one model-agnostic interface (chat(messages, options), embed(text), and streaming), and adapters map it to each provider. This lets you swap models by configuration, run A/B tests, route requests by cost or capability, and fail over between providers.
  • Bedrock's Converse API is one such abstraction at the provider level;
  • in Java, Spring AI (ChatClient, ChatModel, EmbeddingModel) and LangChain4j provide it at the application level.

Common trap: "switching models is just a config change". The abstraction makes switching possible, but prompts, tool-calling behaviour, context limits and output quality differ between models. Re-run your evaluation suite before switching.

Q2. How do you integrate an LLM with Spring Boot?

Short answer:

• Spring AI (1.x) has starters for OpenAI, Anthropic, Bedrock, Azure OpenAI, Vertex AI, Ollama and others:
  • a fluent ChatClient (system and user prompts, options);
  • structured output (mapping the response to a Java record);
  • tool calling (@Tool methods);
  • advisors (chat memory, RAG through QuestionAnswerAdvisor, logging);
  • vector store integrations (PGVector, Redis, Elasticsearch, Pinecone…);
  • observability through Micrometer.
• LangChain4j is an alternative, with AI Services (declarative interfaces), memory, RAG and tools, and a Spring Boot integration.
• Production concerns:
  • timeouts (LLM calls take seconds);
  • retries with backoff for 429 and 5xx responses;
  • circuit breakers;
  • streaming for user-facing chat;
  • virtual threads or reactive clients, because each call holds a thread for a long time;
  • API keys in a secrets manager;
  • per-user rate limits and token budgets;
  • logging of prompts and responses with PII redaction.

record BookingIntent(String intent, LocalDate checkIn, LocalDate checkOut, int guests) {}

@Service
class ConciergeService {
    private final ChatClient chat;
    ConciergeService(ChatClient.Builder builder, BookingTools tools) {
        this.chat = builder
            .defaultSystem("You are a hotel concierge. Answer only about the hotel. If unsure, say so.")
            .defaultTools(tools)                      // @Tool-annotated methods, run with the user's permissions
            .build();
    }
    BookingIntent parse(String userMessage) {
        return chat.prompt().user(userMessage).call().entity(BookingIntent.class);   // structured output
    }
}

Q3. What is prompt engineering? How do you version AI prompts?

Short answer:

• Prompt engineering is designing the model's instructions and context to get reliable, well-formatted, grounded outputs. The techniques:
  • a clear system prompt (role, scope, rules, tone, what to do when unsure);
  • explicit output formats (JSON schemas, or structured output features);
  • few-shot examples;
  • delimiting untrusted content (user input, retrieved documents) clearly from instructions;
  • step-by-step reasoning for complex tasks (or reasoning models);
  • grounding instructions ("answer only from the provided context; cite the sources");
  • keeping prompts concise, because tokens cost money and latency.
• Versioning prompts (prompts are code, and they drive behaviour):
  • store them in the repository or a prompt registry, as templates with variables, with a version ID; never scatter string literals across the code;
  • record the prompt version, model and parameters with every response (traceability);
  • run changes through review, and through an evaluation suite (golden question sets with expected properties, and automated scoring: exact checks, LLM-as-judge, groundedness) before release;
  • roll out gradually (feature flags, A/B tests), and allow quick rollback;
  • tie the prompts to the model version: a model upgrade is a prompt change too.

Q4. What is a streaming response? What are Server-Sent Events?

Short answer:

• LLMs generate token by token. Streaming sends the partial output as it's generated, so the user sees the first words within a few hundred milliseconds (a low time to first token), instead of waiting 5–20 seconds for the full answer. The perceived latency drops dramatically.
• Server-Sent Events (SSE) is a simple standard for server-to-client streaming over HTTP:
  • Content-Type: text/event-stream, and the connection is kept open;
  • the server writes data: …\n\n events;
  • the browser EventSource API reconnects automatically (with Last-Event-ID);
  • it works through most proxies, and over HTTP/2.
  • SSE is one-directional (the client sends requests with normal HTTP). WebSockets are bidirectional, and suit chat apps that need to push in both directions.
• In Spring:
  • WebFlux returning Flux<ServerSentEvent<String>>, or Flux<String> with produces = TEXT_EVENT_STREAM_VALUE;
  • Spring MVC with SseEmitter;
  • Spring AI's chatClient.prompt().user(q).stream().content() returns a Flux<String>.
• Operational details:
  • disable proxy buffering (for example, X-Accel-Buffering: no for Nginx);
  • set idle timeouts and heartbeats;
  • handle client disconnects by cancelling the upstream model call (saving tokens);
  • with streaming, output guardrails must work on chunks, or on the complete answer before critical actions.

@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
Flux<String> stream(@RequestParam String q) {
    return chatClient.prompt().user(q).stream().content();   // cancelled if the client disconnects
}

Q5. How do you implement AI workflow orchestration?

Short answer:

• Patterns (from simple to complex):
  • prompt chaining (step outputs feed the next step: classify → extract → answer);
  • routing (a classifier picks a specialised prompt or model);
  • parallelisation (several calls, then aggregation);
  • tool calling (the model asks the application to run functions);
  • agents (a loop of plan → tool call → observe, until done);
  • orchestrator-workers (one model splits the task among others).
• Prefer deterministic workflows (code decides the steps) where possible; use agentic loops only where flexibility is really needed, with step limits, timeouts and budget caps.
• Tool calling done safely:
  • tools are your code; the model only proposes calls;
  • validate the arguments; enforce the user's permissions;
  • make the tools idempotent;
  • require confirmation for side effects (payments, cancellations).
• Long-running or multi-step flows: a durable workflow engine (Temporal, or a state machine with persisted state) gives retries, timeouts and resumption after failures, and human-in-the-loop approval steps.
• Observability: trace each step (the prompt version, model, tokens, latency, tool calls) with OpenTelemetry, following the GenAI semantic conventions.
• Standards: the Model Context Protocol (MCP) standardises how tools and data sources are exposed to models (Spring AI supports MCP clients and servers).

Q6. How do you integrate AI with a rule engine?

Short answer: Combine their strengths:

• the LLM understands unstructured input (free-text requests, emails, reviews), and extracts structured facts (intent, entities, sentiment), returned as a validated JSON or Java record;
• the rule engine makes the decision, deterministically, auditably and reproducibly, over those facts (eligibility, pricing, refunds, escalation).

The flow: text → LLM extraction (with schema validation and a confidence score) → validation → rule engine → decision → (optionally) the LLM phrases the explanation to the user. Low-confidence extractions go to a human.

Why:

• regulated or money-related decisions need explainability and consistency, which LLMs don't guarantee;
• rules can be tested and audited;
• the LLM widens the input channel.
• The reverse also works: rules act as guardrails on LLM outputs (for example, never offer a discount above X%).

Q7. How do you design an AI fallback?

Short answer: Failure modes: provider outages, rate limiting (429), timeouts, context-length errors, content-filter refusals, low-quality or invalid output, and exhausted budgets.

A fallback chain:

1. Retry transient errors with backoff (within the user's latency budget).
2. Fail over to another model or provider (a secondary region or model), through the model abstraction. Its prompts must be tested on it.
3. Degrade: a smaller or cheaper model, a cached answer for common questions (semantic cache), or a non-AI path (search results, FAQ, forms).
4. Hand over to a human (support queue), keeping the conversation context.
5. An honest message to the user.

Also:

• circuit breakers per provider;
• validation of the output (schema, guardrails), with a repair retry, or a fallback if it's invalid;
• feature flags to switch off AI features quickly;
• monitoring of the fallback rates.

The principle: the core business flow (booking, payment) must never depend on the LLM being available.

Follow-up questions this topic invites — and their answers

Q: What is structured output, and why does it matter? A: Asking the model to return data matching a JSON schema (or a Java record, in Spring AI), often enforced by the provider. It makes outputs machine-usable and validatable, instead of parsing free text.

Q: Why do LLM calls need virtual threads or reactive clients? A: Each call can take seconds. With blocking platform threads, a few hundred concurrent chats would exhaust the thread pool. Virtual threads or non-blocking I/O let one instance hold thousands of in-flight calls cheaply.

Q: What's the difference between temperature and top-p? A: Both control randomness. Temperature scales the probability distribution (low = more deterministic); top-p samples only from the smallest set of tokens whose cumulative probability reaches p. Use low values for extraction and classification tasks.

Q: What is the Model Context Protocol (MCP)? A: An open protocol for exposing tools, resources and prompts to AI applications in a standard way, so one integration (for example, a booking-lookup tool server) can be reused by different models and clients.