Call GPT-4 from Java — chat completions, streaming, function calling, error handling.
Published September 21, 2026
Calling a large language model from a backend is, at its core, an HTTP request: you send a list of messages (instructions and the user's input) to a model endpoint, and get back generated text. Everything that makes it production-ready sits around that call: keeping the key secret, controlling cost and latency, streaming, getting structured output your code can trust, letting the model call your functions, and handling the failures that remote AI services have.
This lesson uses the OpenAI API and Spring AI (Spring's portable abstraction over OpenAI, Anthropic, Azure, Ollama and others). The same concepts apply to any provider.
POST https://api.openai.com/v1/chat/completions
Authorization: Bearer $OPENAI_API_KEY
Content-Type: application/json
{
"model": "gpt-4o-mini",
"messages": [
{ "role": "system", "content": "You summarize support tickets in one sentence." },
{ "role": "user", "content": "Customer says checkout fails with error 502 on the payment step since morning." }
],
"temperature": 0.2,
"max_tokens": 150
}
system sets behaviour and rules, user is the input, and assistant holds earlier model replies, which you resend to continue a conversation. The API is stateless, so "chat memory" means your app resends the relevant history each time.temperature: low (0–0.3) for consistent, factual outputs such as extraction and classification; higher for creative variety.max_tokens: caps the output length, and therefore cost and latency.<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
spring:
ai:
openai:
api-key: ${OPENAI_API_KEY} # from the environment / a secrets manager — never commit it
chat:
options:
model: gpt-4o-mini
temperature: 0.2
(Artifact names and some APIs changed across Spring AI milestone releases. Match the snippets to the version you use.)
@Service
public class TicketAssistant {
private final ChatClient chat;
public TicketAssistant(ChatClient.Builder builder) {
this.chat = builder.defaultSystem("You are a concise support assistant.").build();
}
public String summarize(String ticketText) {
return chat.prompt()
.user(u -> u.text("Summarize this ticket in one sentence:\n{ticket}").param("ticket", ticketText))
.call()
.content();
}
}
Most backend features need data: a category, a priority, extracted fields. Ask for JSON that matches a schema, and map it to a Java type:
public record TicketTriage(Category category, Priority priority, String summary, List<String> affectedComponents) {}
TicketTriage triage = chat.prompt()
.user("Classify this support ticket:\n" + ticketText)
.call()
.entity(TicketTriage.class); // Spring AI adds format instructions and parses the JSON
Providers also offer a strict JSON-schema response format that guarantees syntactically valid JSON matching your schema. Even so, validate the result (enum values, required fields, lengths), because the content can still be wrong. Treat model output like any untrusted input.
A long answer can take many seconds to generate. Streaming sends tokens as they're produced, so users see text appearing immediately:
@GetMapping(value = "/assist", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> assist(@RequestParam String question) {
return chat.prompt().user(question).stream().content(); // Server-Sent Events to the browser
}
Streaming improves perceived latency, not total time or cost. It's ideal for chat UIs, and pointless for background jobs that need the whole result anyway.
With tool calling (function calling), you describe functions to the model. When it needs data it doesn't have ("what's the status of order 1234?"), it replies with a request to call a function with specific arguments. Your code runs the function and sends the result back, and the model uses it to write the final answer.
@Component
public class OrderTools {
private final OrderService orders;
OrderTools(OrderService orders) { this.orders = orders; }
@Tool(description = "Get the current status and delivery estimate of an order by its ID")
public OrderStatus orderStatus(@ToolParam(description = "The order ID, e.g. 1234") String orderId) {
return orders.statusFor(orderId); // runs in your app, with your auth and validation
}
}
String answer = chat.prompt()
.user("Where is my order 1234?")
.tools(orderTools)
.call()
.content();
The model never executes anything itself. It only asks, so authorize every tool call as if it came from the user: check that this user owns order 1234. And never expose tools with side effects (refunds, deletions) without confirmation steps.
Errors and retries. Expect 429 (rate or token limits), 5xx and timeouts. Retry them with exponential backoff and jitter, honouring any Retry-After header. Don't retry 400 errors (bad request, context too long); fix the input. Set explicit timeouts, because default HTTP timeouts are often far too long for a user-facing request.
Cost control. Cost = input tokens + output tokens × price per token, and prices differ widely between models. Use the smallest model that meets quality, cap max_tokens, trim conversation history, cache responses for repeated identical requests, and log token usage per feature (Spring AI returns usage metadata) so you can see where the money goes.
Latency. Time grows with output length. Keep outputs short, stream where users wait, and move non-interactive work (bulk summarization, tagging) to background jobs, or use the provider's discounted batch API.
Security and privacy.
Observability. Log prompts (redacted), model, token counts, latency and errors. Keep a small evaluation set of inputs with expected outputs, and re-run it when you change prompts or models. Model upgrades change behaviour.
Q: How do you keep conversation context if the API is stateless? A: Store the conversation (or a summary of it) on your side and resend the relevant messages each turn. Trim or summarize older turns to stay within the context window and control cost. Spring AI's chat-memory advisors automate this.
Q: How do you make the output reliably machine-readable?
A: Use structured output (JSON-schema response formats, or entity(MyRecord.class) in Spring AI), keep temperature low, and validate the parsed object against your own rules. If validation fails, retry once with the validation error included in the prompt, then fall back gracefully.
Q: How does function calling decide which function to call? A: The model sees each tool's name, description and parameter schema, and decides from the conversation whether a call would help and with which arguments. Clear, specific descriptions matter a lot. Your code performs the call and returns the result, so authorization, validation and side-effect safety are your responsibility.
Q: What should happen when the AI provider is down? A: Degrade gracefully. Show a clear message, hide or disable AI-only features, or fall back to a secondary provider or model. A circuit breaker stops you from hammering a failing API. Core product functions shouldn't depend on an AI call succeeding.
Q: How do you reduce hallucinations in answers about your own data? A: Don't rely on the model's memory. Provide the facts in the prompt (Retrieval-Augmented Generation), tell the model to answer only from the provided context and to say when it doesn't know, and ask for citations. See the RAG lesson.