Kafka Security, Connect & Core Scenarios — Interview Questions

How to use this lesson

Security and Connect questions check your operational maturity. The scenario questions (lag, ordering, exactly-once, schemas) come up in almost every Kafka interview. Answer them by naming the exact configuration, then the caveat.

Q1. How do you secure a Kafka cluster?

Short answer: Defence in layers:

1. Encryption in transit: TLS on client and inter-broker listeners.
2. Authentication: SASL (SCRAM-SHA-512, OAUTHBEARER/OIDC, or Kerberos/GSSAPI), or mTLS client certificates.
3. Authorisation: ACLs per principal (topic read/write, consumer-group access, cluster operations), with allow.everyone.if.no.acl.found=false.
4. Network isolation: private subnets, security groups, no public brokers.
5. Encryption at rest: disk or volume encryption. Kafka has no built-in encryption at rest; add application-level payload encryption for highly sensitive fields.
6. Quotas, to stop noisy or abusive clients.
7. Audit logging and monitoring of authentication failures and ACL denials.

Key points to cover:

• KRaft controllers need securing too, as the ZooKeeper ensemble did in older clusters.

Q2. What security mechanisms does Kafka provide?

Short answer:

Q3. How would you implement encryption in Kafka?

Short answer:

• In transit:
  1. Issue certificates, ideally from an internal CA (cert-manager or Vault PKI).
  2. Configure the broker listeners (listeners=SSL://… or SASL_SSL://…), with keystore and truststore settings.
  3. Configure clients with security.protocol=SSL or SASL_SSL and a truststore.
  4. Use ssl.endpoint.identification.algorithm=https to verify hostnames.
• At rest: encrypted disks or volumes.
• End to end: encrypt sensitive payload fields in the producer (for example, envelope encryption with a KMS), so even broker admins can't read them.

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="orders-svc" password="${ORDERS_SVC_KAFKA_PASSWORD}";
ssl.truststore.location=/etc/kafka/truststore.p12

Key points to cover:

• TLS adds CPU cost, and disables zero-copy transfer for encrypted listeners. Size brokers for it.

Q4. What are the best practices for securing Kafka at scale?

Short answer:

• One principal per application, with least-privilege ACLs managed as code (Terraform or GitOps).
• Prefixed ACLs per team namespace (orders.*).
• Short-lived credentials: OAuth or SCRAM with rotation.
• Automated certificate rotation.
• Quotas per client.
• Schema Registry secured too, with validation of what's produced.
• Separate clusters or tenants for very different sensitivity levels.
• Continuous monitoring and auditing.
• Patching, and prompt Kafka upgrades.

Q5. Discuss Kafka Connect.

Short answer: Kafka Connect is Kafka's integration framework for moving data into Kafka (source connectors: databases through Debezium CDC, files, SaaS APIs) and out of Kafka (sink connectors: Elasticsearch, S3, data warehouses, JDBC), without writing custom producer or consumer code. Connectors are configured through a REST API. The runtime handles scaling, offset tracking, fault tolerance, converters (JSON, Avro, Protobuf) and single message transforms (SMTs).

Learn it in depth → Batch Processing System

Q6. What is Kafka Connect, and why is it useful?

Short answer: It saves you from writing and operating fragile, bespoke integration code. You get reusable, battle-tested connectors, exactly-once support for sources that allow it (and many sinks), automatic offset management, distributed scaling, dead-letter queues for bad records, and standardised operations (REST, metrics). A classic use is CDC with Debezium: streaming every database change to Kafka, reliably and in order, to feed search indexes, caches and analytics.

Q7. How do you scale Kafka Connect?

Short answer:

• Run Connect in distributed mode, as a cluster of workers sharing a group ID. Connectors are split into tasks (tasks.max), and the tasks are balanced across the workers.
• Scale by adding workers and raising tasks.max, up to what the connector supports. For sinks, that's usually the partition count of the input topics. For JDBC sources, it's the number of tables or queries.
• Tune the batch sizes, converters and producer or consumer overrides per connector.
• Run connectors on Kubernetes (the Strimzi operator), with autoscaling.

Q8. What problems commonly occur with Kafka Connect?

Short answer:

• Misconfiguration: wrong converters (JSON with schemas vs Avro), missing permissions, bad SMTs.
• Poison records: configure errors.tolerance=all with a dead-letter queue topic, and log the context.
• Schema drift between the source and the sink.
• Rebalances when workers or connectors change. Incremental cooperative rebalancing helps.
• Connector-specific failures: Debezium replication-slot growth, JDBC sink primary-key conflicts.
• Offset or state issues after upgrades.
• Throughput bottlenecks at the external system, not at Kafka.

Key points to cover:

• Monitor task state through GET /connectors/{name}/status, JMX metrics and lag. Alert on FAILED tasks.

Q9. How do you make sure messages with the same key are processed in the order they were sent?

Short answer:

• Produce with the key (for example orderId), so all of a key's messages land in the same partition, where order is preserved.
• Keep producer ordering safe with idempotence enabled (the default), which preserves order even with retries and several in-flight requests.
• On the consumer side, process each partition sequentially. If you parallelise within a consumer, do it per key (for example, Confluent's Parallel Consumer with key ordering).
• Don't change the partition count of a keyed topic casually.

Q10. Consumers are lagging behind the producers. What do you do?

Short answer:

1. Measure. Is the lag growing on all partitions (capacity) or on a few (hot partitions, or a stuck consumer)? Is processing slow (database or downstream calls) or is fetching slow?
2. Scale consumers, up to the partition count. If the partition count is the limit, add partitions carefully (it affects key ordering), or process in parallel within a consumer, per key.
3. Speed up processing: batch the database writes, remove synchronous remote calls or make them asynchronous, and fix slow queries.
4. Tune fetching: max.poll.records, fetch.min.bytes, fetch.max.wait.ms. Keep max.poll.interval.ms above the batch-processing time, to avoid rebalance loops.
5. Watch for rebalance storms and poison messages that cause endless retries.

Common trap: only "add more consumers". Beyond the partition count, extra consumers sit idle.

Q11. Your application needs exactly-once processing. How do you configure Kafka?

Short answer: Kafka's exactly-once covers read → process → write within Kafka:

• Producer: enable.idempotence=true, plus a transactional.id.
• Consume, process and produce inside a transaction, sending the consumed offsets within the same transaction (sendOffsetsToTransaction).
• Downstream consumers use isolation.level=read_committed, so they never see aborted writes.
• In Kafka Streams, just set processing.guarantee=exactly_once_v2.

producer.initTransactions();
while (true) {
    var records = consumer.poll(Duration.ofMillis(200));
    producer.beginTransaction();
    for (var r : records) producer.send(new ProducerRecord<>("orders.enriched", r.key(), enrich(r.value())));
    producer.sendOffsetsToTransaction(currentOffsets(records), consumer.groupMetadata());
    producer.commitTransaction();                    // outputs and offsets commit atomically
}

Key points to cover:

• External side effects (a database write, an email, a payment API call) are not covered by Kafka transactions. There, you need idempotent processing: deduplicate by event ID in the same database transaction, or use upserts. That gives you "effectively once".

Q12. How do you evolve a topic's message schema without breaking existing consumers?

Short answer:

• Use a schema format with evolution rules (Avro, Protobuf or JSON Schema), and a Schema Registry (Confluent or Apicurio).
• Set a compatibility mode per subject, typically BACKWARD (new consumers can read old data), or FULL.
• Make only compatible changes: add optional fields with defaults, don't remove required fields, don't change types. The registry rejects incompatible schemas at registration or build time.
• Roll out consumers first, when needed.
• For breaking changes, create a new topic version (orders.v2), and dual-publish during the migration.

Key points to cover:

• Check compatibility in CI (the registry's Maven plugin), not in production.

Follow-up questions this topic invites — and their answers

Q: What's the difference between SASL/PLAIN and SASL/SCRAM? A: PLAIN sends the password itself (so it must be over TLS), and brokers verify it against a static configuration. SCRAM uses a salted challenge-response. Credentials are stored hashed in Kafka's metadata, and they can be rotated without restarting brokers.

Q: What is Debezium? A: A set of Kafka Connect source connectors that do change data capture. They read the database's transaction log (MySQL binlog, PostgreSQL WAL), and emit every insert, update and delete as an ordered event. It's commonly used to implement the outbox pattern.

Q: What does read_committed do for consumers? A: It makes consumers skip records from aborted transactions, and not see records from open transactions until they commit. The consumer only reads up to the "last stable offset".

Q: How do you handle a poison message that always fails processing? A: Retry a few times with backoff, then publish it to a dead-letter topic (Spring Kafka's DeadLetterPublishingRecoverer), with error headers, commit the offset, and alert. Don't block the partition forever.