Building Resilient Kafka Producers and Consumers with Spring Boot 3

Apache Kafka is the enterprise backbone for asynchronous event-driven streaming. Building a production-grade integration in Spring Boot requires configuring proper delivery guarantees, dead letter topics (DLT), and non-blocking retry mechanisms.

Modern Asynchronous Producer with CompletableFuture (Spring Boot 3)

In Spring Boot 3 / Spring Kafka 3.x, legacy ListenableFuture has been replaced by Java's native CompletableFuture. Always register non-blocking callbacks to handle broker acknowledgments:

package com.astralsurge.kafka.service;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Service;

import java.util.concurrent.CompletableFuture;

@Service
public class OrderEventProducer {

    private static final Logger log = LoggerFactory.getLogger(OrderEventProducer.class);
    private final KafkaTemplate<String, String> kafkaTemplate;

    public OrderEventProducer(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrderEvent(String orderId, String payload) {
        CompletableFuture<SendResult<String, String>> future = 
            kafkaTemplate.send("orders.v1", orderId, payload);

        future.whenComplete((result, ex) -> {
            if (ex == null) {
                log.info("Message successfully written to topic: {} | partition: {} | offset: {}",
                    result.getRecordMetadata().topic(),
                    result.getRecordMetadata().partition(),
                    result.getRecordMetadata().offset()
                );
            } else {
                log.error("Failed to publish message for orderId [{}]: {}", orderId, ex.getMessage(), ex);
            }
        });
    }
}

Resilient Consumer with Non-Blocking Dead Letter Queue (DLT)

To prevent transient errors from blocking partition consumption, configure a DefaultErrorHandler with exponential backoff and a Dead Letter Publishing Recoverer:

package com.astralsurge.kafka.config;

import org.apache.kafka.common.TopicPartition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaOperations;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOff;

@Configuration
public class KafkaErrorConfig {

    private static final Logger log = LoggerFactory.getLogger(KafkaErrorConfig.class);

    @Bean
    public DefaultErrorHandler errorHandler(KafkaOperations<Object, Object> template) {
        // Automatically route failed records after max retries to .DLT topic
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(
            template,
            (record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition())
        );

        // Exponential backoff: Initial 1s delay, multiplier 2.0, max 3 attempts
        ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
        backOff.setMaxAttempts(3);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
        handler.setLogLevel(DefaultErrorHandler.DEFAULT_LOG_LEVEL);
        return handler;
    }
}