There are three ways in which messages can be sent by producers:

Sumit Rawal answered on May 31, 2023 Popularity 1/10 Helpfulness 1/10

answer There are three ways in which messages can be sent by producers:

There are three ways in which messages can be sent by producers:

Comment

Fire and forget: Message is sent to Kafka with no effort made to verify if the message was received successfully by the Kafka broker. In most instances, the messages will be received by Kafka, since the producer has retries built-in for failures. However, this method is still prone to losing messages when retries exhaust.

Synchronous: Message is sent and the future object returned has its get() method invoked to see if the Kafka broker received the message successfully.

Asynchronous: Messages are sent and a callback is specified, which is invoked when a response is received from the Kafka broker. In this method, the producer doesn’t wait to receive a response from Kafka before continuing to send other messages.

In the example from the previous lesson, we sent the messages synchronously, since the call to future.get() doesn’t return until the Kafka broker responds with success or failure. However, this approach doesn’t scale when we have millions of messages to be sent. If the network latency for a round trip from the producer to the Kafka cluster is 10ms, the time it takes to synchronously send millions of messages will be unbearable for any serious distributed system. The solution, then, is to send messages asynchronously and specify a callback that is invoked when a response from the Kafka cluster is received. Let’s rewrite our previous example as an asynchronous producer.

public class AsyncProducerExample {

    public static void main(String[] args) throws InterruptedException {
        // Set-up mandatory properties
        Properties kafkaProps = new Properties();
        kafkaProps.put("bootstrap.servers", "localhost:9092");
        kafkaProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        kafkaProps.put("value.serializer",
                "org.apache.kafka.common.serialization.StringSerializer");

        // Create an object of KafkaProducer and pass in the properties
        KafkaProducer producer = new KafkaProducer<String, String>(kafkaProps);

        // Create a record/message that we want to send to Kafka. Note, in this example we are
        // passing the topic, the key and the value. Recall, that the message key is optional.
        ProducerRecord<String, String> record = new ProducerRecord<>("datajek-topic", "my-key-async",
                "my-value-async");
        try {

            // We must pass an object of type that implements the interface
            // org.apache.kafka.clients.producer.Callback.
            producer.send(record, new ProducerCallback());
            System.out.println("Kafka message sent asynchronously.");

        } catch (Exception e) {
            System.out.println("Exception sending message asynchronously" + e.getMessage());
        } finally {
            // When you're finished producing records, you can flush the producer to ensure it has all been
            // written to Kafka and then close the producer to free its resources.
            producer.flush();
            producer.close();
        }

        // wait before exiting to hear from Kafka broker
        Thread.sleep(3000);        
    }

    private static class ProducerCallback implements Callback {
        @Override
        public void onCompletion(RecordMetadata recordMetadata, Exception e) {
            if (e != null) {
                e.printStackTrace();
            } else {
                // The RecordMetadata object contains the offset and partition for the message.
                System.out.println(String.format("Message written to partition %s with offset %s", recordMetadata.partition(),
                        recordMetadata.offset()));
            }
        }
    }
}

https://www.educative.io/courses/scalable-data-pipelines-kafka/7DPWqqp0g9w

Popularity 1/10 Helpfulness 1/10 Language java

Source: Grepper

Tags: java messages

Link to this answer
Share Copy Link

Contributed on May 31 2023

Sumit Rawal

0 Answers Avg Quality 2/10

There are three ways in which messages can be sent by producers:

Contents

More Related Answers

There are three ways in which messages can be sent by producers:

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.