Is it Possible to send a Java Object to C# App using Kafka

Question

Is it possible to send send a Java Object (lets say user) to a topic that is consumed and serialised to a User Object in C#?

Lets say I have the following avro schema built from a a Java PoJo (Fields are Name and Age)

{
  "namespace": "io.confluent.developer",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "age",
      "type": [
        "null",
        "int"
      ],
      "default": null
    }
  ]
}

Which generates a User.class

Then sending it like so:

Service
@CommonsLog(topic = "Producer Logger")
@RequiredArgsConstructor
public class Producer {

  @Value("${topic.name}")
  private String TOPIC;

  private final KafkaTemplate<String, User> kafkaTemplate;

  void sendMessage(User user) {
    this.kafkaTemplate.send(this.TOPIC, user.getName(), user);
    log.info(String.format("Produced user -> %s", user));
  }
}

I also have a Schema registry BUT I do not know how to consume the message in C# and deserialise it to a User class with the same fields:

public class Users

{

    public int id = 0;

    public string name = string.Empty;

    public Users()

    {

        // Constructor Statements

    }

    public void GetUserDetails(int uid, string uname)

    {

        id = uid;

        uname = name;

        Console.WriteLine("Id: {0}, Name: {1}", id, name);

    }

    public int Designation { get; set; }

    public string Location { get; set; }

}

Thanks for the help.

question from:https://stackoverflow.com/questions/65847510/is-it-possible-to-send-a-java-object-to-c-sharp-app-using-kafka

Answer 1

Yes it's possible. You can use official .NET kafka client to consume messages.

First thing you have to do is generate C# class based on the same schema you used. You can do that by:

1. Installing avrogen tool: dotnet tool install --global Apache.Avro.Tools
2. Generate the class: avrogen -s user_schema.avsc .

Then you will get User.cs with the class implementation. All you need to do is configure .NET Kafka client and consume the messages:

var schemaRegistryConfig = new SchemaRegistryConfig
{
    Url = "schemaRegistryUrl"
};

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "bootstrapServers",
    GroupId = "group"
};

using var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);
using var consumer = new ConsumerBuilder<string, User>(consumerConfig)
    .SetValueDeserializer(new AvroDeserializer<User>(schemaRegistry).AsSyncOverAsync())
    .Build();

consumer.Subscribe(topicName);

var consumeResult = consumer.Consume(cts.Token);

You can have a look at this example for more info.

Notice, that you can't use the User class you provided in your question, because there are some requirements for the class structure. So you should use the one generated with the tool from your Avro schema.