apache spark - How to create a custom streaming data source?

Question

I have a custom reader for Spark Streaming that reads data from WebSocket. I'm going to try Spark Structured Streaming.

How to create a streaming data source in Spark Structured Streaming?

Answer 1

As Spark is moving to the V2 API, you now have to implement DataSourceV2, MicroBatchReadSupport, and DataSourceRegister.

This will involve creating your own implementation of Offset, MicroBatchReader, DataReader<Row>, and DataReaderFactory<Row>.

There are some examples of custom structured streaming examples online (in Scala) which were helpful to me in writing mine.

Once you've implemented your custom source, you can follow Jacek Laskowski's answer in registering the source.

Also, depending on the encoding of messages you'll receive from the socket, you may be able to just use the default socket source and use a custom map function to parse the information into whatever Beans you'll be using. Although do note that Spark says that the default socket streaming source shouldn't be used in production!

Hope this helps!