As a developer I want create an appication to demonstrate consuming data (simple equity stock buy/sell and then biforcating the data into two topics one for by and one for sell using flink to demonstrate filtering in Flink.
Your task is to create a Kafka service into Aiven and write a piece of code that produces valid JSON data to a topic in that service.
The key should be a valid JSON string containing a random id, e.g. UUID, and the message payload should be a valid JSON object. The payload should be a mock "event" from an interesting use case, e.g. IoT sensor, stock tickets, or financial transactions. The event should include a timestamp represented by a string with the date in ISO 8601 format.
Once you have the data flowing into the kafka topic, integrate your Kafka service with Aiven Flink to filter the data into 2 new topics. You can use any attribute within your JSON payload (e.g. odd/even date seconds, a number modulo 2, etc.) to basically bifurcate your incoming data into two topics
- Part 1: The producer's data should be readable from the Aiven web console from the Kafka: service view > T opics => T opic => Fetch Messages (Format: json)
- Part 2: create 2 Flink jobs to split the data to show the power of Flink filtering
- Grafana enabled to monitor Kafka in default dashboard
- Project is available in github with README
Rewrite app to have a single job that scales to process buys and sells and output to the 2 topics.
● Aiven Docs Center - https://docs.aiven.io/ ● Kafka Quickstart - https://aiven.io/docs/products/kafka/get-started ● Flink Quickstart - https://aiven.io/docs/products/flink/get-started ● M3DB Quickstart - https://aiven.io/docs/products/m3db ● Grafana Quickstart - https://docs.aiven.io/docs/products/grafana/get-started