Hello ποΈ !
Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed π¦ all of that in a new home-made tool named producer-file
. Here are the features :
- π¨ Sending line by line
- πͺ CSV to JSON conversion
- π¦ Support message headers by file and/or by line
- β‘οΈ Delta-changes mode (only send changes occurred after last send)
βοΈ More about the shell way :
β¨οΈ Pipe xlsx files into/from Kafka... From cli with (k)cat π
adriens for opt-nc γ» Oct 10 '22 γ» 2 min read
Motivations
Using jq
+ kcat
with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities π€·π»ββοΈ.
π½οΈ Demo
Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :