How to stream JSON with Kafka?

Four Stars

How to stream JSON with Kafka?


I'm trying to stream the JSON message in Kafka and save it as the .json file. However the output file is empty.

If somebody can help, I will be grateful.


I'm using TOS for Big Data 6.4

Kafka ver.:

Kafka ver. set in TOS:


I'm also looking for examples of solutions where Kafka is used to catch the message in JSON format and the output is in .json file.
I want to evaluate whether TOS is a good tool for reading JSON using Kafka.


Kafka producer and consumer input/output (example):



My simple jobs.





Forteen Stars

Re: How to stream JSON with Kafka?



if Output file empty, it mean - You wrong parse JSON 

You can attache screenshots for tExtractJSON and tWriteJSON, it would be better

Four Stars

Re: How to stream JSON with Kafka?



I've checked two options of the same job.


1. with Kafka input. No data in .jsonfile (file is empty) or no data i .txt file as an output (tFileOutputRaw component)

2. with tFixedFlowInput component. All the data and structure is in the .json file. I've checked also export to .xls file and it works as well.


In both cases I'm not using tWriteJSON component. Should I use it?


1. Kafka input job



tExtractJSON config





Output schema, columns 1:1



2. Job where I used tFixedFlowInput component






JSON here:

{"data":[{"Service_Description":"Pets Allowed","Service_Code":"PET"},{"Service_Description":"Swimming Pool","Service_Code":"SWI"},{"Service_Description":"Tennis Court","Service_Code":"TEN"},{"Service_Description":"Dry Cleaning","Service_Code":"DRY"},{"Service_Description":"Internet Access","Service_Code":"INT"},{"Service_Description":"WIFI Internet Access","Service_Code":"WIF"},{"Service_Description":"Fitness Room","Service_Code":"FIT"},{"Service_Description":"Concierge","Service_Code":"CON"}]}

output .json file



output .xlsx file



Please NOTE that tLogRow_2 displays the same output in both cases/jobs.

Forteen Stars

Re: How to stream JSON with Kafka?

I will check later with your sample
generally I use JSONPath for parse JSON

Forteen Stars

Re: How to stream JSON with Kafka?

I make few test, with Your schema.


My Jobs use little different and this is was not affected for me, but look like all depend from configuration of KafkaInput component


Kafka as any MQ oriented for non stop work, and depending how Your component setup, it open and close output file different.

If You use auto-disconnect by timeout or by number of received messages - all fine:

Screen Shot 2017-06-05 at 1.44.33 AM.png


if You manual stop Job - file not closed properly


as alternative possible use tFlowToIterate and route output in different JSON files, or append in same delimited, something like:


Screen Shot 2017-06-05 at 1.49.56 AM.pngScreen Shot 2017-06-05 at 1.50.06 AM.pngScreen Shot 2017-06-05 at 1.50.20 AM.png


in this case - each file contain single message from Kafka, all of them closed independent and could be processed after


What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.



Introduction to Talend Open Studio for Data Integration.


Downloads and Trials

Test drive Talend's enterprise products.