tWriteJSONField seems to cause a roadblock in the data flow

Five Stars

tWriteJSONField seems to cause a roadblock in the data flow

Hi there... new Talend developer here... I have a data flow (picture hopefully inserted below) where I am querying a table for some rows (3 in this case) and I want to process each one in turn. This was working as I would expect in an earlier version where the steps were: (1) parse the records into fields, (2) map several fields into a combined comma delimited message field, (3) send the message to a queue, (4) update the relevant database row with a processed date. 

 

However when I added a tWriteJSONField component the behavior has changed. Now all 3 rows are processed as before up until the tWriteJSONField component. At that component the flow seems to wait until all rows are processed before proceeding. Then after the tWriteJSONField component the remaining components are run 3 times but all 3 using the final rows values. This of course is no good. Is there somethign I am doing wrong in the setup of the tWriteJSONField component? Or can I not use this inside a flow? I'm confused!

 

Note that the diagram below is on step1 in debug mode. It shows that 3 rows have been processed but not beyond the tWriteJSONField component.

 

Untitled.png

 

 

 

 

 

Five Stars

Re: tWriteJSONField seems to cause a roadblock in the data flow

Me again... to clarify: what I want is for each of the 3 records to go all the way through the flow until the end before the next record is processed.

Highlighted
Nine Stars

Re: tWriteJSONField seems to cause a roadblock in the data flow

I have exactly the same problem and have spent a lot of time trying all sorts of workarounds to get the flow to pass each row through tWriteJSONField without queuing them up first.

 

As in the OP, the rows halt at tWriteJSONField, then the flow proceeds row by row after it.  This behavior messes up my setting global variables with the row values before the tWriteJSONField; the global variables always reflect the last row of data.

 

Why was the tWriteJSONField component designed this way?  Also, as a side effect of this, or as a different anti-pattern, the variables in the schema get nulled out, so you can't expect to persist schema variables through the tWriteJSONField.

 

Unless someone can explain to me the reason for this design and a workaround, I would consider this a bug.  Somebody developed a custom component called tRowToJSON probably because of the stock component's maladjusted behavior.

 

Thanks in advance for any help.

--
Please give Kudos and mark topics as solved where appropriate.
Tags (2)
Five Stars

Re: tWriteJSONField seems to cause a roadblock in the data flow

Oh, someone replied to my post! Smiley Happy

 

In the end I just worked around this by writing out the json formatting in a java component. But it's good to know there is a custom component for when this comes up again.

 

I also find it annoying that a lot of components do not flow the schema variables across them. There must be some design philosophy behind this that will become apparent to me one day.

Tags (1)
Five Stars

Re: tWriteJSONField seems to cause a roadblock in the data flow

Maybe if tWriteJSONField was renamed to tWriteJSONReport there would be less confusion? Smiley Happy

Nine Stars

Re: tWriteJSONField seems to cause a roadblock in the data flow

Hi @theronrohr , I figured out a workaround for this, see my response in my thread here:

https://community.talend.com/t5/Design-and-Development/tWriteJSONField-and-tWriteXMLField-queue-unex...

--
Please give Kudos and mark topics as solved where appropriate.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now