Six Stars

TDM Mapper - need to input multiple Json streams

I have a TDI job that calls an tHMap, and passes into it for input; a JsonDataStream; and generates output in Xml.

The tHMap works fine.

I have a new requirement, where I need to add an additional 1 or possible 2 different Json lookup streams; to be used with the main JsonDataStream; in doing some mapping for the Xml output.

It seems that tHMap only allows 1 input.
I cannot use tXmlMap; because my output Xml structure is highly complex xml with many child of child group and loops; of which tXmlMap doesn't support.

What is the best practice for having additional lookup Json streams along with my main data Json stream into my tHMap?

Should I combine the lookup jsons 'into' my main data Json stream; before loading the tHMap?

Just looking for the best practice for how to do this.  Haven't found any documentation on merging or combining different json files into one; or reading lookup files from within a Mapper.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Eight Stars

Re: TDM Mapper - need to input multiple Json streams

Or you can merge your inputs before calling tHmap (In such case remove data in I/O tab at map level).

 

Here is an example:

Capture.PNG

 

The tReplace remove feed lines and header in XML files

As I read files with tFileInputRaw, input is object type. I convert them into string with tConvertType

tsetGlovalVar store the two input in global variables

In tJavaRow, I merge the to files before calling tHmap (Adding a root element)

output_row.FILEMERGED = "<ROOT>"+((String)globalMap.get("FILE1"))
+ ((String)globalMap.get("FILE2"))+"</ROOT>";

 

If your inputs are Json and not files but stream, you will have to adjust slightly this process.

7 REPLIES
Eight Stars

Re: TDM Mapper - need to input multiple Json streams

Hi,

 

Indeed there is currently a feature request for this requirement:
https://jira.talendforge.org/browse/TDM-5377

 

Nontheless, I can suggest a workaround:
-You can create your structures for all your input files (In my example, I have two input files):

STRUCT1.png

and

STRUCT2.png
-Then in a map you can combine your structures. At root level, you have to make sure you have no I/O function

MAP.png
-For each of your input structure, you assign a READURL function to read you input file

READ1.png
-In your job, your tHmap will have no input flow

 

The weakness of this approach is that you will have to write down your input files on the file system.

 

Let me know if it meets your requirements or you need clarifications.

 

Eric

Six Stars

Re: TDM Mapper - need to input multiple Json streams

Eric,
Thanks for the reply and the suggested work-around.

 

In my usecase, I am not reading my Json inputs from file; they are going to be served up via JSP-APIs, and will be Json Streams.

I haven't used the 'ReadUrl' function; do you know if it is able to also read from a stream as opposed to a filesystem input?


If not, is there an alternative to join Json streams?

 

My other thought I am going to try; is inputting the Jsons into a tMap/or tXmlMap component, (that can receive multiple inputs), and attempt to map them to a single json outfrom from that map component....just haven't gotten it wired up to see if that approach will work.

 

Eight Stars

Re: TDM Mapper - need to input multiple Json streams

No ReadURL only enable reading fils from a file system.

 

Eric

Six Stars

Re: TDM Mapper - need to input multiple Json streams

Thanks Eric.


For now I am trying your suggested approach above.


I have two input Json files.

I have made a structure for each(POs and Orders); and a third combined structure(Multi), that contains them both.

 

I am missing something from your 3 and 4 example of the Multi-Structure.

 

When you say 'at the root level, make sure you have no I/O function.

(my root Node's IO/Database tab is empty)

 

However, when I select either of my sub elements in the structure under the root (POs or Orders); neither of them have an IO/Database tab to place a ReadURL expression object into.

 

Only the Root element of the structure has the 'IO/Database' tab available.
I cannot find any settings to disable it there; or to enable that tab on the POs or Orders Elements.

 

I can see in your last screencap, you do have the POs node under Root selected; and it "is" showing to have a IO/Database tab for you to venue it's ReadURL expression object into.

 

To restate what I did; following your example:
I have:
Structure1: POs
Structure2: Orders

Structure3: Multi (with Root and child siblings of POs and Orders underneath)

 

I feel like I am missing it where you say 'in a map you can combine your structures'.
When I open a new map; it will only allow me to have 1 input structure...Can't see how to combine both into the map.
I can use the Multi Structure; but when I use it as the input structure to my map; the child (POs or Orders) elements don't have the IO/Database tab available on them.

 

What am I missing?

 

Here are my screenshots:

Root.JPG

POs.JPG

Eight Stars

Re: TDM Mapper - need to input multiple Json streams

Hi,

 

In your "Multi" input structure (The one that combine the individual structures) do you inherit from each structure ?

 

You should have something like:Capture.PNG

 

Eric

 

Eight Stars

Re: TDM Mapper - need to input multiple Json streams

In my last reply, I uploaded my solution...

Eight Stars

Re: TDM Mapper - need to input multiple Json streams

Or you can merge your inputs before calling tHmap (In such case remove data in I/O tab at map level).

 

Here is an example:

Capture.PNG

 

The tReplace remove feed lines and header in XML files

As I read files with tFileInputRaw, input is object type. I convert them into string with tConvertType

tsetGlovalVar store the two input in global variables

In tJavaRow, I merge the to files before calling tHmap (Adding a root element)

output_row.FILEMERGED = "<ROOT>"+((String)globalMap.get("FILE1"))
+ ((String)globalMap.get("FILE2"))+"</ROOT>";

 

If your inputs are Json and not files but stream, you will have to adjust slightly this process.