TDM multiple CSV inputs

Highlighted
Four Stars

TDM multiple CSV inputs

Hello,

 

I recently tried to generate a complex structured format (GS1 format in JSON or XML) from a database. The fact the output format is quite complex and pre-exist on TDM samples data drive me to use the data mapper but there is no supported connectors for this database (informix). In the following descripion i'm using 6.2.1 version of Talend but had tried to do it in 7.2.1 without success.

 

To create this complex structure I need to make multiple lookups from the database as it concern data from several entities.

So I though that I could generate flat files (CSV) from informix and then map multiples CSV files to the complex JSON, using the ReadURL function that is described here : https://help.talend.com/reader/njNNGkCowThcKR6notunrw/Q7rNzUuyhHhMMfjNxPvvXA

 

To begin with, I tried with simple structures based on these files :

file1 (first input)

column1;column2;column3
1;"example here";500
2;"second example";834.12

file2 (second input)

column1;column1_ref
"Something";1
"Something else";2
"Again";2

file3 (output example)

{
	"item_id": 2,
	"name": "second example",
	"price": 834.12,
	"tags": [
		"Something else",
		"Again"
	]
}

 

In the TDM, I created a structure that include the file1 and file2 structures which look like this :

multi-input-noheader2_structure.PNGmulti-input structure

And here the problem begin, when I try to "Show Sample" this structure here is what I got (XML Representation):

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010
2;</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>second example</Flat:id>
      <Flat:name>
        <Flat:file2>
          <Flat:row>
            <Flat:description>Something</Flat:description>
            <Flat:item_id>1</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Something else</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
          <Flat:row>
            <Flat:description>Again</Flat:description>
            <Flat:item_id>2</Flat:item_id>
          </Flat:row>
        </Flat:file2>
      </Flat:name>
    </Flat:row>
  </Flat:file1>
</Flat:ROOT>

instead of :

<Flat:ROOT>
  <Flat:file1>
    <Flat:row>
      <Flat:id>1</Flat:id>
      <Flat:name>example here</Flat:name>
      <Flat:price>50010</Flat:price>
    </Flat:row>
    <Flat:row>
      <Flat:id>2</Flat:id>
      <Flat:name>second example</Flat:name>
      <Flat:price>834.12</Flat:price>
    </Flat:row>
  </Flat:file1>
  <Flat:file2>
    <Flat:row>
      <Flat:description>Something</Flat:description>
      <Flat:item_id>1</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Something else</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
    <Flat:row>
      <Flat:description>Again</Flat:description>
      <Flat:item_id>2</Flat:item_id>
    </Flat:row>
  </Flat:file2>
</Flat:ROOT>

 

As well as an error, due to column shift : 

Inkedmulti-input-noheader2_structure-show-sample-error_LI.jpgerror on multi-input structure

Can somebody help me resolving this issue ? It seems that the "price" column which is a float is taking the line terminator (LF) and the next row "id".

Notice that this is not happening on the file1 structure : 

file1-noheader2-structure-show-sample.PNGfile1 structure

 


Accepted Solutions
Moderator

Re: TDM multiple CSV inputs

Hello @CSen 

Here is a comment from our TDM expert in your jira issue:https://jira.talendforge.org/browse/TDM-7510.

Could you please take a look at it? Feel free to add your comment in it.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

View solution in original post


All Replies
Moderator

Re: TDM multiple CSV inputs

Hello,

Could you please file a jira issue and attach your job? Our TDM experts are going to investigate this issue by checking your job and TDM resources.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: TDM multiple CSV inputs

Hello,

I created a JIRA issue : https://jira.talendforge.org/browse/TDM-7510
And published example of TDM resources needed to reproduce.

Best regards,
Moderator

Re: TDM multiple CSV inputs

Hello @CSen 

Here is a comment from our TDM expert in your jira issue:https://jira.talendforge.org/browse/TDM-7510.

Could you please take a look at it? Feel free to add your comment in it.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

View solution in original post

Four Stars

Re: TDM multiple CSV inputs

Hello,

Yes I did, it's working fine with the solution provided.
Just by clearing the "multi-input" structure representations (that was set to Flat) and set all sub-structures to Flat as they were supposed to be.

Thanks for the quick reply,

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog