Parsing and storing with commit and rollback components.

Highlighted
Five Stars

Parsing and storing with commit and rollback components.

Hello,

 

I am using positional parser. It iterates through the file and produces valid and invalid inputs. 

 

If any data passes the validation the job starts storing into database. But i want to wait till entire file has processed. If every line passes validation then only start saving to database and commit else revert back the changes.

 

Thanks,

Shashi

Fifteen Stars TRF
Fifteen Stars

Re: Parsing and storing with commit and rollback components.

Hi,

As a simple solution, replace the tPostgresqlOutput component by a tHashOutput or tFileOutputDelimited.

On subjob OK connect to your tFileInputDelimited, add a dummy tJava connected to a tHashInput (referencing the previous tHashOutput) with an "If" connector with the following condition:

(Integer)globalMap.get("tHashOutput_1_Nb_Line") != null &&
(Integer)globalMap.get("tHashOutput_1_Nb_Line") > 0

If you use a tFileOutputDelimited instead of a tHashOutput, replace the dummy tJava by a tFileRowCount.

At the end, depending on the result condition, get the input and go to the database (or not).

Hope this helps.


TRF
Five Stars

Re: Parsing and storing with commit and rollback components.

Hello TRF,

 

Instead of using file output , i am using using javaflex. In invalid output, i am setting context variable and checking it on "IF" condition. Now condition is working fine. But i am unable to send main data output to the SqlOutput. After if condition of javaflex i am getting only one row rather than entire parsed data.

 

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch