Storing rows in an in-memory table

One Star

Storing rows in an in-memory table

Hi everyone
Let's suppose I have a job where the entire contents of a table is recreated daily with data from a database which is not very available (frequent downtime and slow connection).
How can I erase the local table ONLY if reading all the remote data succeeds?
I want to avoid the situation that the local table is only partially populated if there are timeouts or disconnections during the map operation, which can take very long.
I thought of reading the remote table and putting the data into a temporary in-memory store, and only if this succeeds then proceeding with the map operation.
Is it possible?
thank you
Moderator

Re: Storing rows in an in-memory table

Hi,
Do you want to delete your table by timing? and trigger the subjob if the remote data succeeds? Could you elaborate your case with an example?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seventeen Stars

Re: Storing rows in an in-memory table

What about the good old transactions? I would do anything within a transaction which can only fail and does not leaf inconsistent data or succeed completely.
One Star

Re: Storing rows in an in-memory table

You are right jlolling, I forgot transactions!
Using transactions means that I must use the open/commit transaction components, and if something fails after open, it auto rollbacks, isn't it?
thanks
enrico
Seventeen Stars

Re: Storing rows in an in-memory table

Yes thats the normal way. To use transaction in Talend you have to use the connection component and the commit and rollback components. In your input and output component set the usage of external connection.
If your transaction will be growing to large, you have to build chunks of data to proceed.

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog