Create a historical sequence in talend big data jobs

Seven Stars

Create a historical sequence in talend big data jobs

Hi ,

 

I have a requirement to create sequence in talend.

Basically records are coming from a source file.

for each source row i want to create a unique number.

here is where it gets complicated.

when a new file comes next day , the talend should pick the last generated number and then increment it with 1.

for EX:

today the last generated sequence number is 100.

tomorrow the talend should generate sequence number from 100 . i.e. 101,102,103,104.....

This means talend should keep the history of previously generated last sequence number.

 

Thanks

Fifteen Stars TRF
Fifteen Stars

Re: Create a historical sequence in talend big data jobs

So, in such a case you have to persist this last sequence value somewhere, in the target database (if any) or in a dedicated file.

If the records are stored in a database, you can also get the max value from the corresponding field using the appropriate Select.
When you got the desired value, you need to store it in a global variable, then reuse this variable to initialize the sequence with something like:

Numeric.sequence("yourSequence", (Integer)globalMap.get("yourGlobal"), 1)

Hope this helps.


TRF
Ten Stars

Re: Create a historical sequence in talend big data jobs

Caveat: Either store the "next" value or increment the max key when you retrieve it so you start the sequence on the next number.

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now