Six Stars

Create a historical sequence in talend big data jobs

Hi ,

 

I have a requirement to create sequence in talend.

Basically records are coming from a source file.

for each source row i want to create a unique number.

here is where it gets complicated.

when a new file comes next day , the talend should pick the last generated number and then increment it with 1.

for EX:

today the last generated sequence number is 100.

tomorrow the talend should generate sequence number from 100 . i.e. 101,102,103,104.....

This means talend should keep the history of previously generated last sequence number.

 

Thanks

  • Big Data
2 REPLIES
Nine Stars TRF
Nine Stars

Re: Create a historical sequence in talend big data jobs

So, in such a case you have to persist this last sequence value somewhere, in the target database (if any) or in a dedicated file.

If the records are stored in a database, you can also get the max value from the corresponding field using the appropriate Select.
When you got the desired value, you need to store it in a global variable, then reuse this variable to initialize the sequence with something like:

Numeric.sequence("yourSequence", (Integer)globalMap.get("yourGlobal"), 1)

Hope this helps.


TRF
Ten Stars

Re: Create a historical sequence in talend big data jobs

Caveat: Either store the "next" value or increment the max key when you retrieve it so you start the sequence on the next number.