One Star

Surrogate key generation in talend

Hi everyone,
I want to load a table from db2 to hive and generate a new surrogate key from Talend
ex. say i have two records in the table CUSTOMER
 CUST_ID name
100    k
200    kk
1       100  k
2        200  kk
I am able to achieve this by the using numeric.sequence("S1",1,1) in tmap expression.
However, if i want to load deltas, say this table contains two more records in the future, when i run the talend job again
I want surrogate keys to be 3 and 4 and not 1 and 2 again.
Hope my requirement is clear.
Please help.
Five Stars

Re: Surrogate key generation in talend

You can do this using numeric.sequence, you just need to know what the starting number should be.
You can get this by either counting the rows in the table, or recording the last-used number.
Alternatively, you can externalize your sequence generation using a database, for example.
One Star

Re: Surrogate key generation in talend

Thanks for your reply!
Can you please elaborate on thie? How can I record the last used number?