Bug while generating a Surrogate key for Incremental Load.

Highlighted
Six Stars

Bug while generating a Surrogate key for Incremental Load.

Hi,

I have a job in which i am loading two tables, in those tables i am generating Surrogate id,

 

The process which I am generating this is, before running the job I am taking the max value of that column into a context and using this context for incrementing by 1 for each row.

Here the bug which I found is that, Suppose I am loading the 1st job after that i am using onsubjobOk to start the second job.

in this case the context value of 1st job itself is taking by 2nd job as well, instead of taking its own context value.

Can anyone help me how to avoid this situation.

 

Quick help is appreciated.

 

Regards

ShaX


Accepted Solutions
Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Issue is resolved

Thank you


All Replies
Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Could you please mark where you are updating the context variable in the attached screenshot? Component screenshots will be helpful to understand your flow better .

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

 

Thanks for the reply, PFA images below.

 

 

Capture.PNGThis is the jobCapture1.PNGHere am taking the max value of the column n storing it in the context.Capture2.PNGThat context am using to populate the column.

 

Please help me on this.

 

Regards

ShaX

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Since you are using assignment to context, could you please use tjavarow instead of tMap?

 

     Please also print the context variable before and after to confirm that the value is changing using System.out.println(context.variable_name)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

 

As told by you, instead of Tmap, i took Tjavarow to assign value to the context.

However this method also didn't work.

PFA screenshot for your reference.

picturemessage_eoh35sm0.lxj.png

Can you help me on this.

 

Regards

ShaX

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

   Could you please share the component screenshot along with print output?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi,

 

CaptureA.PNG

 

CaptureB.PNG 

 

This is how I am  loading the tables using contex.

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    That is fine. But where are you verifying the current value of the context variable?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

My issue is resolved,

what is did is, if  i am loading 10 tables in 1 job, earlier i used to use "S1" in all table in numeric sequence, but now am using from "S1" to "S10" for tables respectively.

 

I don't know weather this is the correct approach, but it is working fine as of now.

 

Regards

Shax.

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

 

Am facing same issue again after 2 days of job run.

Can you help me what can be done.

Here you can see am printing the value.

Capture2.PNG

The problem is coming while loading all tables in bulk.

Is there any other way to assign a context and use where i can avoid this issue.

Currently am assigning  "context.TEST = row1.KEY" and using "Numeric.sequence("s2",context.TEST,1) "

 

regards

ShaX.

 

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Any help on this.

 

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Could you please add screenshots of the latest job flow, your current sequence generation process and the stage where you are facing issue with some sample records? 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

PFA screenshots for your reference.

The job:

Capture.PNG

Assigning the value to the context in Tmap2 in above picture.

Capture1.PNG

Using Numaric.sequence and context for to load the table.

Capture2.PNGThe problem is not coming while loading single table.

it is coming wile loading all tables in bulk.

 

Regards

ShaX

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Basically, the error am getting is 

ORA-00001: unique constraint (XXXXXXXX.DW_DIM_LOAN_MASTER_PK) violated.

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Could you please show me how you are allocating context variables using tMap? Ideally you should use a tContextLoad or tjavarow to assign the values to context variables.

 

    Did you print and checked whether the assignment is correctly done?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

I don't have screenshot of the output i printed, but i shared the screenshot in the above conservation.

It works.

 

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,
Can you please mail me on this mail, so that i can share my job.
sharukh.talend@gmail.com
Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Unfortunately I will not be able to share my email id. If you are looking for a screen sharing session, I would recommend you to raise a support ticket and they will look into the issue.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

No worries @nikhilthampi ,

Can you tell me the process to raise the ticket.

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Am using Open studio,

Can I raise a ticket?

Employee

Re: Bug while generating a Surrogate key for Incremental Load.

Hi,

 

    Talend support tickets can be opened only for subscription products. Unfortunately I am not able to identify the issue just by screenshots. If you could attach the job after removing confidential elements, we can do some more analysis.

 

    Another method is to define a similar scenario with dummy values and you can share both expected and current output. Then we can fine tune this dummy job based on the details given.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

 

PFA zip file of the sample job.

 

Regards

ShaX

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Any help on this??

 

Regards

ShaX

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Hi @nikhilthampi ,

 

Any help on this please....

 

Regards

ShaX

Six Stars

Re: Bug while generating a Surrogate key for Incremental Load.

Issue is resolved

Thank you

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog