My requirement is to load data from excel to oracle table with below conditions
Table columns are
cust_001, John, Watch,3, Mumbai, I-------------------->flag should change
1.If record already exists in table then FLAG column should change from "I" to "U"
cust_001, John, Watch,3, Mumbai, U---------------------->Already existing record on table
cust_002, Mathew, Bag, 3, Bangalore, I
cust_001, John, Watch,3, Mumbai, I-------------------->record from excel should insert like this
My job design looks like below screenshot
The problem is new records from excel inserting with flag "I" but existing record flag is not Insert as the new record it just update the existing record Flag
1.InsertRec--->output-->action on table --->Insert
2.Update -->output-->action on table -->Tried with Insert or update ,update or insert,update but it's not inserting as new record
When I tried with Insert as action on table it is inserting the record which is already in table is as below
cust_001, John, Watch,3, Mumbai, U
when I work with start_date,end_date using SCD component it's giving me exactly what I'm looking for.
Hope you understand the problem.
As soon as possible please help me to resolve this!
Thanks in advance
Solved! Go to Solution.
How are setted the "Catch lookup inner join reject" option on both output flows?
It should be "True" for Insertrec flow, and "False" to Update flow.
Also, you can hard code the FLAG value for both flows, "I" for Insertrec and "U" for Update.
Like you said already I kept the same, please have a look at the below screenshot
am I doing any mistake in the variables "updateFlag" and "InsertFlag"?
If I understand your request well, you want to insert the source record also into db even though it already exists. There is a design issue in your job, the existing records always go to the update output on tMap and are updated in db.
I just replaced variable and hard code as "i" and "U"
and I got below output
I kept Action on table for both output is as "INSERT" only
How is the "Catch lookup inner join" option for Insertrec flow?
Regarding the variable "InsertFlag", as soon as the record is not found in db because it is a new one, you should simply hard code the value ("I") in the output flow like this:
Also think you can do the same for Update flow with the value "U", don't you?
I hope I got output like I shown the above screenshot.
I just want to make sure whether it works fine by adding date column.
start_date and End_date
Will work on it and get back to you if I face any problem
I've just read again you original post, so I understand every record from Excel file must be inserted into the db with a value of "I" if it doesn't exists yet in the db, and a value of "U" if it exists.
IMHO, "Action on data" should always be set to "Insert" and you just need 1 output flow for both new and exisiting records.
For this, use an "Left Outer Join" in tMap, then decide the value of the flag field depending on the join result (field will be null for new records, so you just have to replace the value by "I" in this case):
I've not tested but should works.
NO, I don't want hour you can notice data In which 5 records already exists in a table with the start_date as 5/24/2017 and flag as "I" .(ie.for
In Excel Records presents are (cust_006,cust_007,cust_008,cust_009,cust_001)
so here cust_001 already exists in the table with start_date 5/24/2017.
what I expect is those records which are already in the table should update its flag from "I" into "U" and start_date
1.update its flag from "I" into "U" and
2.start_date is same as the date in the table ie(5/24/2017) and end_date becomes the system date.
3.record from Excel should insert as new record
that is (cust_001, john, HandBag, 2, Chennai, I, current date.
I think there is a design issue, should be:
tFileInputEexcel--main--tMap1(inner join)---update..tMysqlOutput (for upate)
on tMap1: do an inner join to get only the records onlywhich already exists in target db, these records will be updated.
Set the flag column to "U"
Set the end date column as TalendDate.getCurrentDate()
on tMap2: just add two new columns in the output table
flag: set it to "I"
start date: set it to TalendDate.getCurrentDate()
Let me know if it works.
Thanks your quick response @shong
I have followed your job flow as expected I got an output for flag column coming to date
1.for new rec its start_date as 'sys date' works fine.
2.for the record which flag changed from "I" to "U" start_date becomes "null" end_date as "sysdate"
my requirement is that start_date should change based on the record coming data.
I hope you can understand the logic which is missing here
have a look at below screenshot
I would like to achieve that
Thanks a million
Finally, I got my expected result.
If you don't mind can you explain me in short what is the concept blind this, it would be helpful for me!
I am implementating SCD type2 without using SCD component in talend but i am not getting Startdate and End as par my requirement.
1.If data is fresh then start date will set as current date and end date as null.
2.if data is already exist in DB but one attribute changed that data then what it should do for previous record end date should be as current date and for new updated record start date as current date and end date will be null.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema