I have a requirement where I will be getting approx. 10-12 files with varied schemas.
I will need to load them to their respective database tables.
There will be very little manipulation, such as:
1.) One Sequence Number Field will be extra on the table(basically row_num sort of a thing)
2.) One Date_load field will be extra, which will be derived from filename.
How do I achieve this in one generic job?
File Name: TableA_04-07-2017.csv
Table Name: TableA_Staging
Expected Output on Table:
Solved! Go to Solution.
Use the dynamic schema functionality in the subscription version of the product.
You will read the name of the file, use string manipulation to extract the table name from the file name, then generic job to read the file using dynamic schema, send to a tmap, add the additional sequence number and date column to the tmap, and out to your database. The output component will use the table name you extracted from the file name.
I am using dynamic schema now. And am able to transform fields that are present on the files using tJavaFlex.
But not sure how to add extra fields.
Can you show me manipulate dynamic schema on the tmap component.
Was able to do it. But now I am getting an error that says
[ERROR]: sat_control.testdynamicschema3_0_1.testDynamicSchema3 - tOracleOutput_1 - ORA-00917: missing comma
Tried that... Got the same error!
Also tried writing to a file and the read all fields as dynamic instead of part dynamic and part conventional fields.
Got another error after that : ORA-00928: missing SELECT keyword.
This is difficult to diagnose by post like this. Do you have an annonymised job that you can export and attached to show what problems you are having?
Join us at the Community Lounge.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend