[resolved] Creating metadata and using it for set of files

One Star

[resolved] Creating metadata and using it for set of files

Hi,
I'm using TOS for Data Integration 5.4.1.
Objective : Creating metadata schema for one time and reading the csv files using metadata without specifying the schema on the fly.
Here i have created metadata for csv file and saved it. So fro other files with the same structure i can use the same metadata while reading the file using tFileInputDelimited(by selecting type as repository) , which works fine .
But facing issues while dealing with a set of csv files with the same field structure . Here i have created a job which reads csv files using tFileList.In next step when i use tFileInputDelimited to read the files , have selected the metadata by choosing type as repository. In FileName/Stream it shows the file which is given while creating the metadata.
So there i wanted to use " ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) " kind of thing. But if i click on the fileName it doesan't allow me to do so.

So in this what can be done to achieve this.
I have attached the screenshot of the job.

Accepted Solutions
Four Stars

Re: [resolved] Creating metadata and using it for set of files

1) Learn string manipulation, use tfilelistcurrentfilepath and postfix or prefix with the timestamp. This will create respective output file for each input.
2) Yes, it is applicable for xml files as well.
Small suggestion - If you have different problem, create a new thread... Mark existing thread as fixed. People or community users will not be able to relate multiple solutions in single thread with single heading...
Please mark this thread as fixed and start new question again...
Hope you understand.
Thanks
Vaibhav

All Replies
One Star

Re: [resolved] Creating metadata and using it for set of files

I have done this be making the metadata "Generic Schema" instead of "File Delimited" This is nice when sharing the metadata with File and Database Table as well ( such as loading the file into a table with the same metadata ).
In the context menu of the schema ( right-click ) there is a "copy to generic schema" option.
One Star

Re: [resolved] Creating metadata and using it for set of files

Hi,
Eventhough if i create Generic schema, how can i use for a more number of file. In the sense how can i use tFileList here and what can be my filename while reading using tFileInputDelimited component.
When i was creating schema on the fly , i was using " ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) "
in Filename of a TFileInputDelimited component but if i go for select the schema using metadata , it does not allow me to provide the samilar filename (Here tFileList is used)

In this case how can i proceed further?
Thanks,
ShreeCS
One Star

Re: [resolved] Creating metadata and using it for set of files

Does this work for you?
1. Create Schema on the fly in the way you already have working with tFileList
2. In tFileInputDelimited component click edit schema
3. Click the floppy image "Save as generic schema"
One Star

Re: [resolved] Creating metadata and using it for set of files

I don't want to create the schema on the fly.
If i'm not able to use the metadata schema for reading different file witthe the same structure, then what is the use of creating metadata?
I have attached the job and tFileInputDelimited screen shot.
One more thing , how to use the generic schema in reading the different file ?
Four Stars

Re: [resolved] Creating metadata and using it for set of files

Hi Shree,
In tfileinputdelimited, click on file name, click on built in and use (Ctrl+space) and select tFileListCurrentFilePath....
So, it will iterate on all the files available in the folder path provided in tFileList component.
Thanks
Vaibhav
One Star

Re: [resolved] Creating metadata and using it for set of files

Hi,
Here i wanted to use metadata schema not the built-in. If i choose a repository as a type then how can i choose the file name.
Thanks,
ShreeCS
Four Stars

Re: [resolved] Creating metadata and using it for set of files

Doing this is going to use metadata schema along with the file metadata details what you have specified, only it is going to change is the input file which is changing based on iterate flow...
Try doing above, you will understand....
Thanks
Vaibhav
One Star

Re: [resolved] Creating metadata and using it for set of files

Hi,
I have tried using the tFileList and tInputFileDelimited with the built-in schema type where i'm able to change the file path to CURRENT_FILE. But my requirement is to create the schema file prior of reading/parsing a file not on the fly and also to validate the schema.
I know that there is a component called as "tschemaComplianceCheck" to validate the schema. So i thought of using the metadata schema and reading the file using tFileDelimited and checking the validation using tschemaCompliancecheck. But i'm struck in tFileInputDelimited in specifying the filename.
So how can i proceed further in this scenario.
Thanks,
ShreeCS
Four Stars

Re: [resolved] Creating metadata and using it for set of files

Hi Shree,
Your first paregraph says that you are able to read the file. And second paregraph says that you are stuck.
Pl put a screenshot with clear description of what you want...
One thing to remember is there is no way, you can have dynamic metadata or metadata file creation or using it while runtime or similar using TOS.
Thanks
Vaibhav
One Star

Re: [resolved] Creating metadata and using it for set of files

Hi,
My 1st paragraph says , i have created a job with the components
tFileList >> tFileInputDelimited >> tLogrow
Here i have used built-in schema where I'm creating the schema on the fly.Here in tFileInputDelimited i'm able to specify the file by giving "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))".
By i do not wanted to create the schema on the fly .So i have created one more job with the components
tFileList >> tFileInputDelimited >> tLogRow
Here i created the metadata schema for FileDelimited and used in tFileInputDelimited by choosing Repository schema option. But if i do this, it gives me the file already choosen while creating the FileDelimited metadata. But i wanted this to change to "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))"> so that i read the corresponding input file and disply the result.
Also i'm not looking for dynamic metadata creation in talend.
I have attached the screen shot .
Thanks,
ShreeCS
Four Stars

Re: [resolved] Creating metadata and using it for set of files

check in screenshot, how it is implemented.
One Star

Re: [resolved] Creating metadata and using it for set of files

Hi,
Finally i got it and you cleared my doubts. Till now instead of selecting built-in i was chhosing repository where i was making wrong.
Thanks a lot and now the job is running properly.
But if i want to use tFileOutputDelimited instead of tLogRow. How it works. How it stores the different input files
as a output file with different name.
And alos whatever you explained is also applicable to xml files right. Is it like all the xml files should be with the same root and row tag or we can have different xml files ?

Thanks,
ShreeCS
Four Stars

Re: [resolved] Creating metadata and using it for set of files

1) Learn string manipulation, use tfilelistcurrentfilepath and postfix or prefix with the timestamp. This will create respective output file for each input.
2) Yes, it is applicable for xml files as well.
Small suggestion - If you have different problem, create a new thread... Mark existing thread as fixed. People or community users will not be able to relate multiple solutions in single thread with single heading...
Please mark this thread as fixed and start new question again...
Hope you understand.
Thanks
Vaibhav
One Star

Re: [resolved] Creating metadata and using it for set of files

Ya sure.
Thanks a lot for your help and guidance.
The thread has been fixed.
One Star

Re: [resolved] Creating metadata and using it for set of files

Resolved