Insert unique value

One Star

Insert unique value

hi,
I have two file File1, File2. Both have two field name,id.
File1:
name ID
akki 123
akshath 124
akki 123
File2:
name ID
aks 125
akki 123
aks 125
These data i want to insert to table. At the end table should have only unique data.
i.e
name ID
akki 123
akshath 124
aks 125
From each file i can get unique row using tUniqueRow component, but how to get unique row between files.
My job is some what like this:
tFileList---->tFileInputde---->tMap---->tUniqueRow---->tVerticaOutput.
So here only file level duplication can be eliminated. i want to achieve row level duplication.
This is just an example i have thousands of file like this.
Please do suggest the answers.
Regards,
Akshath Hegde
Moderator

Re: Insert unique value

Hi,
All the thousand of files are same structure?
If so, you can use tfileinputxxx_1-->main-->tUnite-->tuniqrow-->UnqiRow-->toutputxxx
tfileinputxxx_2-->
See my screenshots
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Insert unique value

Hi,
Thanks for quick reply. But i cant use 1000 tFileInputDelimited for single job right!!! Im fetching file using tFileList because all file is of same extension and same structure. And after some time i may receive some more files, again i have to add tFileInputDelimited !!!. This doesn't look efficient.

Regards,
Akshath Hegde
One Star

Re: Insert unique value

Hi,
Anything else we can do for this??
Note:files are taken one at a time.
Regards,
AKKI
One Star

Re: Insert unique value

Hi,
I think you can get this using tfilelist -->iterate-->tfileinputdelimited-->log
In tfileinputdelimited we can use default global variable "((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))"
something like this.
As you said you have many files, it means tables should be read dynamicaaly.
So as Sabrina mentioned in other forumtopic. to get tables dynamically we should have subscribed version.
If you are using scbscribed version i think you can do this.
Regards,
Vijayvanamala