Lookup multiple files

One Star

Lookup multiple files

Hi, As a requirement I have to join 16-20 inputs as lookup. what is the most efficient way to perform lookup ?
Five Stars

Re: Lookup multiple files

It's difficult to say without seeing your requirement, data structure, and volume; however, I'd say your starting point is to simply join your data using either tMap or tJoin.
One Star

Re: Lookup multiple files

Hi, requirement is something like below-
Input file- file0.dat
15 lookup files- file1.dat,file2.dat....file15.dat
lookup key is same for all lookups, say process_id
now I am fetching certain columns from each lookup file.
1) If I add input file as main flow and rest all  files as lookup on a single tMap, there is a big performance hit.
2) Another way I could think is, perform these lookups on multiple tMaps instead of single tMap. (not a very good option though)
So I am looking around the other possible ways to perform this. Thanks in advance.
Five Stars

Re: Lookup multiple files

I still think it comes down to your data volume and also your expectation.
The simple answer is to join them in a single tMap or try tJoin.
If one or more look-up files is causing an unacceptable performance issue, then you can look for specific ways to improve this.