[resolved] Not able to process multiple files using tFileList

One Star

[resolved] Not able to process multiple files using tFileList

I am unable to configure Talend to process multiple files. I have a job that transforms TXT files to HL7 files and that part of the job is working.
I am trying to iterate on a folder with multiple TXT files to transform them to HL7. I am using tFileList as the component to provide this function. There are 10 TXT files to be process in the source directory.
When I run the job, only the last of the 10 txt files is transformed.
Please help!!! Smiley Happy

Accepted Solutions
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hello everyone,
I was able to develop a workaround regarding the issue of not being able to iterate on a multiple source files and transform and save to multiple HL7 files.   It seems there is an issue with the tHL7output component and it does not allow to iterate on a source directory.  what happens is only the last source file is transformed.  When using tFileOutputDelimited component, the source directory iteration works as expected.
So, because the tHL7output component has a bug, I developed a different was to iterate on the source directory that allows multiple HL7 files to be created based on the number of source files.  The parent job gets the source data from a SP and then creates a separate file for each row returned.  The child job iterates on the source directory and creates an HL7 file in the destination directory for each source file found.  The job ends when source files have been processed.
I am going to spend the next few days evaluating the job to ensure stability and likely make other minor improvements which I'll post.
Happy to answer any questions...


All Replies
Moderator

Re: [resolved] Not able to process multiple files using tFileList

Hi,
How did you set your tHl7output component? Could you please try to use tFileOutPutDelimited with checked "Append" option instead of tHl7output component to see if your job works well?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi xdshi,
thank you for your response. OK, I'll replace the HL7 Output with the fileoutputdelimited control. Once thing I noticed is the connector to the HL7 Output is a main'merge' whereas all examples I've seen do not have a 'merger' connnector. I'll try shortly and post my results... thanks again...
Please note that I am happy to post the job up somewhere if you or anyone from talend would like to have a look...
Moderator

Re: [resolved] Not able to process multiple files using tFileList

Hi,
From your screenshots, you have already got 10 txt files by tFileList. There is an iterate link that loop on files contained in a directory, on rows contained in a file or on DB entries between tfilelist and tfileinputdelimited, that's the reason why there is 1 row in main row. The job design is correct. If you want to get all literared files, you should "Append" them.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi xdshi,
thanks for your response. OK. Yes, when I replaced the HL7 Output component and replaced with a tFileInputDelimited component, all 10 txt files were iterated through and 10 new txt files were created in a separate folder I defined.
It seems like the HL7 output component requires a different iteration process. As I mentioned, when using the HL7 Output component the 'Main' connector becomes a 'Main''Merge connector.
So, how would I transform multiple txt files to hl7 files? It seems like there is a different method I should be using but all instruction indicates doing it just as I have done?
Please advise and thanks again!
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, just noticed I wasn't using your correct name. sorry about that. Smiley Happy
On a hunch a split the 3 segments of the HL7 (MSH, EVN, PID) to 3 tmap inputs. I was hoping that was the problem but unfortunately the results are the same.
Please let me know what other information I can provide. I can post the job as well if you'd like to have a look at it.
Thank you
Moderator

Re: [resolved] Not able to process multiple files using tFileList

Hi,
Could you try to put your multiple txt files on local disk first then loading the flat file into tHL7output?
The workflow should be: tfilelist-->iterate-->tfileinputdelimited-->tfileoutputdelimited(Append)
|
onsubjobOK
|
tfileinputdelimited_1(same path with output file)-->tHL7output.
Let me know if it is OK with you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina,
I made the changes to the job as requested but the results are the same. I've posted an image of the job. The image is called "OnSubJobOK_Example".
I set the 'Append' bit as well. I've also tried both UTF-8 and ISO-8859-15 as encoding types but that did not do anything different. I don't know if that has anything to do with it but was the one thing I've not changed.

Please let me know if I can post the job somewhere for you to have a closer look at it.
I'm desperate!!! Smiley Happy
Thanks!
Moderator

Re: [resolved] Not able to process multiple files using tFileList

Hi,
Could you please send an email to me with your exported job? We will make a further investigation on it.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!
Seventeen Stars

Re: [resolved] Not able to process multiple files using tFileList

hi,
do a simple test.
iterate the repository with tFilelist and link to a tRunJob.
As parameter past the value of the iterate file.
in child job calling by tRunJob, put a tJava & use println() method to see each file going through the process.
when it's ok, in child job manage your data coming from file iteratevely.
if result have to go in a single file , use append.
hope it help
regards
laurent
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, I'm going to try to do what kzone recommended though at first glance I'm not sure I understand.
Would you be able to provide me with an email that I can attach the zipped job?
Thanks
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi kzone, thank you for your help. I am unable to understand how to 'iterate the repository with tfilelist'...
would you be able to provide additional information?
Thank you,
Ray
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, I am going to try to connect a tRunJob that will run the job that does successfully create the last of the 10 HL7 files from the TXT files. I think if I add a tFileDelete after the HL7 output, i can delete the last txt file used. I can then continue to run the run component until no TXT files are left. That will work right? Well I'm off to try it... Please keep me informed with what you are finding with the issue. Thanks!!
One Star

Re: [resolved] Not able to process multiple files using tFileList

hi,
I'm very new to the Talend
I too had the same problem.I'm trying to merge the multiple files into a single file which have the same schema.For that purpose I used workflow as
tFilelist-->tFileInputDelimited-->tFileOutputDelimited(append)
I used tFileInpuDelimited from Repository and in order to merge changed the property from built in into repository and changed the file path as per tFilelist global map variable. Please let me know where I did mistake.
Thanks in advance.
Moderator

Re: [resolved] Not able to process multiple files using tFileList

Hi,
With this job design, it will delete the current txt file that is passed by tRunJob from the main job to child job.
Could you please send your current job to me( the email address is just under my avatar).
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina,
Could u please provide the solution to my problem
Seventeen Stars

Re: [resolved] Not able to process multiple files using tFileList

here is an example to iterate on File repository and manage data from each file in a child jobh (just a tJava for the test) @screenshoots
regards
laurent
One Star

Re: [resolved] Not able to process multiple files using tFileList

Thanks Kzone, going to try this now...
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, the E-mail link below your avatar does not provide a way to attach a job. Would you provide more information on how to attach a job to send to you via email?
Thanks!
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!
One Star

Re: [resolved] Not able to process multiple files using tFileList

Hello everyone,
I was able to develop a workaround regarding the issue of not being able to iterate on a multiple source files and transform and save to multiple HL7 files.   It seems there is an issue with the tHL7output component and it does not allow to iterate on a source directory.  what happens is only the last source file is transformed.  When using tFileOutputDelimited component, the source directory iteration works as expected.
So, because the tHL7output component has a bug, I developed a different was to iterate on the source directory that allows multiple HL7 files to be created based on the number of source files.  The parent job gets the source data from a SP and then creates a separate file for each row returned.  The child job iterates on the source directory and creates an HL7 file in the destination directory for each source file found.  The job ends when source files have been processed.
I am going to spend the next few days evaluating the job to ensure stability and likely make other minor improvements which I'll post.
Happy to answer any questions...