How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Six Stars

How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

I was using the tFTPGet component to save folders/files from a remote server to my computer and then using tFTPPut component to transfer them to another remote location.

 

Occasionally, there are temporary files (~$excel file name    in my case) saved to my computer from the source. When I tried to tFTPPut the files to the remote server, the job failed. But as soon as I manually deleted the temporary file from my computer, the job succeeded.

 

BTW, the temporary file name actually meets the filemask requirement on the tFTPPut, but it just failed the job.

 

Anyone has experienced the same issue? I think I have to exclude these bad temp files either during tFTPGet or during tFTPPut. I would appreciate your help very much!

 

P.S.  With tFTPGet I saved folders/subfolders/files to local computer, and I used tFTPCreateDirectory to create the directories on the destination server and then used tFTPPut to transfer the files.

 

 

 

 


Accepted Solutions
Forteen Stars TRF
Forteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Try this pattern:
tFileList-->tJava--(runIf)-->tFTPPut
Let the tJava as it comes and enter the following expression for the runIf trigger :
!((String)globalMap.get("tFileList_1_CURRENT_FILE")).startsWith("~")
tFTPPut will be called only for filenames not starting with ~

TRF
Employee

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi,

 

     The condition has to be given in the link (the line between two components).

 

      When you click on the link, in the component tab, it will show the name as If1.

 

image.png

 Could you please try it and let us know the result?

 

Warm Regards,

 

Nikhil Thampi

Forteen Stars TRF
Forteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

As you are iterating after the tFileList component, you have to use (String)globalMap.get("tFileList_1_CURRENT_FILE") for the tFTPPut filemask.
Files will be transfered one at a time.

TRF

All Replies
Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Is there a pattern in the name of these files, so that you can put that in the Exclude pattern in the advanced setting while you are moving them to the different server.


Best Regards,
Abhishek
Six Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi abhishek,

 

Thanks for your reply!

 

I am new to Talend. As I know so far, only tFlileList component has the Exclude Filemask option in Advanced setting.

Here is what I tried with tFilelist:

 

1. Without the Exclude Filemask, in the local directory, 9 files will be listed according to my Filemask ("*.xlsm")

    without the Exclude Filemask.PNG

2. When I added the Exclude Filemask ("~$*), 8 files will be listed:

with the Exclude Filemask.PNG

 

3. And then I activated the tFTPPut component and ran the job, it failed:

run the tFTPPut with the Exclude Mask.PNG

Here is the error message -

Exception in component tFTPPut_3 (movefiles)
4:
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:594)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:475)
at pl_project.movefiles_1_0.movefiles.tFileList_3Process(movefiles.java:2092)
at pl_project.movefiles_1_0.movefiles.tFTPConnection_10Process(movefiles.java:2533)
at pl_project.movefiles_1_0.movefiles$4.run(movefiles.java:3796)

 

4. But as soon as I deleted the ~$xxxxx.slxm file from the local machine and ran the job again, it succeeded:

success run.PNG

 

Thanks,

 

TigerMommy

Forteen Stars TRF
Forteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Try this pattern:
tFileList-->tJava--(runIf)-->tFTPPut
Let the tJava as it comes and enter the following expression for the runIf trigger :
!((String)globalMap.get("tFileList_1_CURRENT_FILE")).startsWith("~")
tFTPPut will be called only for filenames not starting with ~

TRF
Six Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi TRF,

Thanks for the suggestion.

I never used tJava before. So I did as you suggested and the job failed at the tJava component -
add tJava.PNG

 

In the tJava, I only copied and pasted your code, didn't touch other places.

 

And here is the error message I got -

tJava error.PNG

 

What could be wrong?

 

Thanks!

Employee

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi,

 

     The condition has to be given in the link (the line between two components).

 

      When you click on the link, in the component tab, it will show the name as If1.

 

image.png

 Could you please try it and let us know the result?

 

Warm Regards,

 

Nikhil Thampi

Six Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

My bad -- TRF did ask me to enter the expression for the RunIf trigger but I overlooked.

Now I think the all the components were set correctly -

corrected_whole.PNG

 

In my tFTPPut component, I have a filemask:

FTPPut filemask.PNG

 

 

If, in my source folder, I have bad ~$ files like below among the good files, my job would still fail. But as soon as deleted the ~$All Supermarkets-TOTAL.xlsm,

the job runs successfully.

corrected_bad files.PNGI

This made me believe that it has something to do with the tFTPPut filemask.

Aren't these ~$ files excluded by the RunIf expression already?

 

Should I tried to put the tFTP Put filemask in the RunIf trigger?

 

Thanks!

 

 

P.S. Here is the error message if the ~$All Supermarkets-TOTAL.xlsm file was in my local source folder:

(the job failed immediately)


000.PNG

Forteen Stars TRF
Forteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

As you are iterating after the tFileList component, you have to use (String)globalMap.get("tFileList_1_CURRENT_FILE") for the tFTPPut filemask.
Files will be transfered one at a time.

TRF
Six Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi TRF,

 

You are correct -- I didn't do that!  Though I put the RunIf condition in the trigger to exclude the ~$ files, but all is done in air. I didn't change the source directory in tFTPPut, the tFTPPut is still pulling all the files, and it doesn't go well with "~$" files.

 

Before your published your solution, I just used tFileCopy after the RunIf trigger to save the good files into another folder. After that, I do my tFTPPut with old filemasks. It worked.

 

But yours are better - with tFileList, I don't need to create a new folder to hold these good files.

 

Thank you!