How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Highlighted
Seven Stars

How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

I was using the tFTPGet component to save folders/files from a remote server to my computer and then using tFTPPut component to transfer them to another remote location.

 

Occasionally, there are temporary files (~$excel file name    in my case) saved to my computer from the source. When I tried to tFTPPut the files to the remote server, the job failed. But as soon as I manually deleted the temporary file from my computer, the job succeeded.

 

BTW, the temporary file name actually meets the filemask requirement on the tFTPPut, but it just failed the job.

 

Anyone has experienced the same issue? I think I have to exclude these bad temp files either during tFTPGet or during tFTPPut. I would appreciate your help very much!

 

P.S.  With tFTPGet I saved folders/subfolders/files to local computer, and I used tFTPCreateDirectory to create the directories on the destination server and then used tFTPPut to transfer the files.

 

 

 

 


Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Try this pattern:
tFileList-->tJava--(runIf)-->tFTPPut
Let the tJava as it comes and enter the following expression for the runIf trigger :
!((String)globalMap.get("tFileList_1_CURRENT_FILE")).startsWith("~")
tFTPPut will be called only for filenames not starting with ~

TRF
Employee

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi,

 

     The condition has to be given in the link (the line between two components).

 

      When you click on the link, in the component tab, it will show the name as If1.

 

image.png

 Could you please try it and let us know the result?

 

Warm Regards,

 

Nikhil Thampi

Fifteen Stars TRF
Fifteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

As you are iterating after the tFileList component, you have to use (String)globalMap.get("tFileList_1_CURRENT_FILE") for the tFTPPut filemask.
Files will be transfered one at a time.

TRF

All Replies
Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Is there a pattern in the name of these files, so that you can put that in the Exclude pattern in the advanced setting while you are moving them to the different server.


Best Regards,
Abhishek
Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi abhishek,

 

Thanks for your reply!

 

I am new to Talend. As I know so far, only tFlileList component has the Exclude Filemask option in Advanced setting.

Here is what I tried with tFilelist:

 

1. Without the Exclude Filemask, in the local directory, 9 files will be listed according to my Filemask ("*.xlsm")

    without the Exclude Filemask.PNG

2. When I added the Exclude Filemask ("~$*), 8 files will be listed:

with the Exclude Filemask.PNG

 

3. And then I activated the tFTPPut component and ran the job, it failed:

run the tFTPPut with the Exclude Mask.PNG

Here is the error message -

Exception in component tFTPPut_3 (movefiles)
4:
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:594)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:475)
at pl_project.movefiles_1_0.movefiles.tFileList_3Process(movefiles.java:2092)
at pl_project.movefiles_1_0.movefiles.tFTPConnection_10Process(movefiles.java:2533)
at pl_project.movefiles_1_0.movefiles$4.run(movefiles.java:3796)

 

4. But as soon as I deleted the ~$xxxxx.slxm file from the local machine and ran the job again, it succeeded:

success run.PNG

 

Thanks,

 

TigerMommy

Fifteen Stars TRF
Fifteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Try this pattern:
tFileList-->tJava--(runIf)-->tFTPPut
Let the tJava as it comes and enter the following expression for the runIf trigger :
!((String)globalMap.get("tFileList_1_CURRENT_FILE")).startsWith("~")
tFTPPut will be called only for filenames not starting with ~

TRF
Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi TRF,

Thanks for the suggestion.

I never used tJava before. So I did as you suggested and the job failed at the tJava component -
add tJava.PNG

 

In the tJava, I only copied and pasted your code, didn't touch other places.

 

And here is the error message I got -

tJava error.PNG

 

What could be wrong?

 

Thanks!

Employee

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi,

 

     The condition has to be given in the link (the line between two components).

 

      When you click on the link, in the component tab, it will show the name as If1.

 

image.png

 Could you please try it and let us know the result?

 

Warm Regards,

 

Nikhil Thampi

Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

My bad -- TRF did ask me to enter the expression for the RunIf trigger but I overlooked.

Now I think the all the components were set correctly -

corrected_whole.PNG

 

In my tFTPPut component, I have a filemask:

FTPPut filemask.PNG

 

 

If, in my source folder, I have bad ~$ files like below among the good files, my job would still fail. But as soon as deleted the ~$All Supermarkets-TOTAL.xlsm,

the job runs successfully.

corrected_bad files.PNGI

This made me believe that it has something to do with the tFTPPut filemask.

Aren't these ~$ files excluded by the RunIf expression already?

 

Should I tried to put the tFTP Put filemask in the RunIf trigger?

 

Thanks!

 

 

P.S. Here is the error message if the ~$All Supermarkets-TOTAL.xlsm file was in my local source folder:

(the job failed immediately)


000.PNG

Fifteen Stars TRF
Fifteen Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

As you are iterating after the tFileList component, you have to use (String)globalMap.get("tFileList_1_CURRENT_FILE") for the tFTPPut filemask.
Files will be transfered one at a time.

TRF
Seven Stars

Re: How to skip temporary files (~$ excel file name) when using tFTPPut to transfer files to remote server?

Hi TRF,

 

You are correct -- I didn't do that!  Though I put the RunIf condition in the trigger to exclude the ~$ files, but all is done in air. I didn't change the source directory in tFTPPut, the tFTPPut is still pulling all the files, and it doesn't go well with "~$" files.

 

Before your published your solution, I just used tFileCopy after the RunIf trigger to save the good files into another folder. After that, I do my tFTPPut with old filemasks. It worked.

 

But yours are better - with tFileList, I don't need to create a new folder to hold these good files.

 

Thank you!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download