Need help with tFTPGet -- when the expected folders are not on the server!

Highlighted
Six Stars

Need help with tFTPGet -- when the expected folders are not on the server!

I am working on a project where I have to save certain folders to local. There folders are named with current fiscal period number (so it changes from month to month).

For example, there will probably be folders on the server like this:

 

Source -> Apple -> 2018-11

Source -> Orange -> 2018-11

Source -> Mango ->2018

Source -> Grape ->2018-11

Source -> Peach

Source -> Strawberry ->2018-11

-- We don't have "2018-11" folders from "Mango" or "Peach".

 

There is my current job design:

tFTPFileList   -> tIterateToFlow -> tMap -> tFlowToIterate -> tFTPGet

( I mapped the source folders to tMap, and in tMap I filtered out these "Mango" and "Peach" folders that don't have the "2018-11" subfolders  using the code below:

!row1.SourceFolder.equals("Mango")&&

!row1.SourceFolder.equals("Peach")

 

And here is the configuration with tFTPGet component:

0911 tFTPGet.PNG

BTW, I have "Die on error" unticked!

The job works fine now. But, if there is any change in the source, for example, if "Orange" folder no longer has the "2018-11" subfolder:

 

 

Source -> Apple -> 2018-11

Source -> Orange

Source -> Mango ->2018

Source -> Grape ->2018-11

Source -> Peach

Source -> Strawberry ->2018-11

 

The job would fail right after the "Apple -> 2018-11" is saved to local.

 

Question:

1. With "Die on error" option unticked, why the job would still fail?

 

2. My current job works but it is hard-coded (the filters). Is there a way to avoid the hard-coding? Because I will never know what will happen next month.

 

I hope I have expressed myself well and I look forward to your suggestions!

 

TM

 

 

 

 

 

 

 

 

 

Seven Stars JGM
Seven Stars

Re: Need help with tFTPGet -- when the expected folders are not on the server!

what stack trace are you getting when it fails?
Six Stars

Re: Need help with tFTPGet -- when the expected folders are not on the server!

Hi JGM,

 

Here is the error message I got the job failed at tFTPGet component:

Exception in component tFTPGet_2 (MoveFilesToPandles)
2: The requested file does not exist
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2198)
at com.jcraft.jsch.ChannelSftp.cd(ChannelSftp.java:343)

 

Sorry, I don't know what  "stack trace" means here.

 

Seven Stars JGM
Seven Stars

Re: Need help with tFTPGet -- when the expected folders are not on the server!

that error is the stack trace Smiley Happy it shows what methods were being called when the error occurred. the stack traces are super important to debugging-- so its worth taking some time learning how to read them. Check out this link for more info:

http://marxsoftware.blogspot.com/2010/10/reading-java-stack-traces-few-tips.html

 

 

Now, on to your error.

it looks like in your job, you're building the path to your expected file in your tFTPGet component. I cant tell exactly with the screenshots, but it looks like something like this: 

globalMap.get("base_dir") + "/" + globalMap.get("date_dir") .....

 

 

because of this, when the path is not exactly what your code expects, it builds a path that does not exist, leading to the error. 

The simple fix to this is to let the paths the tFTPFileList component see's be the ones you use in your job. There is a globalMap variable that the tFTPFileList component will populate once per iteration:

((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH"))

 

 

You'll need to redesign your job slightly to use it as it will be populated once per iteration, and you're using a IterateToFlow that will consume all the iterations. Give this setup a try and see if you can make it work for your use case:

 

 

tFTPFileList   ---iterate-->tJava --onComponentOK--> tFTPGet

 

 

in your tJava, you'll need to construct the remote directory without filename:

 

java.io.File remoteFile = new java.io.File(((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")));

context.RemoteDir = remoteFile.getParent();

context.FileMask = ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"));

 

 

then, in the tFTPGet component, populate the remote directory and filemask with your context vars. 

 

 

hope that helps -- 

Garrett

 

Six Stars

Re: Need help with tFTPGet -- when the expected folders are not on the server!

Garrett,

I feel you probably have a way to help me out, but I don't fully understand your codes (especially the tJava part). I just started to learn to use Talend a few weeks ago.

 

In my current job, I will check each folder on source server and see if they have current FiscalPeriod "2018-11" subfolders. If they do, I will copy the directories and files to local (eventually, I will create the same directories on the destination server and transfer the files there).

 

For a folder on the source server which doesn't have "2018-11", there is nothing we can transfer because we are only transferring the most recent files to the destination server.

 

Let me re-attach my current job design:

0911 current tFTPGet.PNG

0911 current tFTPGet02.PNG

On tMap, I filtered out these folders without "2018-11" subfolders if I already knew:

!row1.SourceFolder.equals("Mango")&&

!row1.SourceFolder.equals("Peach")

 

The Remote directory on above tFTPGet is: ((String)globalMap.get("NewFolderPath"))+"/"+((String)globalMap.get("FiscalPeriod"))  -- because I only care about those with expected files.

 

((String)globalMap.get("NewFolderPath")) should initially be ((String)globalMap.get("tFTPFileList_2_CURRENT_FILEPATH")), I renamed it after tMap and tFlowToIterate.  ((String)globalMap.get("NewFolderName")) should initially be ((String)globalMap.get("tFTPFileList_2_CURRENT_FILE")) as well.

 

I believe the job failed because of ((String)globalMap.get("NewFolderPath"))+"/"+((String)globalMap.get("FiscalPeriod"))  : when it hit the folder "Orange" , since it no long has "2018-11" subfolder, it failed.

 

 

I do need to keep the directories from the source server when saving to local, since I need to transfer files from local to destination with matching directories.

 

Could you please explain your design a little more detailed ?

Thanks,

 

TM

 

 BTW, context.Local_Dir_Base + context.Local_SubDir_Division are the context variables I created for the local directory to save the copied folders. These would be constant in my job and I believe they wouldn't have any impact on my job.

 

Did you mean I should try this?

Copy all the expected files to a local folder (without directories) by using "*2018-11*" in the filemask and reconstruct the directories in the local?  Then I think the re-construction should be done in local after tFTPGet has saved all files. I am lost. Need help! Smiley Happy