Downloading only certain directories from an FTP server

Five Stars

Downloading only certain directories from an FTP server

Hello Talend Users,

 

I have a FTP directories with many subdirectories (representing years from 1999 to 2018) . For now I can only download all files in all folders. However I want to only download the folders from 2011 to 2018.

 

So far my Job looks like this:

Screenshot_1.pngHow can I iterate over the Folders from 2011 to 2018? With tForeach? And if yes how can I connect it to FTPGet?

 

Thanks in Advance.

Moderator

Re: Downloading only certain directories from an FTP server

Hello,

Do you want to list sub-directories in FTP folder?

Could you please let us know if this resolved topichttps://community.talend.com/t5/Design-and-Development/Exclude-File-Mask-in-TFTPFileList/m-p/124806 is OK with you?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Downloading only certain directories from an FTP server

This is not really the solution to my problem as it is about excluding certain files from the download. I however want to download only the directories 2011 to 2018 in a list of directories which reaches from 1999 to 2018. So somehow I have to provide a list to the FTPget job.

Best regards
Maximilian

 

The FTP Directory looks like this:

Screenshot_2.png

Nine Stars

Re: Downloading only certain directories from an FTP server

You could use tSSH to filter out the folders using unix commands. The output of tSSH you could then store in global variable which is tSSH_1_STDOUT

Once you have the list of folders in the global variable just use them to iterate over using

tFTPFileList -> iterate ->tFTPGet

 

Sample unix command to filter out folders starting with "talend" -

"find $(pwd) -maxdepth 1 -type d -name \"talend*\""
Five Stars

Re: Downloading only certain directories from an FTP server

Hi,

ok sounds really complicated to me since I am a Talend Beginner. How do I connect the tSSH job to the FTPfilelist?
Nine Stars

Re: Downloading only certain directories from an FTP server

tSSH - > on subjob ok -> tFTPFileList
It might sound complicate but it's not. The solution is not straight forward because FTP support is with files and not with folders - that's the reason for the tweak.