Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Highlighted
Six Stars

Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Hello,

On my ftp location, I have folders: Alpha, Beta, Chroma, Delta which in turn have folders, Alpha1, Alpha2, Alpha3; Beta1, Beta2, Beta3; Chroma1, Chroma2, Chroma3; Delta1, Delta2, Delta3 respectively. Each of these folders have .csv files with names, Data_date.csv (ex. Data_01_01_2018.csv). The file names are common along folders. I have some of these files on my in folders by the same names as FTP location. I need to compare both, get a list of files that are not present and pick those files to pc storage. Can someone suggest me how to do this?

In future, I also need to have it so that there are no files on PC but only the list of files that were before and whatever filename isn't on the filelist, has to get picked.

Anyone here to help?

 

Thanks in advance.

Twelve Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

use tFtpFileList --iterate--> tIterateToFlow --> tMap --> ListOf nonMatched File
use tFileList --iterate--> tIterateToFlow --------^

ftp: tIterateToFlow use:
(String)globlaMap.get("fFTPFileList_1_CURRENT_FILEPATH")
(String)globlaMap.get("fFTPFileList_1_CURRENT_FILE")

filelist: tIterateToFlow use:
(String)globlaMap.get("fFileList_1_CURRENT_FILEPATH")
(String)globlaMap.get("fFileList_1_CURRENT_FILE")
good luck

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Six Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

@fdenis, how about the folders? This approach works if there's one folder. Not many.

Twelve Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

on tfilelist you can include subdirectories
on tFtpfilelist you have to include the good fileMask to run throw directories.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Six Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

@fdenis, good file mask? They are csv files and I'm using "*.csv" as filemask. What is it I'm missing?

Twelve Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

something like "*\\*.csv"
or "**\\*.csv"
or regexp.
if it's not working you can create a subjob calling him self on folder, llisting files when files.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Six Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

@fdenis, not working. The file names are same across folders. So, that's a problem.

I'm trying to use another approach.

tFileInputDelimited (with names of folders - Alpha, Beta, Chroma, Delta) -> tFlowtoIterate -> tLoop -> tJava

(converting folder names to a context output)-> tFileInputDelimited (with names of subfolders)-> tFlowtoIterate -> tLoop -> tJava (converting subfolder names to context output) -> tFTPFileList -> tIteratetoFlow -> tMap (Connected to a tFileList and tIteratetoFlow) -> tFlowtoIterate -> tFTPGet

 

I'm giving the names of folders and subfolders and using a loop element to run it. But it doesn't work. Can anyone help?

Six Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

@TRF, could you help may be?

Twelve Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

It was working on an other project.
do not use CURRENT_FILE who is only filename. use CURRENT_FILEPATH who get filename and path

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Twelve Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

be careful: context are mead to be job or subjob parameters.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Six Stars

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

@fdenis, got it. Thank you. Figured out a different way with loops. Smiley Happy

Thanks a lot for help. Smiley Happy

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch