How to connect multiple FTP locations in the same job

Five Stars

How to connect multiple FTP locations in the same job

I have one table which contains many distributors and retailers ftp connection details.

The table structure is as follows:

Dist_Id       Reta_Id        Ftp_host       Ftp_user        Ftp_pswd
------------------------------------------------------------------------------------
1                 a                   123               abc                 xyz
1                 b                   456               def                  mno
2                 c                   431               ghi                  p9r
3                 d                   789               jkl                   12e


So for getting all files for Dist_Id=1, I want to connect ftp locations of Retailers a and b. And download files in each ftp location to the same folder.
How can I connect to multiple ftp locations and download files in the same folder??


Accepted Solutions
Highlighted
Six Stars

Re: How to connect multiple FTP locations in the same job

Neena,

 

You can do this many ways. Here is the easiest way I would recommend. First of all, you should send the FTP method (either FTP or SFTP) from the parent to tRunJob.  

Parrent_tRunJob.png

 

Here is the screenshot of child job

 

 

FTP_child_SFTP.png

 

tJava: just to print the FTP method

Child_ftp_tJava.png

 

First order: connect tJava with runif to FTP component.

Child_FTP_If1.png

Second order: connect tJava with runif 

child_ftp_if2.png

tFtp get with "SFTP support" enabled. depending on your design , you can choose the "Authentication method" to public key. provide key Passphrase and private key (if required).

child_ftp_sftp_comp.png

 

The idea is to have runif and decide your flow based on the ftp method sent by your parent. 

 

Hope this helps!

 

 

View solution in original post

Six Stars

Re: How to connect multiple FTP locations in the same job

Neena, 

 

below is the job with tNormalize as Denis suggested.

Demo_ftp_tNormalize.png

 

row 1 - settings:

row1_settings.png

tNormalize - component settings 

tNormalize_details.png

 Output

output_ftp_demo.png

 

Input

Inout_ftp_demo.png

 

 

Regards,

Ragu

 

 

View solution in original post


All Replies
Twelve Stars

Re: How to connect multiple FTP locations in the same job

create your ftp job for one connection one file (use context to store values of line 1)
create a job calling this job (trunjob) with context values from your table.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: How to connect multiple FTP locations in the same job

Thanks for your response.. There is a chance for coming new retailers for each distributor.. At that case, want to  connect to that retailer's ftp location automatically and download the files.. Could you please make it more clear with some examples...

Six Stars

Re: How to connect multiple FTP locations in the same job

Neena,

 

below is the example which Denis was trying to explain.

 

consider that the ftp details are there in a file. so use tfileinputDelimited and connect to tFlowToIterate which would iterate the tRunJob for every output row from the file. 

 

The advantage of tFlowToIterate is that it sets the inflow values in global variable as key-value pair. In tRunJob, pass all the parameters from tFlowToIterate to context variable of the child job.

 

In the child job you can have FTP component. use all the defined context variables in the ftp component.

 

Screen_shot_Parent.pngtFlowToIterate.pngtRunJob.pngChild_Context.pngftp_component.png

 

 

Hope this helps!

Five Stars

Re: How to connect multiple FTP locations in the same job

Thanks for your reply.. Smiley Happy It is very much helpful.. I Designed the job based on your reply.. But when connects to second ftp location it shows some exceptions :

Exception in component tRunJob_1
java.lang.RuntimeException: Child job running failed.
com.enterprisedt.net.ftp.ControlChannelIOException: Control channel unexpectedly closed ('' read so far)

 

Is there any solution??

Twelve Stars

Re: How to connect multiple FTP locations in the same job

try the second job with second ftp context.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: How to connect multiple FTP locations in the same job

I think the second one is a SFTP protocol. So how to handle both FTP and SFTP in the same job??

Twelve Stars

Re: How to connect multiple FTP locations in the same job

add a sftp subjob and switch defore or add sftp connection into subjob and switch inside subjob

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Highlighted
Six Stars

Re: How to connect multiple FTP locations in the same job

Neena,

 

You can do this many ways. Here is the easiest way I would recommend. First of all, you should send the FTP method (either FTP or SFTP) from the parent to tRunJob.  

Parrent_tRunJob.png

 

Here is the screenshot of child job

 

 

FTP_child_SFTP.png

 

tJava: just to print the FTP method

Child_ftp_tJava.png

 

First order: connect tJava with runif to FTP component.

Child_FTP_If1.png

Second order: connect tJava with runif 

child_ftp_if2.png

tFtp get with "SFTP support" enabled. depending on your design , you can choose the "Authentication method" to public key. provide key Passphrase and private key (if required).

child_ftp_sftp_comp.png

 

The idea is to have runif and decide your flow based on the ftp method sent by your parent. 

 

Hope this helps!

 

 

View solution in original post

Five Stars

Re: How to connect multiple FTP locations in the same job

Very Thanks for your reply... It works Smiley Happy .... 

Five Stars

Re: How to connect multiple FTP locations in the same job

Hi,

I have one more block to complete this job..Smiley Sad

 

While using context variable context.FTP_Files  as Filemask in FtpGet component,  it did not download files when more than one file name are provided in the FTP_Files column separated by comma.But downloads when only one file name is provided. How can I download all the files provided in the FTP_Files column?? 

 

Example: 

 

Dist_Id            Reta_Id             FTP_Files

---------------------------------------------------------------

 1                     a                       ab.csv,mn.csv

 1                     b                        pq.csv

 

In this case, for Dist_Id=1, context.FTP_Files only download pq.csv. Actually I want to download all these 3 csv files in the same folder.. (Dist_Id is same).

 

Thanks In Advance..Smiley Happy

Twelve Stars

Re: How to connect multiple FTP locations in the same job

filters are list you cannot dynamically convert strings to list.
so you have to use a tNormalise to split your file liste.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Six Stars

Re: How to connect multiple FTP locations in the same job

Neena, 

 

below is the job with tNormalize as Denis suggested.

Demo_ftp_tNormalize.png

 

row 1 - settings:

row1_settings.png

tNormalize - component settings 

tNormalize_details.png

 Output

output_ftp_demo.png

 

Input

Inout_ftp_demo.png

 

 

Regards,

Ragu

 

 

View solution in original post

Five Stars

Re: How to connect multiple FTP locations in the same job

Thanks for both of you..Smiley Happy It Works...

Five Stars

Re: How to connect multiple FTP locations in the same job

Hi,

 

I have one more doubt..Smiley Tongue 

 

When I download all retailer files and inserted to one common table, it can not distinguish which data is getting from which Retailer.. So I want to append one column(Reta_Id) in all csv files when the file is downloaded from ftp location. Is it possible to append a column in csv at the time of downloading from FTP??

 

Thanks In advanceSmiley Happy

Twelve Stars

Re: How to connect multiple FTP locations in the same job

yes you can but you have to download then open and add column.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: How to connect multiple FTP locations in the same job

Thanks for your reply...

 

I want to add the Retailer_Id column in each file when it is downloaded.. Then only corresponding Retailer_Ids can be added.  Since there are many files Manually open and adding  the column is not possible..

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog