How to filter the records based on the values in the config file?

Six Stars

How to filter the records based on the values in the config file?

How to filter the records based on the values in the config file?

 

Presently am  filtering  out the data based on specific ID. ID is based on the client. As of now we have two client ID. I have placed the two id in config file and filter the records based on that.

 

now the scenario is we may get the multiple clients. just we need to add the client ID in config file but dont want make any changes in the job.

How can we do this?

 

Based on the client ID thosed record file should move to the specific client Blob storage.

Please let me know how can we do this??

 

 

 

Highlighted
Employee

Re: How to filter the records based on the values in the config file?

Hi,

 

   You can read the client file and make it as iteration mode.

 

    For each iteration, you can read the data and in the output expression, you can add the client id from the tFlowtoIterate as the filtering parameter. This will make sure that each iteration will have the data for corresponding client. 

 

    The output data from tMap can be poited to tfileoutputdelimited where you can pass the client name again in the file name.

 

    Please note that this option is best for small and medium files as we need to read the same file multiple times for each client id. 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: How to filter the records based on the values in the config file?

single files will have multiple client IDs. I am able  to filter out the data based on IDs as u mentioned. 

But when u move the files to blob storage it should be based on the specific blob container.

for eg :if we have three client data file i should not add all the three blob storage with one storage component i need to move the files based on the details provided in the config file. 

Employee

Re: How to filter the records based on the values in the config file?

Hi,

 

If you could share your job flow, component screenshots along with some sample files, I will get more idea about your use case.

 

Could you please add these details to Talend community site after removing sensitive information?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 

Six Stars

Re: How to filter the records based on the values in the config file?

Please find the below job flow.

first flow based on the client ID columns i am able to filter the records based on ID with the file name.

 

Secondly i need to move the respective client ID file to the respective blob storage.

eg: 123456.csv

      789123.csv

      916733.csv

Need to move to there respective blob storage.

 

In my design flow second step it is having single blob connection.

I cannot create 3 blob connection for each client.

 

Now I need to configure the client connection in config file and pass the file based on the client.ID to there respective client blob.

Employee

Re: How to filter the records based on the values in the config file?

Hi,

 

    My understanding is that connection can be established without providing the BLOB container name details.

 

    So why don't you establish a connection as the first step and do the put part later in iteration? In this way, you can move multiple files using same connection.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now

How to Modernize Your Cloud Platform for Big Data Analytics With Talend and Micr...

Learn how<SPAN>to modernize your Cloud Platform for Big Data Analytics with Talend and Microsoft Azure</SPAN>

Read