how to provide dynamic file name to fetch the file from azure blob??

Six Stars

how to provide dynamic file name to fetch the file from azure blob??

Please let me know how to give file name if tazurestroageList_1 . it should pick the file name with the pattern. 

 

Eg: filename

123_612673id778_student_plan_2018-20-08.csv

 

everytime 

123_612673id778 and 2018-20-08 will be changing..

Student_plan is constant.

how can i pick the file from the lis in blob.

 

Please find the below screen shot.

 

Let me know how to provide in the bob filter prefix.

 

 

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

For tAzureStorageList the "Blob filter" doesn't work like normal File List filters, and we don't have the option to use regular expression or string functions to match or filter out blobs.
SO, when you define a prefix, you are actually designating a directory level as the blob filter else you just pass null to bring all in.

 

One way to achieve what you have been trying is to bring all the blobs and then use tJavaRow to filter out the blobs using simple regular expression or any string matching functions.

 

Job Layout - 

tAzureStorageList-1.JPGJava Code - 

String blobName = input_row.BlobName;

if(blobName.matches("(?i).*student_plan.*")) {
	System.out.println(blobName);
}
Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

input_row cannot be resolved to be variable. M getting tis kind of error.

 

 

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

Show us the job layout, highlighting the component and error.
Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

Please find the attached screen shot.

 

i need to store the file name in the context variable like p_filename.

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

You could use 

context.p_filename = blobName;

to store the filename/blobname to context variable.

 

Looking at your job I didn't really get what you were trying to achieve here. You are trying to pull all the blobs from the container and trying to download them. It would download all the blobs - you see you have the java code after the tAzureStorageGet to filter out blobnames which would do nothing.

 

On top of this, the error you are getting because you have connected tAzureStorageGet with iterate on tJava which doesn't know about the input_row. If you notice I had the main connecting to tJavaRow directly from tAzureStorageList. Also, I am suspecting the way you have used the blob filter won't work - simply use 

"model_outputs/"

to get all the blobs under the directory "model_outputs"

Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

Actually I am copying the file from bolb to local directory.

i need to store the file in local directory.

 

 

 

Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

I am able to get the filename in the tjava. Whereit is having all the files in the model_output folder.

 Then i need to store that particuler file to local folder. 

 

Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

How can i pick the latest file from blob.?? 

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

In that case do something like this - 

tAzureStorageGet-1.JPG

Basically, you get all the files and then put the filter condition in tJava and store the filtered file in global variable(always prefer to use Talend global variables instead of context variables if you are not using parent-child jobs and you have to pass values between those jobs). Then connect the tAzureStorageGet using "RunIf" trigger to download the selected blobs to local directory.

Java Code - 

String blobName = ((String)globalMap.get("tAzureStorageList_1_CURRENT_BLOB"));
//String p_filename = (String)globalMap.get("p_filename");
//System.out.println("Global variable: "+p_filename);

if(blobName.matches("(?i).*student_plan.*")) {
	//System.out.println(blobName);
	//context.p_filename = blobName;
	globalMap.put("p_filename", blobName);
}

RunIf trigger condition - 

!Relational.ISNULL((String)globalMap.get("p_filename"))
Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

by using the below code m getting the file name.

 

String blobName = input_row.BlobName;

if(blobName.matches("(?i).*_student_plan_.*")) {
System.out.println(blobName);
}

 

i have connected to thecontainer from tAzureStorage_list. In this container there is a folder called model_outputs in that there are multiple file.

Here i have to achive pick the lastest file from folder model_output and store that latest file with the matching file to the local folder by connecting to TasureStorageGet.

 

how can i achive this.. Your solution is helping a lot. Please let me know how to achive it.

 

 

 

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

What would be the meaning of latest file, two blobs with same name couldn't exists under one directory in a container.
Your original question was for filter blobs based on name pattern - if that has been achieved request you to mark the post as accepted solution and start another topic with the new & correct question.
Kudos are always welcome.
Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

No .. I need to pick the file with the name patter from the azure blob under the container there is a folder model_outputs. Inside the folder files will the available .Based on the filepatter we need to get and store it to local.

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

In that case the solution is already provided.
Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

the file is reading with the folder name.

how can i remove the folder name. currrntly file nam reading as mode_outputs/266363_*.csv

how to remove the model_outputs/ folder name in the code????

Nine Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

Use like this - 

String foo = "model_outputs/266363_*.csv";
System.out.println(foo.substring(foo.indexOf("/") + 1));

If you have multiple "/" in your blob filename then would suggest to use this - 

 

String foo = "model_outputs/output/266363_*.csv";
System.out.println(foo.substring(foo.lastIndexOf("/") + 1));

Please mark the topic as "solution provided" if the reply has helped you. Kudos are also welcome :-)

Six Stars

Re: how to provide dynamic file name to fetch the file from azure blob??

I am using the code which u have given above. i need to capture only the file name and assign it to the context variable after splinting the folder and filename.

find the attached screen shot.

 context.p_filename and pass the file name to tfileinputdelimiter.

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend API Services Publish to Talend Cloud

Learn how to publish your API Services to Talend Cloud

Watch Now

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now

Cloud Data Warehouse Trends for 2019

Find out aboutCloud Data Warehouse Trends for 2019

Read