Four Stars

Downloading files of a sub-folder in azure storage container is saving folders as empty text files

Hi,

 

I am using TOS_BD_V6.4.1 on Ubuntu. I am trying to download files from a azure storage container using tAzureStorageGet. My folder structure in azure storage is "dev/TestFolder/inside/abc.xlsx" (dev is the container) and I am trying to download all the files in "TestFolder" directory to a local folder "Test" in downloads directory.

But when I run the package, it is creating a file "TestFolder" inside "Test" directory in my local and then throws "/*/*/inside Not a directory" error.

Below is my tAzureStorageGet config:

talend.pngBelow is the error:
error.png

 

Please help and thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

You have to use the global variable of the tAzureStorageList component.

Can you try giving this value - "((String)globalMap.get("tAzureStorageList_1_ACCOUNT_KEY"))" in the prefix and try again?

AzureGet2.PNG

10 REPLIES
Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

I think in most cloud storage technologies, there is no concept of a 'sub-folder' which is created explicitly.

For example, the name of the file abc.xlsx is actually "TestFolder/inside/abc.xlsx".

So in your component you can download only one file at a time. What I would suggest is to use the tAzureStorageList Component to get all the files names under the TestFolder prefix and connect the iterate component to the tAzureStorageGet Component to download the files.

AzureGet.PNG

 

Four Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

Hi ArvinRapt, thanks for your response.

Earlier, I was following the example that was mentioned here. Even when I use iterate to download the files list as you mentioned, i'm getting the same error. But on iterate over the list i can see the names of all the files and folders in the console using tJava.

Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

can you post a screenshot of the tAzureStorageGet component and how its connected to the iterate link?

That'll be more helpful.

Four Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

ArvinRapt, thanks for your time.

Below is my tAzureStorageGet component

tazurecomp.png

Here is the flow:

 

job-1.png

 

I noticed that if i put the entire file path of the file in prefix ("TestFolder/inside/abc.xlsx") it is downloading that file but i want to download all the files in the directory.

Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

You have to use the global variable of the tAzureStorageList component.

Can you try giving this value - "((String)globalMap.get("tAzureStorageList_1_ACCOUNT_KEY"))" in the prefix and try again?

AzureGet2.PNG

Four Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

ArvinRapt,

I am getting "Can't open the specified file input stream from file" error on using the global variable of the tAzureStorageList component as shown below.new-error.png

 

Four Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

ArvinRapt,

 

I removed the test folder and tried with a new folder and it worked. But it is downloading all the files under dev container (other folders as well). Can you please let me know how i can limit it to download from one folder.

Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

Using the tAzureStorageList component? or by just using the tAzureGet Component?

Either way I would suggest checking the value that you are passing to the prefix parameter.

Four Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

ArvinRapt,

I used ((String)globalMap.get("tAzureStorageList_1_CURRENT_BLOB")) in tAzureStorageGet as prefix and iterated on the tAzureStorageList_1 list. It solved my issue.

Thanks a lot for guiding me through.

Seven Stars

Re: Downloading files of a sub-folder in azure storage container is saving folders as empty text files

That's great. You are welcome.