Latest files and dynamic naming of files

Four Stars

Latest files and dynamic naming of files

I'm working on a talend job that makes an HTTP GET request to get multiple PDF documents. The request returns a JSON file consisting of: documentDate and documentLink. I want to get the latest documentLink(s) from the documentDate and upload the document(s) with a FileFetch component with the filename being "Document_1" if only one document exist of the latest date. If 2 documents have the latest date then the FileFetch component should upload 2 documents with one filename being "Document_1" and the other "Document_2" I'm unsure on how to loop over the JSON file to get the latest date and naming the document(s) correctly.

What I have done so far:

 tHTTPRequest_1 --> tExtractJSONFields_1 --> tXMLMap_1 --> tFileFetch_1

This works uploading one file, but there is no check made for the latest documentDate or naming of the filenames in the tFileFetch_1 component.

The returned JSON looks like this:

{
"documents": [
    {
        "documentDate ": 200119,
        "documentLink": "someLink1",
    },
    {
        "documentDate ": 200119,
        "documentLink": "someLink2",
    },
    {
        "documentDate ": 150119,
        "documentLink": "someLink3",
    }
   ]
}

Do you guys have any idea on how to solve this problem?

Six Stars

Re: Latest files and dynamic naming of files

Hi @jens123 ,

 

I considered the given example/test data as raw input and tried to append sequence numbers at the end of the documentlink value. You can modify the file name as per your requirement. This is one way of forming and we have many ways to do this. Attached job screen shot, and it is not fully completed.

 

tExtractJsonFileds --> tSortRow --> tMap --> tMap --> tLogrow(in your case tFileFetch)

1. Data is extracted from tExtractJsonField

2. Sort data on date column

3. Generate Sequence number by using Numeric.Sequence(row.date, 1,1)

4. Form the destination file in another tMap

 

 

 

 

Four Stars

Re: Latest files and dynamic naming of files

Hi @hpamidipola 

 

Thank your for your answer. I got so far, 

tExtractJsonFileds --> tSortRow

 Regarding the Sequence number by using Numeric.Sequence(row.date, 1,1), how can I use the "row.date" in the Sequence method when it requires a String. Furthermore, how do I get the last date only?

Six Stars

Re: Latest files and dynamic naming of files

Hi @jens123 ,

 

Please give us the example source file name and target file name. So that i will make  a job and will post it in community.

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch