Latest files and dynamic naming of files

Four Stars

Latest files and dynamic naming of files

I'm working on a talend job that makes an HTTP GET request to get multiple PDF documents. The request returns a JSON file consisting of: documentDate and documentLink. I want to get the latest documentLink(s) from the documentDate and upload the document(s) with a FileFetch component with the filename being "Document_1" if only one document exist of the latest date. If 2 documents have the latest date then the FileFetch component should upload 2 documents with one filename being "Document_1" and the other "Document_2" I'm unsure on how to loop over the JSON file to get the latest date and naming the document(s) correctly.

What I have done so far:

 tHTTPRequest_1 --> tExtractJSONFields_1 --> tXMLMap_1 --> tFileFetch_1

This works uploading one file, but there is no check made for the latest documentDate or naming of the filenames in the tFileFetch_1 component.

The returned JSON looks like this:

{
"documents": [
    {
        "documentDate ": 200119,
        "documentLink": "someLink1",
    },
    {
        "documentDate ": 200119,
        "documentLink": "someLink2",
    },
    {
        "documentDate ": 150119,
        "documentLink": "someLink3",
    }
   ]
}

Do you guys have any idea on how to solve this problem?

Four Stars

Re: Latest files and dynamic naming of files

Hi @jens123 ,

 

I considered the given example/test data as raw input and tried to append sequence numbers at the end of the documentlink value. You can modify the file name as per your requirement. This is one way of forming and we have many ways to do this. Attached job screen shot, and it is not fully completed.

 

tExtractJsonFileds --> tSortRow --> tMap --> tMap --> tLogrow(in your case tFileFetch)

1. Data is extracted from tExtractJsonField

2. Sort data on date column

3. Generate Sequence number by using Numeric.Sequence(row.date, 1,1)

4. Form the destination file in another tMap

 

 

 

 

Four Stars

Re: Latest files and dynamic naming of files

Hi @hpamidipola 

 

Thank your for your answer. I got so far, 

tExtractJsonFileds --> tSortRow

 Regarding the Sequence number by using Numeric.Sequence(row.date, 1,1), how can I use the "row.date" in the Sequence method when it requires a String. Furthermore, how do I get the last date only?

Four Stars

Re: Latest files and dynamic naming of files

Hi @jens123 ,

 

Please give us the example source file name and target file name. So that i will make  a job and will post it in community.

 

What’s New for Talend Spring ’19

Join us live for a sneak peek!

Sign up now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads