One Star

Loop through tFileFetch pass new token in url call

Well, this scenario may be quite familiar but I couldn't find the solution. 

Scenario: I am making a REST API call through tFileFetch and I get a json out of it. I parse it to get paging token and more result through tflowtoiterate. Now if more result is equals true, I have to call the same tFileFetch component to get the new set of json using new pagination result.
I have to loop through tFileFetch until the 'more result' is false.

My approach:

Access token-pagination-tFileFetch_1->JSON->tflowtoIterate->more result=true->IF ->tFileFetch_2->JSON->tFlowtoIterate->more result=true->tLOOP [moreresult.equals{true)->tFileFetch_2->
After tFileFetch2 I have used tSetGlobalVar to put pagination as common var to pass to tFileFetch2

Also it should append to same output file when loop executes
I am not sure whether this approach is appreciable or not, please suggest any improvements if any?

Screenshot:

regards
Rakz
4 REPLIES
Fifteen Stars

Re: Loop through tFileFetch pass new token in url call

I believe I have written a tutorial which covers this scenario in it. It deals with resubmitting a call to a rest service to retrieve a song history from Spotify. This uses the idea of getting a new page token (or amended URI). You can find it here. You can download the code as well. The section you may want to look at is the "The GetMySpotifyListeningHistory Job" part. I hope it helps.
Rilhia Solutions
Fifteen Stars

Re: Loop through tFileFetch pass new token in url call

I believe I have written a tutorial which covers this scenario in it. It deals with resubmitting a call to a rest service to retrieve a song history from Spotify. This uses the idea of getting a new page token (or amended URI). You can find it here. You can download the code as well. The section you may want to look at is the "The GetMySpotifyListeningHistory Job" part. I hope it helps.
Rilhia Solutions
One Star

Re: Loop through tFileFetch pass new token in url call

r_hall,
Your article is very exhaustive and to the point, though I had similar idea in mind, your article definitely helped me.
There is follow up to this question. Once tFileFetch is complete, it downloads JSON. I am converting it to csv file inorder to append iterative data. I believe appending not allowed in JSON. My target would be Big Query and Json converted csv has record like this
1086,134,2015-05-21T19:16:45Z,10,Email Test.Subject Line Test,{"attributes":{"name":"Device","value":"iPhone"}}

6th column {"attributes":{"name":"Device","value":"iPhone"}} it being taken as 6th column {"attributes":{"name":"Device" and 7th column "value":"iPhone"}}. Also since it has "" , big query couldnt ingest data which has "", how to remove "" in CSV file?
Fifteen Stars

Re: Loop through tFileFetch pass new token in url call

You have to consider XML and JSON as the equivalent of multiple database tables. By that I mean, when you get loops or arrays in the data, you have to think of it like a join between two or more tables. The example you give below looks as if it would better suit either having two CSV files (one for the main body of data and one for the "Attributes" data) or that you consider multiplying out the number of rows of the main data to accommodate potentially many loops in complex types (which you "attributes" section is). 
However, in the example below, you can have a 1 to 1 relationship it would appear. 
How are you converting the main body of data (the section with no field names or brackets)? Are you using the tExtractJSONFields component? If so you need to make use of the "Get Nodes" and "Is Array" options where you expect complex types. Then dissect those with subsequent tExtractJSONFields components. The first component will extract all of the non complex data and the "nodes". You then pass all of that to the next tExtractJSONFields component, but use that to extract the data from the first complex type....and so on. It can be quite complicated with multiple loops, etc, but will work.
Rilhia Solutions