Essentially you need to follow this process.....
1) Create a globalMap variable to hold your webservice call URL. Do this in a tJava.
2) Connect the tJava to a tLoop using an OnSubJobOk link.
3) Set the loop to be based on the presence of a URL value in globalMap. While globalMap.get("url")!=null, for example.
4) Connect your tLoop to whatever web service component you want to use.
5) After the webservice call, use a tExtractXMLField component to retrieve your data and the pagination URL. Put the pagination URL into the globalMap variable. If there is no pagination URL (because it is the last page), set it to null.
This is a high level explanation of how to achieve this with A LOT of assumptions as to how your web service works. You should be able to tailor this to your web service.
The step by step method I suggested will work. You just need to change step 5 to calculate how many records you have received and use that to change your URL or stop the tLoop.
I have knocked up an example job. This does not use a real REST service, because I couldn't find one suitable (for pagination), but it should give you an idea. Look at the code in the tJava components (it is described there) and look at the tLoop config. You will have to work some of the logic out for yourself with your service (working out how many records are returned, how many are left, etc), but the logic shown in this example should easily be able to be extrapolated from.
The example was written in Talend v6.5.1. If you are using a younger version, download anything above 6.5.1 and you can look at it there.
I am doing the same thing in one job, but when i am extracting json data from tExtractJson using Xpath and there was no response from the RestAPI link, it should extract null values, and my loop should end after that. Please see the Below image -
This is how i am extracting values. The response from postman is below :-
And ModifyURL( tJavaflex) component i am modifying my URL and storing the offset value coming from tExtractJson. offset value is used to end the loop.
But as i am not getting any values, offset should be null and loop should end. But in this case loop is not getting end. It's keep on iterating, i think because V_loop value is not getting set as false.
Please see the below response in tlogRow : -
tlogRow_6 is the values i am getting after tExtractJson.
This is how i am setting my loop.
Also i am not able to see, my tJavaflex println values on the console. I don't know why, please help me in this.
Why do you have this code....
....it looks like it should be more like....
This might not be the only issue. If your code compiles and runs, it means that you have a context variable called "offset". What is that supposed to do and where is it set?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema