Six Stars

tLoop using While condition on Context variable

Hello,

I've been trying to get an API call that returns JSON to loop through result pages and am not having any luck. The API allows 250 records returned and at the end indicates a 'vidoffset' that is used to determine starting point of next API call and 'hasmore' indicating if there are more records (true/false).

I am able to iterate through the vidoffsets and get additional data but when I hit the last one (where hasmore = false) my tLoop keeps going...and clearly won't stop.

The parent job (tLoop - > tRunJob) doesn't seem to be picking up the context variable I've set in the child job for 'hasmore'. I'm attempting to use the While loop type and both my Declaration and Iteration values are blank. I've only set the Condition value (context.hasmore != "true") in an attempt to stop when hasmore no longer = 'true'

 

In the child job I am establishing the context variable of context.hasmore:

- sql that stores last iteration values -> TFlowToIterate -> tFixedFlowInput -> tSplitRow - > tContextLoad). This works fine.

I pass the vidoffset and other parameters into the next 2 subjobs:

- tRestClient -> tmap -> tExtractJSONFields -> tMap -> SQL output  -> client data I need in reporting. This works fine

- tRestClient -> tmap -> tExtractJSONFields -> tMap -> SQL output -> data that I need for next iteration. This works fine.

 

If someone has advice on passing the context variable to tLoop in a manner which tLoop will recognize and stop iterating I would greatly appreciate it. I've tried to attach images of both the parent and child jobs.

Thanks,

Ryan

 

1 ACCEPTED SOLUTION

Accepted Solutions
Six Stars

Re: tLoop using While condition on Context variable

Alas...one of the first things you taught me! I've changed the conditional assignment above to:

if (((String)globalMap.get("row2.Childhasmore")).equals("false"))
{
((java.util.Map) context.sharedMap).put("Childhasmore",((String)globalMap.get("row2.Childhasmore")));
}

 

That did it! Thank you!

17 REPLIES
Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,

Returning value from a child job using context variable is not as simple as it looks like because of Java rules.

Here is an explaination https://www.talendbyexample.com/talend-returning-values-from-subjobs.html.

However, you should never compare strings variables using the = operator.

Instead you should use the String.equals or String.equalsIgnoreCase methods (but this will not solve your case):

"true".equals(context.hasmore)

Let us know.

 


TRF
Six Stars

Re: tLoop using While condition on Context variable

Thank you. I'll see if I can wrap my brain around passing variable up to my parent job. Unfortunately I am not very familiar with Java but this could be my crash course.

 

It does appear that I'm handling context.hasmore incorrectly. I have tried !"true".equals(context.hasmore) in the while loop Condition and see the same behavior (not getting the updated context.variable from the child job and making it available in the tLoop while condition).

Is there a better way to accomplish this? I was under the impression that when using tLoop I needed to iterate over a child tRunJob and clearly that requires passing a modified context.variable up to the parent job.

To reiterate, I am attempting to collect data from an API that only allows 250 rows per request. At the end of each JSON set returned are 2 fields that can be used to determine if there are more records (hasmore - true/false) and where to start my next API call (vidoffset). My thought was to iterate using tLoop on the subjob. The first call I need to set hasmore = true so the subjob will be called at least once. The subjob will then make the API call using the previous run's vidoffset. Once complete if the current iteration hasmore = true I need to run again...thus the need to pass the hasmore variable back up to the parent job to make that decision.

 

Your suggestions are much appreciated.

 

Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,
As written in my previous answer, using "String.equals" is not a solution to solve your case but you need to apply this each time you have to compare 2 string variables.
To solve your case, follow the explainations from "Talend by example". You can start by copying the example under "ReturnValueExample (Parent Job)". As soon as you'll get the result, you'll be able to repeat the same in your job (and child job) and probably solve your case.

TRF
Six Stars

Re: tLoop using While condition on Context variable

Thanks again!

I am able to get the example working just fine. When trying to translate that logic to my job I'm having some difficulties.

I've attached a few screen shots to give you an idea of how I've tried to incorporate:

ReturnValuetoLoop.png - here I've added the tJava component where I am attempting to instantiate ConcurrentHashMap. I assumed here this needed to be the context variable I was assigning in the Child job.

ReturnValuetoLoop_details.png - this show how I am attempting to use the 'hasmore' variable in the While loop Condition. From what I understand this needs to be the condition where looping stops (i.e. where hasmore is false).

ReturnValuetoLoop_child.png - this probably needs explaining.

the first job reads a DB table and assigns context variables (hasmore, hasmore_int, and vidoffset). It is reading this from a table that is updated at each run and assigns latest vidoffset and hasmore.

the second subjob calls the API (using vidoffset from prior step) and gathers data needed (client information)

the third subjob calls the same API with the same vidoffset to get the new vidoffset and hasmore variable. I did it this way because I was having trouble parsing the JSON returned. 

 

At the very end of the third subjob is where I believe I need to assign the value that will be returned to the parent...hasmore. As you can see I am not really sure how to assign this as the example provided is a bit different (creating a string and adding the tFileOutputDelimited_1_NB_LINE on the end). I need to assign the current value of hasmore to the context in order to map back to the parent call.

 

Hopefully this makes sense.

 

Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,

Try to follow these steps.

 

In main job

Declare a context variable called "hasmore" with the datatype "object":

Capture.PNG

Don't touch your tJava_1 as it is ok.

Define a Context Param called "hasmore" in your tRunJob:

Capture.PNG

Change the tLoop condition like this:

!((Boolean) ((java.util.Map) globalMap.get("hasmore")).get("hasmoreChildJob"))

 

In the child job

Change your tJava_1 like this:

((java.util.Map) context.hasmore).put("hasmoreChildJob", false);

Set the "hasmoreChildJob" variable to true as soon as the condition to stop the loop is reach (this is your reponsability to find the place to do that but you probably need an other tJava component somewhere in your child).

 

Hope this helps.


TRF
Six Stars

Re: tLoop using While condition on Context variable

Thank you and my apoloqies for the back and forth.

In the main (parent) job I've got the context variable 'hasmore' set to an object type. I've left tJava_1 as is. I also have the Context Param of 'hasmore' set to globalMap.get("hasmore").

In the tLoop condition it looks like you've suggested this will break the loop when 'hasmore' does not equal 'hasmoreChildJob'.

!((Boolean) ((java.util.Map) globalMap.get("hasmore")).get("hasmoreChildJob"))

I have 2 questions here:

- On the first iteration I don't believe either 'hasmore' or 'hasmoreChildJob' have yet been assigned. And after making the suggested changes I'm getting a NullPointerException on this node. So on the first iteration do I need to set 'hasmore' and 'hasmoreChildJob' to true prior to entering the loop to ensure at least one iteration?

- Presuming the condition has not been met ('hasmore' = true and 'hasmoreChildJob' = true) I enter the child job. 

First I read an Aurora table that picks up the vidoffset from the last run and assigns to context variable 'vidoffset' (tells subsequent step API call where to start). 

vidoffset.PNG

----> SQL is "select  vidoffset from incr_table where build = 'Contacts'" ---> assign to  vidoffset using (String)globalMap.get("row10.vidoffset") and a tContextLoad

 

Then I call the API using context.vidoffset from first step to gather Client information and write to Aurora table:

ClientAPICall.PNG

Then I call the API a second time using context.vidoffset from first step to get new vidoffset and hasmore that is returned at the end of the current JSON data. I write both of these to the incr_table where I read from each iteration (I don't think I really need the hasmore value written to a MySQL table here because as you're suggesting I should be passing that using variables as the job runs) but I do need the vidoffset because I'll need this on the next iteration - whether that is within this job execution or the next day when the job is running again and I want to start from the vidoffset stored in incr_table :

SecondAPICall.PNG

Also, as you can see above and as you have suggested, I have another tJava that I think is assigning the hasmoreChildJob as false to the object?

((java.util.Map) context.hasmore).put("hasmoreChildJob", false);

- What I think I need here is to set hasmoreChildJob to the hasmore result coming out of the JSON file (that I'm writing to a MySQL table now). This then gets passed back to the parent job and if true runs loop again, if false then loop terminates.

 

I need a suggestion on how to do that too. You mentioned that I need to decide where to change hasmoreChildJob but technically I think I need to assign 'hasmoreChildJob' at each iteration and pass that back to parent. So initially, start the parent job with both set to 'true' so it enters the loop and then change to value of hasmoreChildJob in the child job to the value in the JSON data at each iteration?

 

Does that make sense? 

 

 

 

Six Stars

Re: tLoop using While condition on Context variable

Hello,

Just seeing if there are any other suggestions here.

Thanks

Six Stars

Re: tLoop using While condition on Context variable

Anyone?

Six Stars

Re: tLoop using While condition on Context variable

An update:

((java.util.Map) context.hasmore).put("hasmoreChildJob", false); 

This results in an error: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map

 

Six Stars

Re: tLoop using While condition on Context variable

Hello,

I was hoping you could respond to the question I had earlier. I am able to return the subjob variable to the parent and see it's changed value. The only hold up here is in the tLoop while condition:

 

- On the first iteration I don't believe either 'hasmore' or 'hasmoreChildJob' have yet been assigned. And after making the suggested changes I'm getting a NullPointerException on this node. So on the first iteration do I need to set 'hasmore' and 'hasmoreChildJob' to true prior to entering the loop to ensure at least one iteration?

Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,
To avoid the null pointer exception, add a tSetGlobalVar to initialize your variables.
Hope you can now mark your case as solved.
Kudos also accepted.

TRF
Six Stars

Re: tLoop using While condition on Context variable

Thanks! I've been trying that but getting additional errors. Below you see how I'm setting hasmore and Childhasmore (changed the name of subjob return value).

The error I get on tLoop is "java.lang.Boolean cannot be cast to java.util.Map".

If you recall my while condition is: !((Boolean) ((java.util.Map) globalMap.get("hasmore")).get("Childhasmore"))

SetGlobalVar.PNG

Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,

 

Sorry, wasn't a good idea to initialize those global variables.

You just have to check if "Childasmore" is null or not:

((Boolean)((java.util.Map)globalMap.get("hasmore")).get("Childasmore")) == null || 
((Boolean)((java.util.Map)globalMap.get("hasmore")).get("Childasmore"))

At the 1st ireration "Childasmore" is not defined (null), so the child job is called.

The loop will stop as soon as the child job return false for "Childasmore".

 

Finally, you must remove tSetGlobalVar and you can remove tJava2 too if you declare the hashmap using tLoop Declaration parameter.

Here is how it is one my side:

 

Capture.PNG

Hope this one is the good one!


TRF
Twelve Stars TRF
Twelve Stars

Re: tLoop using While condition on Context variable

Hi,
Does this solve your case.
If so, please select the better answer to close it.
Kudos also accepted.

TRF
Six Stars

Re: tLoop using While condition on Context variable

It's very close. I was working on it yesterday and first implemented the suggestions you had above. I quickly realized that the Childhasmore variable I'm returning from the subjob was not of type Boolean. I was returning a String which would cause a Java error (java.lang.String cannot be cast to java.lang.Boolean) in tLoop where you suggested my condition be:

((Boolean)((java.util.Map)globalMap.get("sharedMap")).get("Childhasmore")) == null || ((Boolean)((java.util.Map)globalMap.get("sharedMap")).get("Childhasmore"))

 

I tried changing the subjob to return a Boolean for Childhasmore but am not having luck getting that to update sharedMap. So I need to either convert it to Boolean in order for it to work in tLoop condition, or I need to compare Strings in tLoop instead of Boolean. 

But I'm also going to have to change the tLoop logic because you mentioned on the first iteration (where Childhasmore is null) the loop will run, which it does. Then once Childhasmore has been set (not null) it will break the loop. However, it will be set to True in many cases and the loop needs to run again if that is the case. I think (once I'm able to pass Childhasmore to the condition properly) I will need to adjust the logic because once it's not null I believe the first check will cause the loop to break:

((Boolean)((java.util.Map)globalMap.get("sharedMap")).get("Childhasmore")) == null

 

I'm thinking I could conditionally set Childhasmore in the subjob (if True then don't update it) and the loop will run again?

((java.util.Map) context.sharedMap).put("Childhasmore",((String)globalMap.get("row2.Childhasmore")));

 

So open questions now

- Better to get subjob to return Boolean and update Childhasmore OR in tLoop while condition somehow compare Strings instead of Boolean?

- Since subjob will return True and need to run tLoop again how to handle it not being null and stopping the loop with condition of Childhasmore == null.

 

Thanks again for thinking through this. I'm learning a lot in the process ;')

 

Six Stars

Re: tLoop using While condition on Context variable

Here is what I'm trying to do in the variable assignment in the subjob. It's not assigning though when Childhasmore = false

if (((String)globalMap.get("row2.Childhasmore")) == "false")
{
((java.util.Map) context.sharedMap).put("Childhasmore",((String)globalMap.get("row2.Childhasmore")));
}

Six Stars

Re: tLoop using While condition on Context variable

Alas...one of the first things you taught me! I've changed the conditional assignment above to:

if (((String)globalMap.get("row2.Childhasmore")).equals("false"))
{
((java.util.Map) context.sharedMap).put("Childhasmore",((String)globalMap.get("row2.Childhasmore")));
}

 

That did it! Thank you!