Highlighted
Four Stars

Dynamically Change the URI on the tHttpRequest Component

Hey guys, this is my first post and I am new to ETL and Talend.  I've created a new job that makes a .txt file from Mixpanel's API, cleans it up and then inserts the data into our DataWarehouse.  This job is working flawlessly, however, it is a manual process that I now need to automate.  Each day I have to go and update the URI on my tHttpRequest Component with yesterday's date to then retrieve the latest full day of data.  The URI looks like this: "https://data.mixpanel.com/api/2.0/export/?from_date=2017-08-21&to_date=2017-08-21" (and yes the dates are inclusive).  Now, I've looked at Context Variables and other ways to attempt this but none of it is making sense to me.  Is there a way that I can have the from_date and to_date update with the previous day every day automatically?  If so, the steps to accomplish this would be incredibly helpful.  I've attached a screenshot of the job if it helps any of you visual people out there.  Thank you!

1 ACCEPTED SOLUTION

Accepted Solutions
Five Stars

Re: Dynamically Change the URI on the tHttpRequest Component

You can achieve this 2 ways
1. Using global variables
2. Using context variables

Whichever way you go, the logic will remain the same.

With global varables, you update the required value (your date value) into a variable that you define within the job.

With context variables, the variable needs to be declared before hand in the 'context' tab of your job. Then this context variable's value needs to be updated in the job before your main flow starts.

Now, to update this variable, you can use a tJava component. You will need to write a bit of code to find the current date, decrement it by a day and then format it to the way you need it. This formatted value needs to be assigned to the variable that you created using either of the above 2 ways.
2 REPLIES
Five Stars

Re: Dynamically Change the URI on the tHttpRequest Component

You can achieve this 2 ways
1. Using global variables
2. Using context variables

Whichever way you go, the logic will remain the same.

With global varables, you update the required value (your date value) into a variable that you define within the job.

With context variables, the variable needs to be declared before hand in the 'context' tab of your job. Then this context variable's value needs to be updated in the job before your main flow starts.

Now, to update this variable, you can use a tJava component. You will need to write a bit of code to find the current date, decrement it by a day and then format it to the way you need it. This formatted value needs to be assigned to the variable that you created using either of the above 2 ways.
Four Stars

Re: Dynamically Change the URI on the tHttpRequest Component

Thanks for your input, this put me on the path that led to a resolution.  Here is what I did:

  1. Created a Global Variable with Key: "YesterdaysDate" | Value: TalendDate.formatDate("yyyy-MM-dd", TalendDate.addDate(TalendDate.getCurrentDate(),-1,"dd"))
  2. Trigger OnSubjobOk to my tHttpRequest Component
  3. Set URI to the following: "https://data.mixpanel.com/api/2.0/export/?from_date=" + globalMap.get("YesterdaysDate") + "&to_date=" + globalMap.get("YesterdaysDate")

This helped me iterate through a date value in a tHttpRequest Component's URI perfectly.  Thanks again!