topic Re: Dynamically Change the URI on the tHttpRequest Component in Talend Studio

Dynamically Change the URI on the tHttpRequest Component

Anonymous — Tue, 22 Aug 2017 20:44:50 GMT

Hey guys, this is my first post and I am new to ETL and Talend. I've created a new job that makes a .txt file from Mixpanel's API, cleans it up and then inserts the data into our DataWarehouse. This job is working flawlessly, however, it is a manual process that I now need to automate. Each day I have to go and update the URI on my tHttpRequest Component with yesterday's date to then retrieve the latest full day of data. The URI looks like this: "https://data.mixpanel.com/api/2.0/export/?from_date=2017-08-21&to_date=2017-08-21" (and yes the dates are inclusive). Now, I've looked at Context Variables and other ways to attempt this but none of it is making sense to me. Is there a way that I can have the from_date and to_date update with the previous day every day automatically? If so, the steps to accomplish this would be incredibly helpful. I've attached a screenshot of the job if it helps any of you visual people out there. Thank you!

Re: Dynamically Change the URI on the tHttpRequest Component

Anonymous — Tue, 22 Aug 2017 21:13:33 GMT

You can achieve this 2 ways
1. Using global variables
2. Using context variables

Whichever way you go, the logic will remain the same.

With global varables, you update the required value (your date value) into a variable that you define within the job.

With context variables, the variable needs to be declared before hand in the 'context' tab of your job. Then this context variable's value needs to be updated in the job before your main flow starts.

Now, to update this variable, you can use a tJava component. You will need to write a bit of code to find the current date, decrement it by a day and then format it to the way you need it. This formatted value needs to be assigned to the variable that you created using either of the above 2 ways.

Re: Dynamically Change the URI on the tHttpRequest Component

Anonymous — Wed, 23 Aug 2017 21:18:42 GMT

Thanks for your input, this put me on the path that led to a resolution. Here is what I did:

Created a Global Variable with Key: "YesterdaysDate" | Value: TalendDate.formatDate("yyyy-MM-dd", TalendDate.addDate(TalendDate.getCurrentDate(),-1,"dd"))
Trigger OnSubjobOk to my tHttpRequest Component
Set URI to the following: "https://data.mixpanel.com/api/2.0/export/?from_date=" + globalMap.get("YesterdaysDate") + "&to_date=" + globalMap.get("YesterdaysDate")

This helped me iterate through a date value in a tHttpRequest Component's URI perfectly. Thanks again!

MixpanelJob.PNG
MixpanelJobGlobalVariable.PNG

Re: Dynamically Change the URI on the tHttpRequest Component

Anonymous — Tue, 30 Oct 2018 23:18:59 GMT

Hi jswalton85,

Can you please share your configuration for thttprequest .

I am having trouble setting up the connection . This is what i used -

URI -

"https://data.mixpanel.com/api/2.0/export/?from_date=2018-09-10&to_date=2018-09-10"

Method - POST

checked Need authentication and provided user - **************

getting error as - 411 Length Required

Have also tried Content-Length:0 in header but ran into error.

Can you please share your configuration for thttprequest.

Thanks