[Solved] Talend REST service prevent context variables overwritten by simultaneous requests

[ajkerr]

It's been some time since I've developed regularly in Talend, so happy to rework the job as much as possible to achieve this. This is 100% an issue with how I've designed the job and not a Talend bug.

I have a Talend Studio REST job that works correctly when requests are processed one at a time, however when two requests are sent almost simultaneously, request/response context variables responsible for passing logging info can be overwritten while the first request is still running. This causes data from one operation to be mixed into the other request’s processing, logging, or response. 

Is there a recommended Talend native way to keep request specific values isolated for the full lifetime of each REST request, or to make a new request wait until the current request has finished processing?

I understand Routes could potentially solve this issue, but jobs work better from a design perspective for our current goals. I'm thinking of scrapping storing the logging data in context variables and passing it all through the main component flows to avoid subsequent calls overwriting the data.

When requests overlap, those context values may be replaced by the second request before the first request reaches the update Joblet. This can cause the wrong log record to be updated or cause response and logging data to become mixed between requests.

For further context, at the beginning of each request, an insert logging Joblet creates a database record with values such as an ID, timestamp, operation, request details, and initial status.

At the end of the flow, an update logging Joblet updates that same record with the final response, HTTP status, outcome, duration, error details, and completion timestamp.

Screenshot of part of the job attached.

Any help would be much appreciated. Thanks in advance.

[XRocq1696001865]

We’d like to replicate your design, but globalMap and context aren't thread-safe, so they can't prevent race conditions during simultaneous calls.

Also, regarding the OnComponentOk link on tRequest: it seems intended to trigger the log table insert first, but the main output flows (Get, Post, WrongCall) will actually execute before it.

Since only the row (Main) links are thread-safe, they can't be bypassed. If I were you, I’d place a tMap right after your tJavaRow with two outputs:

• Output 1: to your tRESTResponse components.

• Output 2: to your database update step.

Both outputs would simply pass through the same values originating from the tJavaRow

Be Careful, even like that and with order 2 for the DB update output, your tRestResponse will always wait for the dataBase update complete because you stay into the row(main) end to end

[ajkerr]

Hi there, thanks for your reply. Since I created this approach, I've gone through a couple of design iterations, both of which were possible because I discovered a really handy feature for tRestRequests that seems hardly documented, and I wanted to share its potential use cases. 

The following I typed up to my team to explain.

By checking the "Use Business Correlation" checkbox, a global (flow-only) CORRELATION ID is exposed with the following description as per Talend's documentation

The documentation is pretty sparse, and I couldn't find any additional info in the community about it, so I thought I'd share.

Select this checkbox to enable the correlation option so that chained service calls will be grouped under the same correlation ID. tRESTRequest will extract the correlation ID from the request header and store it in the component variable for further use in the flow.

If this option is not enabled on the client side, a correlation ID will be generated automatically in tRESTRequest.

That means the Correlation ID can be configured before the client uses the endpoint, allowing us to set up the job in advance.

My use case is to insert the Correlation ID into a database record to log an initial request body, and before the response, update the same record with response details. Because this ID is live for the flow, I can call the Correlation ID right before the log update, rather than passing it through the entire operation flow. It turns out it's actually impossible to achieve this in Talend without this feature.

To call the Correlation ID, the easiest way I've found is inside a tJava(Row) component; hit CTRL + Space to pull up the available variables and functions.  Scroll down till you find the equivalent version for your job/route (It was quite far down the list for me): 

The variable, as mentioned in the description, will take the name of your initial sub job component followed by its default name.

((String)globalMap.get("tRESTRequest_1_CORRELATION_ID"))

In my scenario, I assign it to a variable at the beginning and end of an operation flow, which is then passed as the primary key for a DB Insert and Update as mentioned above. This provides a direct association between the DB log and the flow that cannot be broken by simultaneous client requests overlapping.