Sequentially Iteration with updated tRESTClient-Params

unTalended — Sat, 16 Nov 2024 02:22:43 GMT

Hi @all 🙂

I am new at talend and struggling with following issue. Please be lenient with me.

The main intention is to make multiple request to a graphql interface with different values (number of objects and cursor).

I created following jobs:

job for making chuncked requests

job to execute request (executeChunkedStationsRequests)

In my approach overhead I tried to execute the requests sequentially. After every request I return the cursor from child to parent. In the parent I update the cursor value in the global map. I am delivering the global map value back to the context of the executeChunkedStationsRequests-subjob.

The issue occurs when I make more then one requests. In the first request I use a initialized default value. This works fine.

Calling the executeChunkedStationsRequests-subjob multiple times doesn't work. The context var for the cursor is filled with 'null' and occurs an NullPointerException.

tJava: initProcessingVars

globalMap.put("query_stations_leftCount", context.totalCount);
globalMap.put("query_stations_first", context.query_stations_firstDefault);
globalMap.put("query_stations_cursor", context.query_stations_afterDefault);
globalMap.put("triggerChunkedStationsRequestsFlag", true);

tJava: updateProcessingVars

//----------------------------------------------------------------------
// Updating der Vars für das Processing der Chuncked Requests
//----------------------------------------------------------------------

//----------------------------
// Updating query_cursor
//----------------------------
globalMap.put("query_stations_cursor", row23.cursor);

//----------------------------
// Updating query_stations_leftCount
//----------------------------
int query_stations_first = ((int) globalMap.get("query_stations_first"));
int query_stations_leftCount = (((int) globalMap.get("query_stations_leftCount")) - query_stations_first);
globalMap.put("query_stations_leftCount", query_stations_leftCount);


if ( query_stations_leftCount > 0){
	if (query_stations_leftCount < query_stations_first){	
		//Make request with remaining amount of stations
		globalMap.put("query_stations_first", query_stations_leftCount);
	}
	//Make request with default amount of stations
	
}
else{
	// Stop repuesting for stations
	globalMap.put("triggerChunkedStationsRequestsFlag", false);
}

tLoop: triggerChunkedStationsRequests

tLoop

tRunJob: executeChunkedStationsRequests

tRunJob

Error message:

Starte Job OpenDataContentProvider am 14:12 20/05/2020.

[statistics] connecting to socket on port 3980
[statistics] connected
null null
.--------------------.
| #1. showTotalCount |
+------------+-------+
| key        | value |
+------------+-------+
| totalCount | 1157  |
+------------+-------+

---------------------------------------------------
-- processChunkedStationsRequests (initVars)
|-> ((String) globalMap.get(query_stations_cursor)) -> none
---------------------------------------------------
---------------------------------------------------

---------------------------------------------------
-- processChunkedStationsRequests (updateVars)
 |-> row6.cursor -> null
 |-> ((String) globalMap.get(query_stations_cursor)) -> null
 |-> ((String) globalMap.get(query_stations_leftCount)) -> 1147
 |-> ((String) globalMap.get(triggerChunkedStationsRequestsFlag)) -> true
---------------------------------------------------
---------------------------------------------------
tRunJob_9 in processChunkedRequests_GraphQL call etl_opendata.executechunkedstationsrequests_graphql_0_1.executeChunkedStationsRequests_GraphQL with:

[--father_pid=NkMjsk, --root_pid=QqmQ9G, --father_node=tRunJob_9, --context=Sandbox, --stat_port=3980, --parent_part_launcher=JOB:processChunkedRequests_GraphQL/NODE:tRunJob_9, --context_param query_stations_first=10, --context_param query_stations_after=null]

Exception in component tJava_1 (executeChunkedStationsRequests_GraphQL)
java.lang.NullPointerException
	at etl_opendata.executechunkedstationsrequests_graphql_0_1.executeChunkedStationsRequests_GraphQL.tJava_1Process(executeChunkedStationsRequests_GraphQL.java:497)
	at etl_opendata.executechunkedstationsrequests_graphql_0_1.executeChunkedStationsRequests_GraphQL.runJobInTOS(executeChunkedStationsRequests_GraphQL.java:4346)
	at etl_opendata.executechunkedstationsrequests_graphql_0_1.executeChunkedStationsRequests_GraphQL.runJob(executeChunkedStationsRequests_GraphQL.java:4177)
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.tJava_19Process(processChunkedRequests_GraphQL.java:1266)
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.runJobInTOS(processChunkedRequests_GraphQL.java:2120)
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.runJob(processChunkedRequests_GraphQL.java:1953)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.tRunJob_1Process(OpenDataContentProvider.java:767)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.runJobInTOS(OpenDataContentProvider.java:1144)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.main(OpenDataContentProvider.java:961)
Exception in component tRunJob_9 (processChunkedRequests_GraphQL)
java.lang.RuntimeException: Child job running failed.
java.lang.NullPointerException: null
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.tJava_19Process(processChunkedRequests_GraphQL.java:1283)
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.runJobInTOS(processChunkedRequests_GraphQL.java:2120)
	at etl_opendata.processchunkedrequests_graphql_0_1.processChunkedRequests_GraphQL.runJob(processChunkedRequests_GraphQL.java:1953)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.tRunJob_1Process(OpenDataContentProvider.java:767)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.runJobInTOS(OpenDataContentProvider.java:1144)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.main(OpenDataContentProvider.java:961)
Exception in component tRunJob_2 (OpenDataContentProvider)
java.lang.RuntimeException: Child job running failed.
java.lang.RuntimeException: Child job running failed.
java.lang.NullPointerException: null
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.tRunJob_1Process(OpenDataContentProvider.java:792)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.runJobInTOS(OpenDataContentProvider.java:1144)
	at etl_opendata.opendatacontentprovider_0_1.OpenDataContentProvider.main(OpenDataContentProvider.java:961)
[statistics] disconnected

Job OpenDataContentProvider endet am 14:12 20/05/2020. [exit code=1]

My fundamental questions are:

Did I have choosen the right approach to solve the problem?
Why did the cursor value not updating before the next triggering of the subjob is initiated?
Why is the context value loaded with 'null' an not the last value stored in global map?

I would be very grateful for your help! And excuse my bad english! 🙂

Re: Sequentially Iteration with updated tRESTClient-Params

Anonymous — Thu, 21 May 2020 10:12:49 GMT

Hi
Using tJavaRow instead of tJava for updateProcessingVars, you can't access the value of input data flow on tJava, you can read the difference between tJava and tJavaRow in this topic.
https://community.talend.com/t5/Design-and-Development/What-is-the-difference-between-tJava-tJavaRow-and-tJavaFlex/ta-p/21505

Regards
Shong

Re: Sequentially Iteration with updated tRESTClient-Params

unTalended — Mon, 25 May 2020 10:45:40 GMT

Using tJavaRow works for me fine! Thank you very much @shong! 🙂

topic Re: Sequentially Iteration with updated tRESTClient-Params in Talend Studio

Sequentially Iteration with updated tRESTClient-Params

Re: Sequentially Iteration with updated tRESTClient-Params

Re: Sequentially Iteration with updated tRESTClient-Params