Talend 7.2 Context variable value limit?

[Vinlogs]

I'm trying to process a sql query in redshift component on talend 7.2 and using a context variable for filter condition in the sql query, the query runs fine if there are less values(100's) in filter condition. If the filter condition values cross (1000's) then the query fails to run.

I think there is some context variable limit in talend.

Can anyone let me know what is this limit for a string type context variable and suggest any alternate ways to achieve this. Thanks in advance!

[Anonymous]

Hello,

Firstly, please make sure the query is optimized and executing based on the best query plan.

What's error are you getting when the filter condition values cross (1000's)?

Is the volume of data set very big?

Best regards

Sabrina

[tnewbie · Accepted solution]

There might be multiple ways of handling this.

Did you try putting your filter conditions in a temporary table and joining that table to your main table to filter out data and process only the data you wish to. Secondly, you can also cache your filter condition values to a thashoutput and use that as a lookup to filter out the data.

[Anonymous]

No, there are actually no limits for the size of context variables.

I recommend to printout the whole query just in case the tDBInput component fails.

You can do that because tDBInput provides a return value in the globalMap containing the actual query. Check the Outline of the component (most seen on the left side under the repository view).

System.err.println(globalMap.get("tDBInput_1_QUERY"));

(Adapt the unique id of your component here and add this code to a tJava triggered by OnComponentError of your DB input component.)