Hi
We are having major problems with data preparation. We have embedded data prep in a workflow within the studio. For the majority of our feeds, the recipe cleans and prepares the data as expected. However, there is a feed that keeps failing with an error 500. The feed in question is 5 columns wide and 10m rows. We have broken the feed up into 2m chunks and are running prep on each of the 2m chunked files.
The error received is as follows:
[ERROR] 12:14:41 match_attributes_transform_0_1.*****- Unexpected error: java.io.IOException: Server returned HTTP response code: 500 for URL: https://tdp.eu.cloud.talend.com/api/preparations/****/***
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1939) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1525) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250) ~[?:?]
at dev.pl_match_attributes_transform_0_1.****$1RequestsRunner_jlet_DataPrepExecFull_1_tDataprepRun_1_DataprepOut.sendRecords(***.java:29171) ~[classes/:?]
at dev.pl_match_attributes_transform_0_1.****$1RequestsRunner_jlet_DataPrepExecFull_1_tDataprepRun_1_DataprepOut.writeRecords(***.java:29122) ~[classes/:?]
at dev.pl_match_attributes_transform_0_1.***$1RequestsRunner_jlet_DataPrepExecFull_1_tDataprepRun_1_DataprepOut.run(***.java:29094) [classes/:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
When we run all files through in a loop we receive this error against 1 or more of the input files. If we then take a failed input file and run on its own it works. So, it feels like a memory issue but we are really struggling to identify the cause as an error 500 doesn't really provide much more than it's errored...
Any advice or direction would be really appreciated.
Dave
