Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
My Talend job fails after about 3000 iterations of reading data from a CSV file while querying and insert/update to MongoDB. I am getting the following error. Is there a way I can tweak the collection pool or someting for Mongo within Talend? What could be the cause for the error below?
Talend Open Studio for Big Data 6.4.0.20170150_1410 on Windows 7
Exception in component tMongoDBInput_1 (Fetch_Write_TS_Data)
com.mongodb.MongoException$Network: Exception opening the socket
at com.mongodb.DBPort.<init>(DBPort.java:117)
at com.mongodb.DBPort.<init>(DBPort.java:95)
at com.mongodb.DBPortFactory.create(DBPortFactory.java:28) at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186) at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnec
Hello,
Are you creating more connections than you need? What does your whole job design look like? Did your master job start with a tMongoDBConnection and ends with a tMongoDBClose?
Best regards
Sabrina
Hi Sabrina, Thanks for your reply.
I am not opening or closing the connections in parent flow since I do not know if Talend OS can access the MongoDB Open/Close connection components of parent flow within the subflow. I am running parallel in the Iterator which is located at the end of the parentflow. I gave 50x but not sure what would be the optimal number.
Please find my flows attached.
There isn't any logic in iterator. I am using defaults within FlowToIterate. I branched out using Iterate with Enable parallel execution and Number of parallel execution as 50. I had attached Iterator and Iterate screenshots.
What happens if you turn off Parallel Execution? Do you get the same error? You might also try using fewer parallel threads if the job really needs to be run in parallel.
I am getting the same error even if I turn off parallel execution. Since the parallel execution is running the flow faster, the error shows up little early. I tried with smaller execution count, but I am getting the same error.
Is it possible that the connection setup/close can be made a singleton?