Mongo Error while multiple reads and writes

[Anonymous]

My Talend job fails after about 3000 iterations of reading data from a CSV file while querying and insert/update to MongoDB. I am getting the following error. Is there a way I can tweak the collection pool or someting for Mongo within Talend? What could be the cause for the error below?

Talend Open Studio for Big Data 6.4.0.20170150_1410 on Windows 7

Exception in component tMongoDBInput_1 (Fetch_Write_TS_Data)
com.mongodb.MongoException$Network: Exception opening the socket
    at com.mongodb.DBPort.<init>(DBPort.java:117)
    at com.mongodb.DBPort.<init>(DBPort.java:95)
    at com.mongodb.DBPortFactory.create(DBPortFactory.java:28)    at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnectionProvider.java:186)    at com.mongodb.PooledConnectionProvider$ConnectionItemFactory.create(PooledConnec

To see the whole post, download it here

[Anonymous]

Hello,

 Are you creating more connections than you need? What does your whole job design look like? Did your master job start with a tMongoDBConnection and ends with a tMongoDBClose?

Best regards

Sabrina

[Anonymous]

Hi Sabrina, Thanks for your reply.

I am not opening or closing the connections in parent flow since I do not know if Talend OS can access the MongoDB Open/Close connection components of parent flow within the subflow. I am running parallel in the Iterator which is located at the end of the parentflow. I gave 50x but not sure what would be the optimal number.

Please find my flows attached.

SubFlow.PNG
ParentFlow.PNG

[Anonymous]

What is the logic you are using in the Iterator? Can you post a screen shot of the component settings? Since you are running parallel subjobs and getting an error indicating the port is already in use by another process, that component is likely where the problem is.

[Anonymous]

There isn't any logic in iterator. I am using defaults within FlowToIterate. I branched out using Iterate with Enable parallel execution and Number of parallel execution as 50. I had attached Iterator and Iterate screenshots.

Iterator.PNG
Iterate1.PNG

[Anonymous]

What happens if you turn off Parallel Execution? Do you get the same error? You might also try using fewer parallel threads if the job really needs to be run in parallel.

[Anonymous]

I am getting the same error even if I turn off parallel execution. Since the parallel execution is running the flow faster, the error shows up little early. I tried with smaller execution count, but I am getting the same error.

Is it possible that the connection setup/close can be made a singleton?