Multiple issues with Talend 6.1.1 if amount of rows increases

[Anonymous]

I upgraded Talend 5.1.2 (r90681) to 6.1.1.20151214_1327. This due to the fact I needed to convert UTC to GMT and vice versa, which was not possible in 5.1.2.
But in this version I have multiple issues, all based on how much records which have to be processed.
In general all jobs work perfect, until the throughput is above a several hundreds, of several thousands of rows. Then I get errors on drivers, or in this case wrong sorting and garbage in .csv output.
Source files:
http://  www . filedropper . com / statussenentijdennaarchainware02
If component cwReturnJobLogs1 has this where clause:
WHERE ID > 161670059
  and ID < 161760492
All goes smooth and I get the result I want, in the right order. (Result_1.csv)
But if the where clause is changed to:
WHERE ID > 161640059
  and ID < 161763492
The result isn't in the correct order. (Result_2.csv)
If you look in the .csv files, you can search for 25050690. In the correct order, the second line says: LO START and later in the file LO GEREED. But in the second result file, you first see LO GEREED and then LO START.
The Where clause if normally dynamic. Every 15 minutes the job is started, retrieving the MAX_ID of the last session and processing all ID since then, saving the new MAX_ID in the database. So the amount of rows processed can be between 0 and thousands of rows.
In 5.1.2 we never had these issues.
If something is not clear, ask!

[Anonymous]

How can I resolve this??

[Anonymous]

Anybody ?

[Anonymous]

This is a very difficult post to answer since there is no job example to look at. You say that this is affecting a lot of your jobs once the number of rows go over a few hundred. Is it possible to maybe step through an example with screenshots of your job and maybe that will give us a clue. Also, have you checked that your environment is supported for the version of Talend you are using?
I have to be honest and say that this is likely a problem with your environment or with your jobs, since v6 is being used by a lot of people for jobs that process hundreds of thousands of rows. There have been some installation issues with v6, but they have usually precluded people from doing anything with it. 

[Anonymous]

I have posted a link in my message which is a link to a Zip file (I had to do with spaces, because I was not allowd to add live URL's)
This zip file contains the source data, the output data and the project itself.
If it's an environment issue, where do I start looking to find the problem?
Windows 7, Talend 6.1.1

[Anonymous]

I can't get anything from that link I am afraid. It just takes me to an upload page. Can you use the "Upload" button here to share your project?
About the environment, first check that your environment (including Java, DB, etc) are covered in the system requirements found here (https://help.talend.com//pages/viewpage.action?pageId=264282428). Click "Next" and see how your environment compares.

[Anonymous]

As far as I can see, my environment is in sync with the requirements.
Are there requirements regarding SQL Server? (SQL server 2008R2)

[Anonymous]

File 1:
StatussenEnTijdenNaarChainware.zip_20160425-0400.zip
File 2:
Data.zip_20160425-0400.zip

[Anonymous]

I'm afraid I don't know the requirements for SQL Server off the top of my head. I suspect that 2008 would be OK though...even if technically not supported.
I will take a look at your job and get back to you. But I won't be able to do it immediately. 
.......that doesn't stop anyone else from jumping in if you think you may have seen these symptoms before 🙂

[Anonymous]

I've just had a quick look and have spotted something that is not going work as I suspect you expect. Now, I don't have SQL Server on my current machine so have only been able to take a quick look. But the area which I have put a red box around is a problem.....

I assume that you are using the Iterate to call the next query once per LopendID. The query it is firing is below....

"SELECT ID,
   TripID,
   JobID,
   JobReference,
   LogTime,
   LogTimeZone,
   LogType,
   LogCode,
   TemplateMessageID,
   EAIRecordId,
   LogStatus,
   Latitude,
   Longitude,
   PositionTime
FROM	CarrierWeb.dbo.cwReturnJobLogs
WHERE ID=" + context.LopendID

The context.LopendID is set in the tJavaRow which is connected by a flow. You cannot guarantee the timing of this. Iterates and Flows work very different. What you need to do here is use a tFlowToIterate and then pass the globalMap variable which is generated for the tFlowToIterate, to the next query.
I suspect that you issue is related to either this logic, or a similar type of issue.