Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
hi,
could you read all fields as String, store somewhere and filter your data 'later' ?
it's often easier&helpful to extract raw data as String only.
If you don't have to do some calculation on your data, never mind to read number as a string
regards
could you read all fields as String, store somewhere and filter your data 'later' ?
it's often easier&helpful to extract raw data as String only.
If you don't have to do some calculation on your data, never mind to read number as a string
regards
Thanks! I'm already trying this it seems to work
Now I have another problem, I wanna connect those 2 CSV Files with tMap and output one combined CSV File, but it seems to be too much Data because it needs extraordinary long and only gets ~1462 rows/s and stops after 150 seconds with the Error
"Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"
Any idea on that?
Regards
Now I have another problem, I wanna connect those 2 CSV Files with tMap and output one combined CSV File, but it seems to be too much Data because it needs extraordinary long and only gets ~1462 rows/s and stops after 150 seconds with the Error
"Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded"
Any idea on that?
Regards
I wanna connect those 2 CSV Files with tMap and output one combined CSV File
what kind of 'combination' ? are you doing some mapping, join filter. ?
Do you need all the data to be in the tMap ?
First thing, it's to manage Only the data you need => filter, extract data to keep the right ones.
You can also increase allocated memory of your jvm.
GC overhead limit exceeded mean that Garbage collector is working too hard, too often
regards
laurent
what kind of 'combination' ? are you doing some mapping, join filter. ?
Do you need all the data to be in the tMap ?
First thing, it's to manage Only the data you need => filter, extract data to keep the right ones.
Join filter.
No, i'm filtering Data in the tMap, seems to be a bad idea
What components are good for filtering? I'm trying tFilterColumns and tFilterRow.
Thanks for all that help!
i'm filtering Data in the tMap, seems to be a bad idea
it's depend on several things ...
but if you're made some join, it's better to filter before tMap.
be aware that all data from lookup flow are store in memory by default . it could be a reason for your "out of memory".
Are you constraint by the memory that you can allocate to your jvm (base on the production environment) ?
A better way, if it's not a "real-time" application , could be filter data in a job & store in another file or tables (I/O with mysql isam engine could be a good solution).
Read the filtering data and join your data in a separate job.
When you got some problem with memory, try to cut treatment in to several jobs.
Using tables to store raws data and make filter with a where clause can also be a solution.
When you haven't got a lot of allocated memory for your jvm, store data in database can help you.
for example by using ELT component to join data you will work with DB engine ressources => less needed memory for your jvm.
Keep in mind that you have to optimize your Talend job before thinking to increase JVM memory
https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669?content...
great , put post as resolved if it's the case