Vertica Loading Very Slow

[Anonymous]

Hi all,
I'm using TOS DI 5.2.2 in Red Hat. I have 25,000 files to load. my jobs as follows:
tfileList----> tFileInputDelimited---->tMap---->tVerticaOutput
Action on table is "Insert on table"
it's taking 35min. to load 50files!!!! and each file size is less then 5MB. But same job if i write it to file (instead of vertica) it's taking less time!!
What i'm missing here..!! i need to change any settings!!
Please suggest solution
Regards,
Akki

[Anonymous]

hi,
   thank u for reply Jlolling. Are you suggesting me to write a script to load files instead of talend job??
Regards,
Akki

[Anonymous]

Hi Akki,
Do you have indexes on your target vertica table? Because index would slow down the performance of inserts...
Vaibhav

[Anonymous]

hi all,
   Using script we loaded around 13 crore records data to 45columns table in 6min. Can we achieve this using talend jobs?? This what our concern now. because we have designed job to load table so if its not giving results as expected then we have to load uisng script only right?
Vibhav,
   we don have index on table, it columnar database so we are not creating any index for table.
Regards,
Akki

[Anonymous]

Hi Akki,
Loading data through database scripts is lightning fast and don't have much limitations... Similar performance you can't expect from ETL tool specifically based on Java... I would recommend to go for the data loading through script only, but you can enhance script to perform some auditing and logging using sp..
Vaibhav