I have created Excel metadata and reading one sheet and store it in a DB. I have been using Memory Consuming (user mode).
Our sysadmins complained about the high memory usage (duh..) and asked me to look at optimizing it. I saw the Event mode where the file would be read as a stream.
In Memory consuming mode, based on metadata, data is read from row 8 till the first blank line. It works awesome
In Event mode, all rows are being read. I think the meta-data is not being honored.
All items are the same except for the memory mode. What is going on here? What am I missing here? Please help. Any example that differentiates the two modes will be appreciated.
Thanks in advance.
Yes, I did see that. I still don't see much input on how reading excel file as a flow works vs the memory consuming mode. Any tips will be appreciated.!
In your job there is actually (except this checked option) no difference. The component reads the file not completely at once instead it traverse through the Excel (XML based file) and returns row by row without keeping them in the memory.
In your job there is actually (except this checked option) no difference. The component reads the file not completely at once instead it traverse through the Excel (XML based file) and returns row by row without keeping them in the memory.
I noticed one thing which the low memory mode does not allow.
"Stop reading on encountering empty rows"
In memory consuming mode, this option is available for selection. This is causing my code to work incorrectly in the two modes.
There is an alternative to the build-in Excel components:
tFileExcelWorkbookOpen + tFileExcelSheetInput in Talend Exchange.
The tFileExcelWorkbookOpen component provides an option memory saving mode.
This option works different from Talends methods. It keeps a fix amount of rows and release the rest of them. This is done by a special kind workbook made by the Apache developers.
By the way, the tFileExcelSheetInput is much more powerful as the Talend build-in components.
Visit the site:
https://www.talendforge.org/exchange/index.php and search for fileexcel -> you will see the whole bunch of components dealing with Excel -> all are wide spread used in various projects (some of them have enabled projects for me).
There is an alternative to the build-in Excel components: tFileExcelWorkbookOpen + tFileExcelSheetInput in Talend Exchange. The tFileExcelWorkbookOpen component provides an option memory saving mode. This option works different from Talends methods. It keeps a fix amount of rows and release the rest of them. This is done by a special kind workbook made by the Apache developers. By the way, the tFileExcelSheetInput is much more powerful as the Talend build-in components. Visit the site: https://www.talendforge.org/exchange/index.php and search for fileexcel -> you will see the whole bunch of components dealing with Excel -> all are wide spread used in various projects (some of them have enabled projects for me).
