Retrieve dynamically schema columns from CSV

[Anonymous]

Hi to all
I have this problem. I need to create a job that get data from a CSV and load it inside a Excel file. This job will run a lot of time but the data inside CSV will not be the same: I write down 3 scenario:
SCENARIO 1) The data inside the CSV
colA;colB;colC;colTemp1;colTemp2
"...";"...";"...";"...";"..."
"...";"...";"...";"...";"..."
SCENARIO 2) The data inside the CSV
colA;colB;colC;colTemp1;colTemp2;colTemp3; colTemp4
"...";"...";"...";"...";"...";"...";"..."
"...";"...";"...";"...";"...";"...";"..."
SCENARIO 3) The data inside the CSV
colA;colB;colC;colTemp1
"...";"...";"...";"..."
"...";"...";"...";"..."
So the problem is that the first 3 columns (colA, colB, colC) are fixed colums while the next colums are dynamic (one time could be two colums like SCENARIO 1, another time could be 4 colums like SCENARIO 2, another time could be 1 colum like SCENARIO 3, and so on).
How can I make the schema dynamic so I retrieve in correct way the data from the CSV and make the right table inside the Excel? I read a lot of posts about the dynamic schema. Is there a way to make it in TOS DI?
I thought to get the first line from CSV using tFullInputRow-> tNormilize and then make a XML schema with the right number of columns. After, use this xml schema inside the tFileInputDelimited. It's an idea but I don't know how to make the XML Schema dynamically.
Thanks in advance,
Tommaso

[Anonymous]

Hi,
For your requirement, i think Dynamic schemas can meet your needs which allow you to design of Jobs with an unknown column structure (unknown name and number of columns). If necessary, dynamic columns can be mapped directly to the target using Pass-through mode.
However, dynamic schema feature is only available in the Enterprise version (on subscription) of Talend.

Best regards
Sabrina

[Anonymous]

For this purpose I create the tFileInputTextFlat component. You can specify a schema and check the option "Use column header to find position". If your file has a column header it will adjust the positions by the header line.
Check out this component from talend exchange.
There is also the possibility to define an alternative name for the header in case of the header contains names which breaks the rule of talend schema column names (e.g. if the name contains spaces or other chars which are not allowed in Java identifiers).

[Anonymous]

Hi Jloling,
I have tried to use ur component tFileInputTextFlat in my job, but always got error msg as attached.
I have checked the file "cimt.talendcomp.flatfileimport-1.3.jar" inside "D:\Talend_Erjan\Talend-Studio-r78327-V5.0.2\plugins\org.talend.designer.components.localprovider_5.0.2.r78327\components\tFileInputTextFlat"
Can you help me to resolve??
Thanks
Erjan

[Anonymous]

Hi Erjan,
Due to tFileInputTextFlat is custom component, make sure that you have installed it successfully.
Here is the reference on Talend Help Center
Installing a custom component.
What's more, did you put the custom component tFileInputTextFlat in this directory?

D:\Talend_Erjan\Talend-Studio-r78327-V5.0.2\plugins\org.talend.designer.components.localprovider_5.0.2.r78327\components\tFileInputTextFlat"

which is not the right place I think.
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
thats true, but very often the normal install process for user components fails. You can beliefe me, I have written a lot and get a lot of problem reports from users especially for the missing jar problem!
I have suggested a change to your mentioned help page.

[Anonymous]

Hi jlolling,
It is fantastic, thanks for your dedication to us which is very useful and helpful.
Best regards
Sabrina

[Anonymous]

Hi All,
Now it's working as expected!
Thanks
Erjan

[Anonymous]

Hi cyberjan,
How do you resolve your problem. Put the custom component in the right place? Or do as @jlolling said? Would you mind sharing your own experience with us.
Best regards
Sabrina

[Anonymous]

Hi Sabrina
I have created the custom components folder and point it in Talend and did as told by jloling "to delete the configuration/ComponentCache.javacache and restart TOS".
Now it worked, the missing jar message is not there, but subsequent login to TOS now it prompted the new error msg (attached).
Then I tried to delete the componentcache file again and it can login, but next login will prompt the same error msg.
What happen here??
Thanks
Erjan