topic Re: Importing fixed-length files into MySQL in Talend Studio

Importing fixed-length files into MySQL

Anonymous — Tue, 22 May 2012 20:20:47 GMT

Hi,
I'm working in getting data from very large txt fixed-length files and I would like to load them into MySQL. I have several of these files and they are about 28GB each. I found "Talend Open Studio for Big Data" and I would like to integrate this tools to my solution. At this point I'm working to see if it is possible to load such big file using Talend.
Any thought, recommendations and/or advice are welcome.
Thanks,
jjsai

Re: Importing fixed-length files into MySQL

Anonymous — Wed, 23 May 2012 03:15:04 GMT

Hi
Welcome to Talend Community!
Talend Open Studio for Big Data mainly aims at HDFS, Hive and Hbase and extends capabilities for Pig.
According to your description, you'd better use Talend Open Studio for Data Integration to do this ETL.
The components about Mysql from Talend support bulk load and can handle with big data well.
There are some available features you can select for better performance.
Regards,
Pedro

Re: Importing fixed-length files into MySQL

Anonymous — Wed, 23 May 2012 19:16:08 GMT

Pedro,
Thank you for your comment, it makes sense. I will go ahead and install Talend Open Studio for Data Integration.
Once again, thank you so much for the advice.
jjsai

Re: Importing fixed-length files into MySQL

Anonymous — Thu, 24 May 2012 20:16:42 GMT

HI,
I did install TOSDI and I'm working in the job to load my data. I have this huge file with 184 fields. I got the job tFileInputPositional to start playing. I check the basic settings and in the pattern I entered the length of the fields. When I ran the job it worked flawlessly but taking a close look the data was truncated according to the schema definition. Also make sense.
I need to change this schema definition and add my 184 field but it will take forever so I notice there is an import XML feature (can you guess this is my first time working with Talend?) so I created a formula within excel that replicates the XML format in which the schema is extracted. Then I copy the column and pasted in a text editor saving it as XML. I load my brand new file and the schema window goes blank. I don't see any error message about this. Do I need to load the schema manually?
Thanks,
jjsai

Re: Importing fixed-length files into MySQL

Anonymous — Fri, 25 May 2012 00:41:16 GMT

Please ignore mi previous comment. Thanks.