Skip to main content
Announcements
Introducing a new Enhanced File Management feature in Qlik Cloud! GET THE DETAILS!
cancel
Showing results for 
Search instead for 
Did you mean: 
IanM
Contributor III
Contributor III

text file import - problems

Hi, Help please... I've been trying to get this to work all day!
I need to import an oracle .dat file created using the UTL_FILE functionality within Oracle.
It is a Unicode text file formatted as follows (shortened):

groupNbr|0100028
language|
extendedRangeProductNumber|
articleLifecycleStatus|60
Maximum Temperature Coefficient|+50ppm/°C
Length|6.5mm
Minimum Temperature Coefficient|-50ppm/°C
EOR
groupNbr|0100034
language|en
extendedRangeProductNumber|
articleLifecycleStatus|60
Maximum Temperature Coefficient|+50ppm/°C
Minimum Temperature Coefficient|-50ppm/°C
Length|6.5mm
EOR
groupNbr|0100040
language|en
extendedRangeProductNumber|
articleLifecycleStatus|60
Maximum Temperature Coefficient|+50ppm/°C
Minimum Temperature Coefficient|-50ppm/°C
Length|6.5mm
EOR


The field separator is CRLF ("\r"+"\n") and the row separator is "EOR"+"\r"+"\n".
I will take you through what I have done so far......

The I can't get the tFileInputDelimited to recognise the end of the record so no records are passed.
I have been able to get tFileInputMSDelimited to work using the same delimiters, and selecting a blank column as the record identifier (so it picks up all records)

This works, until you realise that the data order is not always the same!! Look at the data above and you will see data 'Length' is in position 6 in the first record and in position 7 in the other two...

I'm now thinking I need to read this file like a pseudo XML or JSON and use the value pairs to assign the table columns.
Any ideas anyone?

Bear in mind I need to process 20+ million records!
Many thanks
Ian

Labels (2)
1 Reply
Anonymous
Not applicable

This isn't exactly what you need, but it should give you the idea. I wrote this tutorial on dealing with dynamic schemas using Talend Open Studio. I believe this method will work for you if you know the column headings you are expecting.....

http://www.rilhia.com/node/38