Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
I try to build a generic job which processes several hundreds of tables. It reads a flat txt-file with csv formatting into a dynamic schema with a tFileInputDelimited component and after processing writes another csv file. Between input and output the data is processed with a tJavaFlex component which iterates through all columns and processes the data according to the type an length provided in the schema file. The problem I have is, that the input file has no header row which names the columns of the file but the output file requires the correct header row. The corresponding schema of the table is provided in a separate xlsx interface specification file which looks more or less like the one attached. It is read by another subjob prior to the data file.
I managed to transpose the ColumnName into one row/String and add some ; but after several attempts I have no idea how to get this 'schema' row i front of the input data stream so that is recognized by the dynamic schema or to change the metadata column names in a java component after reading the file.
Simplest solution whould be to write a new temporary data file, insert the header row and appand the date from the original file, but this is definitely no practical solution because the input files can reach several GB in size and handeling this whole date just to add one row seems not very efficient to me.
I try to build a generic job which processes several hundreds of tables. It reads a flat txt-file with csv formatting into a dynamic schema with a tFileInputDelimited component and after processing writes another csv file. Between input and output the data is processed with a tJavaFlex component which iterates through all columns and processes the data according to the type an length provided in the schema file. The problem I have is, that the input file has no header row which names the columns of the file but the output file requires the correct header row. The corresponding schema of the table is provided in a separate xlsx interface specification file which looks more or less like the one attached. It is read by another subjob prior to the data file.
I managed to transpose the ColumnName into one row/String and add some ; but after several attempts I have no idea how to get this 'schema' row i front of the input data stream so that is recognized by the dynamic schema or to change the metadata column names in a java component after reading the file.
Simplest solution whould be to write a new temporary data file, insert the header row and appand the date from the original file, but this is definitely no practical solution because the input files can reach several GB in size and handeling this whole date just to add one row seems not very efficient to me.
