topic tFileInputParquet - How to read generic parquet files and extract schema? in Talend Studio https://community.qlik.com/t5/Talend-Studio/tFileInputParquet-How-to-read-generic-parquet-files-and-extract/m-p/2302191#M74206 <P>Hello.</P><P></P><P></P><P>I want to make a job that reads a parquet file, in order to apply transformations to the data. This job needs to be generic, which means that I cannot know in advance the column names of my parquet files.</P><P>Unfortunately, tFileInputParquet is not compatible neither with dynamic fields, nor with reading the lines as a single string, which would allow me to circumvent this issue.</P><P></P><P></P><P>How can I solve this issue without resorting to a custom Spark program? Thank you in advance.</P><P>&nbsp;</P><P>imho, I am very surprised that schema guessing is not already a feature for most components. Since Spark's error stacktrace actually lists the column names in the error message, this looks like a technical possibility. Even default index numbers would help tremendously. As far as I am concerned, industrializing any of our jobs through Talend is either difficult or almost impossible.</P> Sat, 16 Nov 2024 01:14:22 GMT CPorrot1602485748 2024-11-16T01:14:22Z tFileInputParquet - How to read generic parquet files and extract schema? https://community.qlik.com/t5/Talend-Studio/tFileInputParquet-How-to-read-generic-parquet-files-and-extract/m-p/2302191#M74206 <P>Hello.</P><P></P><P></P><P>I want to make a job that reads a parquet file, in order to apply transformations to the data. This job needs to be generic, which means that I cannot know in advance the column names of my parquet files.</P><P>Unfortunately, tFileInputParquet is not compatible neither with dynamic fields, nor with reading the lines as a single string, which would allow me to circumvent this issue.</P><P></P><P></P><P>How can I solve this issue without resorting to a custom Spark program? Thank you in advance.</P><P>&nbsp;</P><P>imho, I am very surprised that schema guessing is not already a feature for most components. Since Spark's error stacktrace actually lists the column names in the error message, this looks like a technical possibility. Even default index numbers would help tremendously. As far as I am concerned, industrializing any of our jobs through Talend is either difficult or almost impossible.</P> Sat, 16 Nov 2024 01:14:22 GMT https://community.qlik.com/t5/Talend-Studio/tFileInputParquet-How-to-read-generic-parquet-files-and-extract/m-p/2302191#M74206 CPorrot1602485748 2024-11-16T01:14:22Z