Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Qlik Open Lakehouse is Now Generally Available! Discover the key highlights and partner resources here.
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Storing Data from KafkaInput in smaller files

Hi,

 

In my usecase, i am receiving data in Kafka 24*7 in ebcdic. I read the data as byte array from kafkainput and parse the data using the tHMAP. Output of tHmap goes to tjavarow as byte array. output from the tJavaRow is getting stored in tHDFSOutput. Initial tJava component to create the filename with datetimestamp.

 

Here are my challenges

1) my data is getting stored as a single file in hdfs. I need to have them smaller file, as i might need to see the data from morning to afternoon (or to any point of time). Because of me having the data in a single file, i am unable to fetch data.

2) I tried updating the filename in the tJavaRow (that is the reason for inserting the component here). But i am unable to change the file name.

 

Need some suggestions to get this done.

 

Below is a flow, recreated for understanding.

0683p000009Lupt.png

 

 

 

 

Labels (2)
1 Reply
Anonymous
Not applicable
Author

Hi

I think you need to iterate each row and change the filename dynamically if you want to generate one file for each row or several rows. The job looks like: tJava1-onsubjobok--tkarafInput--tHMap--main--tflowtoIterate--iterate--tjava2--oncomponentok--tfixedFlowInput--main--tHDFSOutput. tJava2: change the file name. tfixedFlowInput: generate the current data. Let me know if you have further questions.

Regards

Shong