Sending document to kafka

[Anonymous]

Hello,

So I am using the tRestClient to perform a curl command to get some data through a API GET request and am trying to stream this data into my kafka topic. Initially, I tried using the tSystem component but it was only sending null values so after digging for some time I cam across using the tRestClient component. I was able to get my data and store it into a variable "body" which is of a type document. This was achieved using the tRestClient and tXMLMap components. However, I am trying to convert the document to a byte array so that the tKafkaOuput, that only accepts serializedValue of type byte array, can accept the input and push the data into the topic accordingly. Initially, I used a tConvertType to achieve this but it only converted the data into a null value. Can You please tell me how I can go about doing this? This is very important and need help as soon as possible?

Is using the tRestClient the right approach, or do I have to go back using the tSystem component. If the later, can you please tell me how I can accomplish pushing the data using a curl command to my kafka using tSystem?

[Anonymous]

Hi,

    Your approach to use tRest and XML map components are correct. I have created a similar flow as shown below but I have created the data using trowgenerator. But in your case, it will be your source flow.

tJavarow converts the data type of XML from String to byte[].

Inside the tjavarow, you need to provide the below code snippet.

output_row.SerializedData =
input_row.SerializedData.getBytes();

Now the code is ready to be published using Kafka.

I would suggest to avoid the tSystem command since you are trying to run shell scripts and it is actually diluting the concept of graphical ETL tool. Unless it is really required, do not go in that route.

I hope I have resolved your query.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

Thank You for your response. So essentially I am trying two ways. The first is connecting the tKafkaConnection -> tFixedFlowInput -> tXMLMap -> tRESTClient -> tKafkaOutput. I am passing the "body" from tRestClient to tKafkaOutput of type document. However, I need to convert the type document to type byte array in order for the data to be produced into the kafka topic. 

The second way was connecting the tKafkaConnection -> tRESTClient -> tXMLMap -> tJavaRow -> tKafkaOutput, as per your suggestion. This works!, as I am passing a String from tXMLMap to tJavaRow which then converts it into a byte array and then feeds that into the kafka topic. However, I want the data in the kafka topic to look exactly like the JSON data streaming in using the "curl" command. However, after looking at the kafka output, I believe the tXMLMap destroys the structure and just gives me plain unformatted text instead of the json data in {} format. 

My xml map for the second approach can be seen in the attachment. I also have "Convert response to DOM Document" checked in the advanced settings of the tRESTClient component, for the second approach, otherwise the data will not go through.

The first approach gives me the data in the format I need but cannot be pushed into the kafka topic due to the conversion issue. The second approach pushes the data into the topic but of the wrong format (i.e. I want json format).

Can you please assist me on this issue. I greatly appreciate your help.

xmlmap.PNG

[Anonymous]

Hi,

    I could see that you are using String data type in the output of tXMLMap. I would suggest you to use Document type so that format will be in tact. Please refer the sample scenarios of tXMLMap for details.

https://help.talend.com/reader/ixBASPZJ7IvqUQVupZwWbg/hEDmNY8roDW2fqyyeYwLbw

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

That still hasn't answered my question. In that case, I get the same problem as presented in scenario 1! I have a document type and I need to convert it into a byte array or string or whatever so I can feed it into kafka. Also, I am getting my output as xml unformatted or formatted. However, I want my output in json!!

These are two problems I dont know how to fix, and you idea seems to root back to the problems in scenario 1 which is harder to tackle due to the conversion problem. This is why I used string in the tXMLMap in scenario 2 so that it can read the data into a string from the tRestClient.

[Anonymous]

@sai06kiran

    I have given samples in above examples where you can convert the data from one format to another and I have also specified the component tWriteJSONFileds to convert the data to JSON. To remove the fog, lets discuss further with your specific example.

Could you please provide the screen shot of your current job, your component details screenshots, the error you are facing for the data so that Talend community members can help you further? It would be great if you can add the result of your REST call so that we can see the sample data and how you want to transform it. Please remember to remove any sensitive data before posting to the community.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂