topic Re: Best architecture to process MQTT messages (statefull) in Talend Studio

Best architecture to process MQTT messages (statefull)

Anonymous — Sat, 16 Nov 2024 05:47:27 GMT

Hi,

this is my use case and I really don't know which is the best option between Talend ESB o just Data Integration.

I need to subscribe to a "wildcard" topic since I need to get all messages sent on specific channels. The topic name contains even the device which is sending the message, so I need to even process the topic string to get the "device id".

The message has a JSON payload with a three level status (1, 2, 3). I need to add a row with device id and status in a table for each message (ok this is the easy task). I would like even to keep the "last status" for a device and when I receive a new one message store for the previous status the triplet device id, status and time elapsed until the new status.

Of course I can use a Java job, there is no problem to write some code.

Ideas?

Thank you, Stefano.

Re: Best architecture to process MQTT messages (statefull)

Anonymous — Mon, 03 Jun 2019 10:30:32 GMT

Hello,

Please have a look at talend data mapper structure:TalendHelpCenterata mapping.

Best regards

Sabrina

Re: Best architecture to process MQTT messages (statefull)

Anonymous — Mon, 10 Jun 2019 17:07:43 GMT

The ESB Module is the best choice if you need to consume the MQTT messages in a permanent way (24/7) and we offer this with the cMQTT component in the Route Builder. For the JSON part we recommend to use our TDM module as said by my colleague before it allows you to handle also complex Json data, the related component is cMap (or via cTalendJob also tHMap in case you like to use more of our tXXX components with the route. )

HTH,

Dietmar

Re: Best architecture to process MQTT messages (statefull)

Anonymous — Tue, 09 Jul 2019 08:58:06 GMT

Thank you for all your answers. Actually with an cMQTT component I can connect and get the json body of the messages then processed with a job.

But a problem remains: can I transfer to the job even the topic associated with the received message? The topic contains the client id and I need a "generic" subscription for a variable set of clients all publishing to a topic like:

brand/[client_id]/status

Being able to get the topic I can enrich the json data received and store the relevant information with the client id.

Thank you!