extracting a kafka "stream" with TOS BigData

[reapeleg]

Hi eb

I need to implement a continuous consumption (extraction) of kafka events into a target DB.

Events rate is low and should not exceed a few to a few dozens a second (at picks).

But then this steady flow of events will continue forever.

Would you attempt this at all with TOS big data or go straight away to a licensed streaming big data or ESB solution?

These two feels to me like an over complication for such a modest flow of events...

In these two former discussions for doing so with the TOS bigData:

One seems to have implemented successfully tLoop over tKafkaInput here:

https://community.talend.com/s/question/0D53p00007vCsxRCAS/problem-with-running-tkafkainput-in-a-loo...

The other seem to have struggled with tInfiniteLoop over tKafkaInput and was advised to put the kafka extraction in a child job in order to have it work??

https://community.talend.com/s/question/0D53p00007vCnPJCA0/how-to-rerun-tkafkainput-component-in-a-j...

Appreciate your thoughts on this!

TIA

Rea

[andyhilton27]

Thanks for this valuable information...I really need this kind of information mcdvoice mybkexperience