topic Re: connect to Postgresql DB from Talend spark batch job in Talend Studio

connect to Postgresql DB from Talend spark batch job

yarivsc — Sat, 16 Nov 2024 10:00:23 GMT

Hello,
I have a spark job input as csv file and output as avro file.
For the source and target folder_name and file_name I want to use context that is getting values from postgres according to the current workflow.
Is there a way to connect to postgres from spark batch job? the palette for this type is very little.
Thanks

Re: connect to Postgresql DB from Talend spark batch job

Anonymous — Thu, 16 Mar 2017 09:16:04 GMT

Hi,

For RDS, you can use spark component to achieve it. Have you tried to use tjdbc (Generic JDBC Component) a spark component to connect to postgres RDS?

Best regards

Sabrina

Re: connect to Postgresql DB from Talend spark batch job

vapukov — Thu, 16 Mar 2017 09:52:04 GMT

It look like really missed functionality:

JDBC could be used only for lookup
no tLoadContext
no possibility load context in Job properties (same as for all DI Jobs)

You can only hardcode context in tab, but not load it

Re: connect to Postgresql DB from Talend spark batch job

Anonymous — Thu, 16 Mar 2017 13:49:24 GMT

I've been told by Talend R&D that it is their intent to keep tContextLoad out of the big data batch job. There is a previous post by me if you search for it. You have to do you context loading it a standard job then tRunJob call the big data back job. Then do a buffer if you need to send something back.

Re: connect to Postgresql DB from Talend spark batch job

vapukov — Thu, 16 Mar 2017 20:15:58 GMT

It is variant, thanks
but why, if they allow to have Context environment - why this stupid steps with load values?
Question for Talend Team!

Re: connect to Postgresql DB from Talend spark batch job

Anonymous — Fri, 17 Mar 2017 13:53:16 GMT

I agree, found the more detailed answer from Talend that I referenced earlier:
" So to confirm, tContextLoad is unavailable as a component for Big Data Batch jobs.
But you can still use the tContextLoad in a Big Data Batch job, but gluing a DI job to a tRunJob that calls a Spark job .
The DI job does the tContextLoad as part of the preparation and sends it to the Spark job.
The reasoning behind this is -- because of the way big data jobs are structured, the context can't be modified in a Spark component.
Just like you can't use the global map in a spark job, both the idea of the global map and a "modifiable" context is a kind of "job state" that can't be distributed.
(R&D says there ARE techniques to do this, but it's almost always a bad idea!)
R&D says they think of DI jobs as "orchestrators" that set up the Spark job to be run. "

This might make sense if they want to keep spark job more modular, functional, and with a very specific purpose which would make sense in a true scala/spark application, but it doesn't fit the Talend component, ease to development model well.