topic Re: Retrieve schema from materialized view in Talend Studio

Retrieve schema from materialized view

Anonymous — Mon, 19 Nov 2018 11:50:22 GMT

When retrieving schemas from a database connection, I'm unable to see any materialized view. Tables and views work OK. Any workarround?

I'm using a PostgreSQL connector, version 9.x, with Talend Open Studio 7.0.1.x. The database is an Amazon Aurora PostgreSQL.

Re: Retrieve schema from materialized view

Anonymous — Wed, 21 Nov 2018 17:51:32 GMT

Hi,

The current Talend retrieve schema functionality gets the schema for tables and views and not for Materialized views. In between, I am really curious why you are using materialized views. I would say it is sort of things of past considering the high processing capability of DB servers.

Could you please share the reason why you selected materialized view in your overall technical solution?

Warm Regards,

Nikhil Thampi

Re: Retrieve schema from materialized view

Anonymous — Thu, 22 Nov 2018 08:17:48 GMT

Thanks for your answer, @nthampi. We are using a series of materialized views to cleanse and transform raw data. This way, advanced users can look into the MVs and trace results back to the source. If we were to use normal views, queries would take minutes instead of seconds, which would be too much.

I disagree with you in that MVs are a thing of the past, at least for some scenarios.

Re: Retrieve schema from materialized view

Anonymous — Mon, 26 Nov 2018 05:50:53 GMT

@Arturo

When we consider the MV, we need to consider the loading time also and there are many new ways of achieving it without MV. For example, whenever the data is getting loaded to the source database, you can stream the same data to a Data Lake using a Bigdata Streaming job.

Within the Data Lake, you can cleanse the data and keep the data in the trusted layer.

Since you are using a Bigdata cluster, your analytical operations will use the power of cluster rather than normal database and it will be faster. Also it is the way to future since you will need to do analysis of lot of data and MV will not be able to hold them all at a lower price.

Warm regards,

Nikhil Thampi