Inquiry: Direct Replication from RDBMS to Iceberg Tables in S3

[Ayman1]

We are currently evaluating Qlik Replicate (Attunity) as a solution for replicating data from various RDBMS systems (e.g., Oracle, MariaDB, PostgreSQL) directly into Iceberg tables in an S3-based data lake. We are seeking clarity on whether Attunity offers native support for writing directly to Iceberg tables without intermediary steps or reliance on other query engines like Snowflake.

Background: Our goal is to enable seamless replication from RDBMS systems into Iceberg-backed tables in S3. We aim to simplify our architecture by directly writing to Iceberg tables, leveraging Glue or other catalogs for table management.

Questions:

1. Does Attunity currently support direct replication from RDBMS to Iceberg tables in S3?
2. Are there native connectors or configurations available for this use case, particularly for Glue-backed Iceberg catalogs?
3. If such a connector exists, are there any specific setup instructions, limitations, or prerequisites we should be aware of?

Why This is Important: Direct integration with Iceberg would eliminate dependencies on intermediary systems, streamline our data replication process, and align with our Lakehouse MVP initiative.

We would appreciate any guidance, documentation, or recommendations regarding direct Iceberg integration.

Thank you!

[john_wang]

Hello @Ayman1 ,

See my reply in your another similar article:

Issues with Metadata Table Creation and VARCHAR Length Constraints in Snowflake to Iceberg Conversion

feel free to let us know if you need any additional information.

Regards,

John.

[jasonhallqlik]

Hey @Ayman1 this question is very timely as Qlik just completed the acquisition of Upsolver, who specializes in real time ingestion into Iceberg.  The Upsolver technology is currently getting integrated into the Qlik portfolio, and we expect to have the technology available soon.  If you'd like to discuss further, including possibly getting into our Early Access Program, please don't hesitate to reach out to me.

[jshia1]

We are currently using QLIK Replicate to move data from on-prem Oracle data warehouse to Azure Databricks in delta lake format.

Any ETA on when QLIK Replicate will support Iceberg format as destination end point?

We want to move from delta lake format to Iceberg due to large growth in metadata that is causing issues.

If QLIK only supports delta lake format, it will continue to pose issues for us.

Thank you.

--John

[jasonhallqlik]

Hey @jshia1 I don't have an ETA that I can share, but I can tell you it's being worked on.  

If you'd like to connect with your account team, we can have a call to better understand your exact use case and come up with a plan forward.

[jshia1]

Guy in charge of QLIK will be contacting QLIK support.

We are currently using QLIK Replicate to replicate data from on-prem Oracle to Azure Databricks delta lakehouse in form of raw delta tables.

We want the replication output to be in Iceberg format instead since Iceberg has centralized metadata instead of distributed JSON log files (delta tables).

The issue is inherent in the architectural design of delta tables and cannot be resolved unless we go to some other architecture like Iceberg.

We can't use Snowflake due to security issue.

[Sreekanth_333]

Hello Qlik Team,

Do we have update on this request for replication in Iceberg format ?