New to Data Integration - Do we need Amazon S3 environment?

[jmcy2408]

Hi,

We have up to now been completing our ETL via app scripts for core reportable areas in one of our source systems.  These then store outputs into a data space for consumption by other apps, reports, insights and users so all working from the same cleansed data source.

Recently we've been looking at using the data integration side of our Qlik Cloud tenant however unsure how to get started?

We do not have a staging area, is this mandatory? 

We want to be able to transform our data (from combination of PostgreSQL, MS SQL and MY SQL databases along with REST APIs) and store back into your Qlik Cloud tenant as QVDs for the same reasons as above.

It seems i cannot create a data project without Amazon S3 - is that correct? - would had thought (naively perhaps) the functionality would all be within our Qlik Cloud tenant - but i'm guessing not.

Thanks

[jochem_zw]

when using QCDI for storing into QVD you need an AWS account for landing and storage in a S3 Bucket

[AnaSilva]

@jmcy2408 , 

I don't know if you still need this information, but here's the documentation that helped me setup our first pipeline: Landing data from data sources | Qlik Cloud Help

One thing that was interesting is the permission needed for the IAM user, we had to grant full S3 permission set for the connection to work.

[TimGarrod]

Qlik Talend Cloud (the data integration capabilities within Qlik Cloud) provide a set of data integration capabilities to move, transform and curate data.  When looking at this from a pure Qlik Cloud perspective, the data integration capabilities can be used to ingest the raw data from your sources into QVD's.    

Any transformation of that data is still performed using Qlik script or some upcoming functionality. 

The main capabilities for transforming data or automating a data lake house / warehouse architecture require a data platform like Snowflake, Databricks, BigQuery, SQL Server etc. as the target platform.   Data is ingested using CDC technology into that target and transformations are performed using push-down SQL designed by custom SQL or graphical no-code transformation flows.  

If you do not have a data platform where you are collecting and transforming the data, the data integration features still support uploading to QVD's, but any transformation will be handled by features in the Analytics activity center.