Skip to main content

Catalog and Lineage

Discussion Board for collaboration around Catalog and Lineage.

Announcements
June 21, 10AM ET: Q&A with Qlik, Live Chat! Qlik Lineage for your Data Dynasty. REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
akshaye_c_navale

Qlik Catalog Publish to AWS S3 failing for large datasets i.e. 17 M rows

Hi Community,

I am trying to publish dataset to AWS S3 from Qlik Catalog. It works if Entity has less number of rows.

I try to publish entity with 17M rows and 20 fields and it get failed with below error message,

Task DataShippingHandler failed: ExecException: Stopping execution on job failure with -stop_on_failure option

Does anyone has faced this issue for large datasets?

What size of entity can be publish in terms of # of rows & Fields. Des anyone know limitations of Qlik Data Catalog Publish feature?

Thanks,
Akshaye

Labels (2)
2 Replies
DaveHowland
Employee
Employee

Hello Akshaye

There are a number of different factors that contribute to the limitations in ingestion of large datasets. Is the QDC product the full hadoop version or of the single server variety?  If you're wanting to ingest / catalog large datasets, the hadoop version is advisable. That said, for the most part : limitations in large file publishes are constrained by QDC product type and then by infrastructure(disk size, network,etc.). There are no hard caps in the product itself. 

You might consider opening a support case, in that they could help you triangulate on the cause of this error faster. Hope that helps. 

David

 

akshaye_c_navale
Author

We have Single Node architecture where we have dedicated Web Server and Database server..