topic Re: Using Bulk Load component in Talend Data Catalog

Using Bulk Load component

Anonymous — Wed, 07 Jan 2015 17:56:41 GMT

Hello,
I'm experiencing some issues with loading data into my MDM entity.
Steps completed:
Created DataModel called Location
Created DataContainer called Location
Created Role called DataOwner with Admin Permissions and assigned Roles to Entity Location
Deployed All to Server.
Created Job to Load Data into Entity and receiving the following error
I'm also not able to see the entity on the Master Data Browser page.****
Exception in component tMDMBulkLoad_1
java.io.IOException: An exception occurred while processing last record.
at org.talend.mdm.bulkload.client.InputStreamMerger.throwLastFailure(InputStreamMerger.java:191)
at org.talend.mdm.bulkload.client.InputStreamMerger.close(InputStreamMerger.java:262)
at mdmsolution.locationload_0_1.LocationLoad.tWriteXMLField_1_InProcess(LocationLoad.java:1385)
at mdmsolution.locationload_0_1.LocationLoad$1ThreadXMLField_tWriteXMLField_1_Out.run(LocationLoad.java:641)
Caused by: org.talend.mdm.bulkload.client.BulkloadException: An exception happened during bulk load on MDM server.
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IllegalArgumentException: Type 'Location' does not exist in data model 'Location'.
at com.amalto.core.servlet.TransactionFilter.doFilter(TransactionFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
disconnected
at com.amalto.core.servlet.ErrorFilter.doFilter(ErrorFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393)
at org.apache.catalina.authenticator.MDMSingleSignOn.invoke(MDMSingleSignOn.java:66)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.servlet.ServletException: java.lang.IllegalArgumentException: Type 'Location' does not exist in data model 'Location'.
at com.amalto.core.servlet.LoadServlet.doPut(LoadServlet.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.amalto.core.servlet.TransactionFilter.doFilter(TransactionFilter.java:38)
... 24 more
Caused by: java.lang.IllegalArgumentException: Type 'Location' does not exist in data model 'Location'.
at com.amalto.core.servlet.LoadServlet.getTypeKey(LoadServlet.java:175)
at com.amalto.core.servlet.LoadServlet.doPut(LoadServlet.java:108)
... 29 more
at org.talend.mdm.bulkload.client.BulkloadClientUtil.bulkload(BulkloadClientUtil.java:77)
at org.talend.mdm.bulkload.client.BulkloadClientUtil$AsyncLoadRunnable.run(BulkloadClientUtil.java:167)
at java.lang.Thread.run(Unknown Source)

Re: Using Bulk Load component

Anonymous — Wed, 07 Jan 2015 19:29:21 GMT

Donovan,
Could you add a screenshot of your Data Model definition?

Re: Using Bulk Load component

Anonymous — Wed, 07 Jan 2015 19:47:00 GMT

Here you go.

Thank you

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 15:12:36 GMT

Things to add.
Also created Views for entity and its still not showing up in the web interface.

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 19:10:59 GMT

I created two elements: one has an "e" symbol with a complex type that I assigned. I'm unable to seed this in the data container view drop down.
The next element I created was a "anonymous type" this was the default when creating an element. This shows up in the drop down under the data container. See image two.
What is the anonymous type for? I want to type the element a complex type I created.
Also, still unable to see entity in MDM GUI interface. I loaded the Product Families demo correctly and the entity that I deployed shows up in the "Master Data Browser"

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 19:37:53 GMT

Some things to check within the WebUI is if the rightmost panel, the container and model selector are both set to location. Also, from within the studio the Role has to have been granted read or even read/write permissions by double clicking the role and using the pulldowns and and other controls.
Also in 219091/blob_20150108-2006.png the little blue "e" at the beginning of the entity doesn't look right, but the icon for locations2 and locations3 looks correct.

I believe an anonymous type is one that doesn't come from, nor inherit the properties of your menu of predefined, or user-defined types as listed in the right panel.

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 20:10:58 GMT

Both are selected see image

What object type do I select for Admin role permissions?

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 22:31:07 GMT

Found issue: Need to Add Key to main element. This will remove "e" label and turned into a document.
Now onto the Bulk component issue....

Re: Using Bulk Load component

Anonymous — Thu, 08 Jan 2015 22:48:32 GMT

Receiving the following error after running job.
Jan 08, 2015 4:46:58 PM org.apache.commons.httpclient.HttpMethodBase getResponseBody
WARNING: Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
Exception in component tMDMBulkLoad_1
java.io.IOException: An exception occurred while processing last record.
at org.talend.mdm.bulkload.client.InputStreamMerger.throwLastFailure(InputStreamMerger.java:191)
at org.talend.mdm.bulkload.client.InputStreamMerger.close(InputStreamMerger.java:262)
at mdmsolution.locationload_0_1.LocationLoad.tWriteXMLField_1_InProcess(LocationLoad.java:1385)
at mdmsolution.locationload_0_1.LocationLoad$1ThreadXMLField_tWriteXMLField_1_Out.run(LocationLoad.java:641)
Caused by: org.talend.mdm.bulkload.client.BulkloadException: An exception happened during bulk load on MDM server. javax.servlet.ServletException: javax.servlet.ServletExcept

To see the whole post, download it here
OriginalPost.pdf

Re: Using Bulk Load component

Anonymous — Fri, 09 Jan 2015 16:14:27 GMT

A little further: I'm able to remove the validation check and bulk load will work, however, when I turn validation on it throws the following stack trace.
Exception in component tMDMBulkLoad_1
java.io.IOException: An exception occurred while processing last record.
at org.talend.mdm.bulkload.client.InputStreamMerger.throwLastFailure(InputStreamMerger.java:191)
at org.talend.mdm.bulkload.client.InputStreamMerger.close(InputStreamMerger.java:262)
at mdmsolution.locationload_0_1.LocationLoad.tWriteXMLField_1_InProcess(LocationLoad.java:1329)
at mdmsolution.locationload_0_1.LocationLoad$1ThreadXMLField_tWriteXMLField_1_Out.run(LocationLoad.java:613)
Caused by: org.talend.mdm.bulkload.client.BulkloadException: An exception happened during bulk load on MDM server.
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.RuntimeException: java.lang.RuntimeException: com.amalto.core.save.SaveException: Exception occurred during save:
at com.amalto.core.servlet.TransactionFilter.doFilter(TransactionFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.amalto.core.servlet.ErrorFilter.doFilter(ErrorFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:524)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393)
at org.apache.catalina.authenticator.MDMSingleSignOn.invoke(MDMSingleSignOn.java:66)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.servlet.ServletException: java.lang.RuntimeException: java.lang.RuntimeException: com.amalto.core.save.SaveException: Exception occurred during save:
at com.amalto.core.servlet.LoadServlet.doPut(LoadServlet.java:129)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.amalto.core.servlet.TransactionFilter.doFilter(TransactionFilter.java:38)
... 24 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: com.amalto.core.save.SaveException: Exception occurred during save:
at com.amalto.core.save.context.BulkLoadSaver.save(BulkLoadSaver.java:43)
at com.amalto.core.servlet.LoadServlet.doPut(LoadServlet.java:119)
... 29 more
Caused by: java.lang.RuntimeException: com.amalto.core.save.SaveException: Exception occurred during save:
at com.amalto.core.load.action.DefaultLoadAction.load(DefaultLoadAction.java:81)
at com.amalto.core.save.context.BulkLoadSaver.save(BulkLoadSaver.java:41)
... 30 more
Caused by: com.amalto.core.save.SaveException: Exception occurred during save:
at com.amalto.core.save.context.Init.save(Init.java:66)
at com.amalto.core.load.action.DefaultLoadAction.load(DefaultLoadAction.java:72)
... 31 more
Caused by: com.amalto.core.util.ValidateException: :-1:-1: cvc-type.2: The type definition cannot be abstract for element Location.
at com.amalto.core.schema.validation.XmlSchemaValidator.validate(XmlSchemaValidator.java:61)
at com.amalto.core.save.context.Validation.save(Validation.java:57)
at com.amalto.core.save.context.ApplyActions.save(ApplyActions.java:47)
at com.amalto.core.save.context.Security.save(Security.java:102)
at com.amalto.core.save.context.GenerateActions.save(GenerateActions.java:144)
at com.amalto.core.save.context.ID.save(ID.java:113)
at com.amalto.core.save.context.Init.save(Init.java:56)
... 32 more
at org.talend.mdm.bulkload.client.BulkloadClientUtil.bulkload(BulkloadClientUtil.java:77)
at org.talend.mdm.bulkload.client.BulkloadClientUtil$AsyncLoadRunnable.run(BulkloadClientUtil.java:167)
at java.lang.Thread.run(Unknown Source)
disconnected
Job LocationLoad ended at 10:12 09/01/2015.

Re: Using Bulk Load component

Anonymous — Tue, 04 Aug 2015 10:53:24 GMT

Hi,
loading of data into exasol has been very slow of the order of 2rows/sec. any idea how to improve data loading capacity ?

Re: Using Bulk Load component

Anonymous — Tue, 04 Aug 2015 11:46:20 GMT

Hi raviteja440,
Could you please show us the screenshots of your current job design?
Best regards
Sabrina

Re: Using Bulk Load component

Anonymous — Thu, 06 Aug 2015 06:18:04 GMT

Hi Sabrina,
Thanks for your reply. I am extracting SAP table data directly and loading into Exasol database no other transfomations. Initially my data speed was 2rows/sec. Later i used commit control option and set it to '10000' rows. Now the speed is hiking to 70 rows/sec at max. I couldn't find any other way.
Plz help.

Re: Using Bulk Load component

Anonymous — Fri, 14 Aug 2015 08:42:23 GMT

Hi,
Anyone Could you please help with the issue?