topic Re: Data Profiling Report Generation Failure in Data Quality

Data Profiling Report Generation Failure

hiitzsdg — Sat, 16 Nov 2024 10:15:58 GMT

Hi,
I performed a Columnar analysis and then I am trying to generate the "Basic" report. Being new to Talend DQ, some of the reports are generated successfully. But for few, I am getting an error message and the report generation fails. The error message that I am getting is "Error during historization of data of report <Report_Name>: could not insert : [org.talend.dataprofiler.datamart.hibernate.TdqValues].

The error message that I am seeing in the Error Log is as under:
org.hibernate.exception.GenericJDBCException: could not insert: [org.talend.dataprofiler.datamart.hibernate.TdqValues]
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2202)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2595)
at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:51)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:232)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:139)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at org.talend.dq.persistence.DatabasePersistence.persist(DatabasePersistence.java:156)
at org.talend.dataquality.reporting.engine.ReportDocGenerator.historizeData(ReportDocGenerator.java:505)
at org.talend.dataquality.reporting.engine.ReportDocGenerator.generate(ReportDocGenerator.java:275)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator.generate(LocalReportDocGenerator.java:129)
at org.talend.dataprofiler.core.tdq.ui.action.GenerateReportFileAction$2.run(GenerateReportFileAction.java:342)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: java.sql.SQLException: ORA-12899: value too large for column "DQ_PORTAL"."TDQ_VALUES"."VAL_STRING" (actual: 500, maximum: 255)

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2182)
... 16 more

As per my understanding this is because of space. So should I contact with the Admin Team to have the space increased for TDQVALUES tables? Can the DBA's change this size without any impact?

Your help in this regards is highly appreciated.

Re: Data Profiling Report Generation Failure

Anonymous — Wed, 09 Nov 2016 06:49:04 GMT

Hi,
Could you please indicate what's the build version you are using? Is there any special characters in your column? Screenshots of data profiling setting will be preferred.
Best regards
Sabrina

Re: Data Profiling Report Generation Failure

hiitzsdg — Wed, 09 Nov 2016 06:59:30 GMT

Hi,
I am using Talend Big Data Platform (6.1.1.20151214_1327). There are Chinese characters in columns. We are performing Simple, Advanced, Pattern Frequency.

Re: Data Profiling Report Generation Failure

Anonymous — Wed, 09 Nov 2016 07:21:09 GMT

Hi,
This issue is probably caused by Charset(Some special characters get 2 bytes in UTF-8, while they get only one byte in other charset.)
Did you set file delimilted metadata for your column? If so, what does file setting look like? Are you using UFT-8 Encoding?
Best regards
Sabrina

Re: Data Profiling Report Generation Failure

hiitzsdg — Wed, 09 Nov 2016 07:22:53 GMT

We are using Oracle Database as our metadata connection.

Re: Data Profiling Report Generation Failure

Anonymous — Wed, 09 Nov 2016 07:26:19 GMT

Hi,
Have you tried to modify the length of this column in DB side to see if it works?
Best regards
Sabrina

Re: Data Profiling Report Generation Failure

hiitzsdg — Wed, 09 Nov 2016 13:21:15 GMT

Modify to what length? 500 as being suggested in the error message?

Re: Data Profiling Report Generation Failure

hiitzsdg — Thu, 10 Nov 2016 08:41:13 GMT

Hi,

I modified the field length of the column TDQ_VALUES.VAL_STRING to 500 and the report generation was successful for most of the analysis. But it failed for one of the reports with the below error message in the Error_Log:

java.lang.StringIndexOutOfBoundsException: String index out of range: -3
at java.lang.String.substring(Unknown Source)
at org.jfree.text.G2TextMeasurer.getStringWidth(G2TextMeasurer.java:80)
at org.jfree.text.TextUtilities.nextLineBreak(TextUtilities.java:294)
at org.jfree.text.TextUtilities.createTextBlock(TextUtilities.java:235)
at org.jfree.chart.axis.CategoryAxis.createLabel(CategoryAxis.java:1140)
at org.jfree.chart.axis.CategoryAxis.refreshTicks(CategoryAxis.java:1106)
at org.jfree.chart.axis.CategoryAxis.reserveSpace(CategoryAxis.java:823)
at org.jfree.chart.plot.CategoryPlot.calculateDomainAxisSpace(CategoryPlot.java:3144)
at org.jfree.chart.plot.CategoryPlot.calculateAxisSpace(CategoryPlot.java:3211)
at org.jfree.chart.plot.CategoryPlot.draw(CategoryPlot.java:3257)
at org.jfree.chart.JFreeChart.draw(JFreeChart.java:1230)
at org.jfree.chart.JFreeChart.draw(JFreeChart.java:1114)
at net.sf.jasperreports.charts.util.DrawChartRenderer.render(DrawChartRenderer.java:62)
at net.sf.jasperreports.engine.export.JRPdfExporter.exportImage(JRPdfExporter.java:1505)
at net.sf.jasperreports.engine.export.JRPdfExporter.exportElements(JRPdfExporter.java:854)
at net.sf.jasperreports.engine.export.JRPdfExporter.exportPage(JRPdfExporter.java:818)
at net.sf.jasperreports.engine.export.JRPdfExporter.exportReportToStream(JRPdfExporter.java:695)
at net.sf.jasperreports.engine.export.JRPdfExporter.exportReport(JRPdfExporter.java:417)
at org.talend.dataquality.reporting.JasperReportBuilder.run(JasperReportBuilder.java:300)
at org.talend.dataquality.reporting.engine.ReportDocGenerator.generateReport(ReportDocGenerator.java:679)
at org.talend.dataquality.reporting.engine.ReportDocGenerator.generateDoc(ReportDocGenerator.java:413)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator.generateDocInternal(LocalReportDocGenerator.java:175)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator.access$0(LocalReportDocGenerator.java:174)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator$1.run(LocalReportDocGenerator.java:163)
at org.talend.repository.RepositoryWorkUnit.executeRun(RepositoryWorkUnit.java:99)
at org.talend.core.repository.model.AbstractRepositoryFactory.executeRepositoryWorkUnit(AbstractRepositoryFactory.java:258)
at org.talend.repository.localprovider.model.LocalRepositoryFactory.executeRepositoryWorkUnit(LocalRepositoryFactory.java:3217)
at org.talend.core.repository.model.ProxyRepositoryFactory.executeRepositoryWorkUnit(ProxyRepositoryFactory.java:2024)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator.generateDoc(LocalReportDocGenerator.java:168)
at org.talend.dataquality.reporting.engine.ReportDocGenerator.generate(ReportDocGenerator.java:293)
at org.talend.dataprofiler.core.tdq.action.LocalReportDocGenerator.generate(LocalReportDocGenerator.java:129)
at org.talend.dataprofiler.core.tdq.ui.action.GenerateReportFileAction$2.run(GenerateReportFileAction.java:342)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Re: Data Profiling Report Generation Failure

Anonymous — Thu, 10 Nov 2016 08:49:19 GMT

Hi,
For your Talend Big Data Platform product, could you please report a ticket on talend support portal so that we can give you a remote assistance on your issue through support cycle with priority?
Best regards
Sabrina