topic Re: tSchemaComplianceCheck not validating dates with timezone in Talend Studio

tSchemaComplianceCheck not validating dates with timezone

Rimvis — Sat, 16 Nov 2024 04:20:19 GMT

Hi all,

I'm facing a problem when using tSchemaComplianceCheck component to validate incoming date/time.

The data comes in the format "yyyy-MM-dd'T'HH:mm:ssXXX", for example "2019-10-02T21:45:33+01:00".

Schema check fails with the message "wrong DATE pattern or wrong DATE data"

However, when I'm parsing the same date using the same format, it's working without any issues:

TalendDate.parseDate("yyyy-MM-dd'T'HH:mm:ssXXX",... )

I have tried setting "Ignore TimeZone when check date" without any effect.

Using Talend Open Studio for Data Integration, v7.1.1

Any ideas?

Thanks,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Fri, 18 Oct 2019 14:28:47 GMT

Hi,

If its not working for tSchemaComplianceCheck component alone, could you please create a feature request using JIRA?

https://jira.talendforge.org

The interim feature will be to parse it using date parse function and catch any error and convert it to yes or no flag for rejection.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Fri, 18 Oct 2019 14:50:52 GMT

Thanks @nthampi

At the moment I'm just trying to understand, if I'm doing something wrong, or the component does not support date formats with timezone (XXX) at all. I'm happy to raise a feature request if this is the case.

Thanks,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Fri, 18 Oct 2019 21:59:46 GMT

Hi Rimvis,

Apologies for the confusion. Just now I checked it again.

You can definitely check the results for this format. You need to read the data as String and do the date check in tSchemaComplainceCheck component as shown below.

If the data is correct, it will be transferred to the main output as shown below.

If the data is wrong, it will be transferred to Reject flow as shown below.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Mon, 21 Oct 2019 15:57:48 GMT

Hi @nthampi ,

It appears that the problem exists only if local time zone is set to UTC. It works as expected under other time settings.

Do you know if there is any workaround for that? Without changing the time settings that is

Thanks,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Mon, 21 Oct 2019 16:19:14 GMT

Hi,

Could you please share a sample value you are using and the error you are getting with screenshots?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Tue, 22 Oct 2019 07:58:58 GMT

Hi @nthampi ,

It's the same value: "2019-10-02T21:45:33+01:00"

The error: wrong DATE pattern or wrong DATE data

Here's screenshot with time zone set to UTC+2 (Helsinki , ...)

And here's with UTC:

I'm attaching my sample job as well.

Thanks,

Rimvis

test_schema_check.zip

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Thu, 24 Oct 2019 04:32:50 GMT

Hi,

I could not understand any difference between 2 screenshots. I was able to run your job with the data you have given in the sample job.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Thu, 24 Oct 2019 05:20:11 GMT

Hi @nthampi ,

In the first screenshot record is validated, and it goes to Main output (Check OK)

In the second screenshot validation fails, and the record goes into Rejects (Check Failed)

All I did between those runs is changed time zone to UTC.

Are you saying it runs OK for you in UTC time zone?

Thanks,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Thu, 24 Oct 2019 14:53:58 GMT

Hi,

I could not see any difference in the values in both screenshots. Where are you changing the UTC time zone?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Thu, 24 Oct 2019 17:08:50 GMT

Hi @nthampi ,

I'm not changing the value, that's the point The same value is being validated if process is running in UTC+2 timezone, but fails if running in UTC.

I'm changing time zone in Date & Time settings on my computer:

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Thu, 24 Oct 2019 17:23:59 GMT

Hi,

I got your point now 🙂

It looks like a bug and could you please create jira ticket for this issue https://jira.talendforge.org

If you are an enterprise customer, please create a support case so that they can report the bug on your behalf and fast track the resolution.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Mon, 04 Nov 2019 12:01:41 GMT

Hi @nthampi ,

Sorry for not responding for a while. I was about to create a ticket, but I'm not sure which project should I pick. Can you please advise?

Thanks,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Tue, 05 Nov 2019 05:16:12 GMT

Hi,

You can select Studio or Talend Data Integration.

In case you selected wrong project accidentally, they will change it to right project 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Tue, 05 Nov 2019 09:53:37 GMT

Thanks @nthampi ,

I have raised an issue: https://jira.talendforge.org/browse/TDI-43136

Let's see how it will go.

Cheers,

Rimvis

Re: tSchemaComplianceCheck not validating dates with timezone

Rimvis — Wed, 06 Nov 2019 07:39:44 GMT

It appears that there is a check box called Use Strict Date Check (in Advanced settings). Validation is working correctly under UTC if this check box is enabled.

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Fri, 10 Jan 2020 20:31:09 GMT

@nthampi

Thanks, But if we use String as data type then invalid date like "0000-00-00T00:00:00.000" will be treated as valid date. how to capture this scenario then?

Re: tSchemaComplianceCheck not validating dates with timezone

Anonymous — Mon, 13 Jan 2020 10:16:20 GMT

It appears that there is a check box called Use Strict Date Check (in Advanced settings). Validation is working correctly under UTC if this check box is enabled.