topic Re: [resolved] Save rejected rows in Talend Studio

[resolved] Save rejected rows

Loneliness — Wed, 24 Sep 2014 09:10:03 GMT

Hi everybody, i am reading from a delimited file and writing the output in a table. The source file has this schema :
a:varchar
b:integer
c:date
c:varchar
Some of the rows have more fields than the expected and they should be rejected, what i need to do is to save these rejected rows somewhere. Can i achieve that without knowing before how many fields the row has?
Than you all

Re: [resolved] Save rejected rows

Anonymous — Wed, 24 Sep 2014 09:20:15 GMT

Hi,

Some of the rows have more fields than the expected and they should be rejected,

Could you please elaborate your case with an example with input and expected output values? Are your looking for TalendHelpCenter:tSchemaComplianceCheck which helps to ensure the data quality of any source data against a reference data source.
Best regards
Sabrina

Re: [resolved] Save rejected rows

Loneliness — Wed, 24 Sep 2014 10:22:35 GMT

Hi Sabrina, as you see in the screenshot, my file has 2 fields: region_id: integer and region_name:varchar.
The last 2 rows of the file have 3 fields and what i want to do is to save the rejected rows in all cases. tSchemaComplianceCheck would check types, nullability and length of rows against reference values but what if in my file i have a row that has the 2 first fields correct but it has other fields? tSchemaComplianceCheck will not detect this row. I want the third row of my file (3,b,c) to be rejected and saved somehow...is it possible?
Thank you

Re: [resolved] Save rejected rows

Anonymous — Wed, 24 Sep 2014 10:51:25 GMT

Hi,
You can set a length limitation for your column "Region_name" to avoid your third row of file (3,b,c).
See my pic, feel free to let me know if it is Ok with you.
Best regards
Sabrina

Re: [resolved] Save rejected rows

Loneliness — Wed, 24 Sep 2014 11:36:12 GMT

Thank you Sabrina for your reply, actually, this resolve only in part my problem, i still need to save the whole rejected row in my output, including the additional fields if any, and your solution only show the first 2 fields because the source file schema has only 2 fields.
For example: the row (3,b,c) is correctly rejected, but in the rejected output i only have:
|=---+----+-------------------------------
|code|name|errorMessage
|=---+----+-------------------------------
|3 |b | name.length() > 1 failed
so i loose the third field (c)...
Thanks again

Re: [resolved] Save rejected rows

Anonymous — Thu, 25 Sep 2014 05:16:25 GMT

Hi,
For your input data:
1,Europe
2,Asia
a,b,c
3,b,c

If you want to use "," as field separator, you need add a column filed 3 for your third field (c)...
See my pics
Best regards
Sabrina

Re: [resolved] Save rejected rows

Anonymous — Fri, 26 Sep 2014 04:28:35 GMT

Hi cheaito,

Any update for your issue? Is this solution resolve your issue?
Best regards
Sabrina

Re: [resolved] Save rejected rows

Loneliness — Fri, 26 Sep 2014 08:23:22 GMT

Hi Sabrina, actually no...what i wanted to achieve is the following:
I define the file schema as 2 fields Field1: int; Field2: String
For some reason, the source file has some wrong rows that do not respect this schema and these rows have more fields (more separators)
I want my job to reject these rows and to show me the rejected row entirely and not only part of it!
I can't modify the schema definition of the source file because it should have only 2 fields and i want the job to reject those rows, if any, that have more fields and in my output i want to see something like this:
The row: "3,b,c" was rejected because it has more fields than the expected!
Thank you again