topic Re: How to skip files which contain error in Talend Studio

How to skip files which contain error

Anonymous — Sat, 16 Nov 2024 13:40:07 GMT

Hi everyone,
I am trying to read multiple XML files from an input directory and then using tMap transform them in desired output format.
While doing so, some of the input files in the directory are having some error (related to XML tags) which is leading my job to fail in between on encountering those non-compliant files. Since the number of files is huge (in the directory), I am not able to check for the files which are non-compliant so that I can remove it. Also, I prefer the job to do this activity so that I can have a log with me.
Can you suggest me a module(s) and arrangement so that my job can skip non-compliant files whenever it encounters one and logs the same?
I am using tFileList to iterate on the directory, then I have tFileInputXMl followed by tMap and tAdvancedFileOutputXML.
Thanks,
Ankur

Re: How to skip files which contain error

Anonymous — Wed, 25 Nov 2009 13:51:22 GMT

Hello
Move the process 'parsing xml file and merging' into a child job, like
on father job:
tFileList--iterate-->tRunJob(run the child job)
on child job:
tFileInputXML--main--->tMap--main-->tAdvancedFileOutputXML
1)Pass the current file path to child job, set it as file path on tFileInputXML.
2)On tRunJob, uncheck the 'die on child error' option.
Best regards
shong

Re: How to skip files which contain error

Anonymous — Thu, 17 Sep 2015 03:46:18 GMT

Hi Shong
May I know what's row component is in child Job ?
Because I'm also doing kind of same task and need to understand this job !!

Re: How to skip files which contain error

Anonymous — Thu, 17 Sep 2015 10:13:42 GMT

Hi ashish123
It is main link here, not a component. I will update my post.
Best regards
Shong

Re: How to skip files which contain error

Anonymous — Thu, 24 Sep 2015 06:30:01 GMT

Hi Shong
I'm working on one talend job and I want whole file to be rejected containing any single error.
Right now I have a job who checks files record wise and reject only bad one and insert good records from file. But I dont' want to insert anything from file having any single error but reject the whole file.

I also want to store reason why that file got rejected in mysql log table through tlogcatcher or any other component.
But in this job am storing the rejected records in some table with tschema error message in mysql table.
Please help me for the same.
Best !!

Re: How to skip files which contain error

Anonymous — Sat, 26 Sep 2015 03:28:30 GMT

Hi
To achieve the request, you have to redesign the job as I shown in my previous post, and you need to implement transaction management with tMysqlConnection+tMysqlcommint and tMysqlRollback.
on father job:
tFileList--iterate-->tRunJob(run the child job)
on child job:
tMysqlConnection
|
onsubjobok
|
tFileInputXML--main--->....
-onsubjobok--tMysqlCommit
-onsubjoberror--tMysqlRollback
1)Pass the current file path to child job, set it as file path on tFileInputXML.
2)On tRunJob, uncheck the 'die on child error' option.
3)In the child job, use an existing DB connection on tMysqlOutput component.
Best regards
Shong

Re: How to skip files which contain error

Anonymous — Sun, 27 Sep 2015 17:33:41 GMT

Hi Shong
My Jobs is like this
Parent Job: tFileList------iterate----->tRunJob
Child Job: tFileInputDelimited----main---->tMap----main----->tOracleOutput
- OnSubJobOK ---->tOracleCommit
- OnSubJobError---> tOracleRollBack
First Question is that, In job which you had suggested me in your previous post why did you use tOracleConnection ? (am using oracle that's I'm using oracle instead of mysql)
Second is that, Why am getting ora - 00911 invalid character error ? I guess I have settled everything ok at my end but if you think still am missing something please let me know for this error.
Best !!